One-Class discriminatory classification with imbalanced, heterogenous Negative background?

One class learning

I wouldn't be too quick to throw out one-class classification methods (option 2) - the key is to model the positive (minority) class with the one-class model.

There has been research demonstrating cases where one-class classification out-performed other approaches like sampling for highly imbalanced data as often seen with protein classification tasks.

I couldn't find the research I recalled, but I did find some other comparisons, showing using one-class classifiers (typically modeling the minority class) achieved as good or better performance than binary classification typically with sampled "negatives" from the large set of proteins not known to be positive.

Additionally this approach also gives the advantage of much improved run-time - since you only need to train the classifier on the smaller, positive set.  A couple papers:

"Prediction of protein-protein interactions using one-class classification methods and integrating diverse biological data"

"A One-Class Classification Approach for Protein Sequences and Structures"

At the very least I would try some one-class methods and compare the performance using validation with your binary/multi-class classification approaches.  There are also open source implementations for many of these so it shouldn't be too costly to try them out, for example LibSVM has a one-class SVM implementation.  Additionally, it might prove valuable for use in an ensemble with binary classifiers, since there may be more disagreement in their predictions.

Higher level representation embedding / clustering

Along the lines of what you were thinking with (1) and the other post suggesting PCA, approaches like clustering, sparse coding, or even topic modeling - treating each protein as a document string and different protein families as different topics - could yield a representation that might make classifying the proteins straightforward.

I.e., you could identify which group/cluster a protein belongs to or classify the cluster memberships / embedded representations.

E.g., such embedding approaches as sparse coding can yield representations that reveal which cluster a protein belongs too - so that some sets of features are only active (non-zero) for proteins in the same cluster - which can make classifying them much easier.

Additionally class labels or known cluster membership can be incorporated in the embedding process for most methods.

Ensemble

Ensembles of multiple classifiers tend to work best - especially when the classifiers are very diverse and can achieve comparable performance individually.

There are at least two ways use ensembles for this problem.

You can build an ensemble of binary classifiers by sampling multiple different same-size negative sets and training a classifier on each.
You can build an ensemble from different approaches, such as binary classifiers with different negative samples, combined with a one-class classification approach, combined with classification models trained on the embedded data.


Building Recommendation engine with Python

I don't know of a well documented python option for collaborative filtering, but one option is to use Spark's MLib. There is a wrapper that allows you to run it in python (pyspark) and it is straightforward to get it running on your local machine.


Boosting with highly correlated features

Actually, your understanding of a random forest is not 100 percent correct. Variables are sampled per split, not by tree. So every tree has access to all variables.

In general, tree based models are not too strongly affected by highly correlated features. There are no numeric stability issues as with least squares. You can easily add a variable twice without numeric problem. Note however that most interpretability tools like split importance or partial dependence plots are affected by multicollinearity. So be careful with them in such cases.


What is the most effective unsupervised ML algorithm to use when outliers are present in data set?

You could try a hierarchical clustering approach. As an example, K clusters could initially be found for the data points. Then, for each of the K clusters, an arbitrary number of clusters could be found from the data points within the cluster to further refine the clustering.


How to approach graphed data for a binary classification system?

Super comprehensive question. You are basically asking for tones of directions! I'd suggest to start with link prediction problem. So assume you have a directed/signed/weighted graph and you want to find the most probable next interaction (link/edge). I'd suggest to have a look at Jerome Kunegis papers and you may see it as a starting point and go on from there.


Predict compatibility of 2 people as boolean classification problem

Your approach to modeling compatibility seems sound, and definitely makes more sense than market basket analysis. From what I could gather, your goal is to predict compatibility, and you have access to a certain signal that you can use as your compatibility label for the training set (for example, you know for a fact that persons X and Y are compatible but Y and Z are not). In these circumstances, supervised learning definitely sounds appropriate.

Based on the example you gave, there are a few things about this problem that you may consider (if you haven't already):

Should the features (e.g. compassion, extroversion, humor) be binary, or on a continuous spectrum? This depends on your data, of course, but perhaps you may get a better measure of compatibility if you can get the finer grain of detail that continuous features provide.
Same for the class: is it exclusively binary (classification) or could this be treated as a regression problem (predict the degree of compatibility)? Even if treating it as classification, certain models output class posterior probabilities, which could be interpreted as degrees of compatibility.
Compatibility is most likely a symmetrical relationship: if a is compatible with b, then b is compatible with a. If your dataset is formed of the two individuals' data concatenated, should you create two examples per pair (one in each order)? Is there another way to enforce symmetry in your model? Also, can the signal you use for your compatibility label really be interpreted in this symmetrical sense, or does it only tell you that, for instance person A likes person B, but not necessarily the other way around.
How will you evaluate your model? If compatibility is more continuous than discrete, for example, then perhaps recall, precision and accuracy aren't the best metrics, and you may want to use something like RMSE?


Finding an optimal learning rate is an important step in optimizing a neural network. As discussed at length here it is not a trivial question, but there are some ways to get a good starting value. Main idea here is to plot loss vs learning rate for different values and choose the learning rate where the slope is the highest:

This image is from the link above. This figure tells me that anything from a little right of 10-5 to around 10-3 can be a good learning rate. One can also set a relatively high learning rate, and reduce it when the loss function reaches a plateau, so in the above example, you would be better off by starting with the highest end; and lower the rate later as needed. This can be achieved by a learning rate scheduler (such as the one in Keras callbacks). This way, you won't spend a lot of time at the initial epochs where there is a lot to learn and the loss drops quickly. Three key parameters of a scheduler are "factor","patience" and "min delta" . If the loss function does not change by "min delta" in "patience" number of epochs, the learning rate is reduced by the ratio defined in "factor".

You also mention decay in your post, which can be considered as the default learning rate for some optimizers. And if I remember correctly, it actually associates a different learning rates for each parameter, and reduces them at a different rate. You can find a more detailed comparison here. But the main reason to use a LR scheduler even when a decay is present is to have more control on the learning process. When the training reaches a plateau, you might have to wait for a long time before the default decay in Adam (for example) gets to a low enough value to get over that region and start learning again. But with a scheduler, you can define when to lower the learning rate by a significant amount (compared to default decay), so that further learning is possible. Keep in mind that there is not much to learn in the later epochs compared to the ones in the beginning.


How to implement a basic query management and recommendation system

You might want to take a look at the solr project to see if it is a good fit.


Can't save save/export and load a keras model that uses eager execution

I ran into the same problem and solved it by running the keras that comes with tensorflow:

from tensorflow.python.keras.models import Model, load_model

instead of:

from keras.models import Model, load_model

I suspect there's a version mismatch at the core of this problem. Loading the model worked with the Keras included with the current Tensorflow 2.0.0-beta1.


Yes, they are support vectors. This is because they contribute (or serve as support) in the computation of the hyperplane that separates the positive and negative regions. This hyperplane is given by:
$$\mathbf{w}^T x + b = 0 $$
Where $x$ is our $\text{n-dimensional}$ input vector that contains the $n$ features of our problem (the input samples during the learning process), and $b$ and $\mathbf{w} = (w_1,w_2,...,w_n)^T$ are the parameters that are optimized. More concretely, $b$ is the intercept of the hyperplane and  $(w_1,w_2,...,w_n)^T$ is its normal vector (justification in Mathworld).
But why the misclassified points contribute?
More detailed justifications can be found in the Andrew Ng notes about SVM which I recommend.
To understand why they contribute, we need to have a look at the cost function, $J$, that is used when the data is not linearly separable, like the case of the question (this cost function is also used to prevent the influence of outliers):
$$J = \frac{1}{2}\Vert \mathbf{w}\Vert^2 + C \sum_{i=1}^m \xi_i$$
$$\text{subject to}\begin{cases}
y^{(i)}(\mathbf{w}^Tx^{(i)}+b)\geq 1-\xi_i, \,\,\,\,\,\,\,\,\,\, i = 1,...,m\\
\xi_i \geq 0
\end{cases}$$
Where $\xi_i$ is the slack of the input sample $x_i$, being $\xi_i = 0$ only when the input sample $x_i$ is correctly classified and presents a functional margin $\geq1$.
In order to solve this in a efficient way, it can be proved that by applying the Lagrange duality to the problem presented before (minimizing $J$ w.r.t. $\mathbf{w}$ and $b$), we end up with a equivalent problem of maximizing the next function w.r.t. $\alpha$:
$$ \max_{\alpha}\,\,\,\, \sum_{i=1}^m\alpha_i - \frac{1}{2}\sum_{j=1}^m\sum_{i=1}^my^{(i)}y^{(j)}\alpha_i\alpha_jx_ix_j$$
$$\text{subject to}\begin{cases}
0\leq \alpha_i\leq C, \,\,\,\,\,\,\,\,\,\, i = 1,...,m\\
\sum_{i=1}^m\alpha_iy^{(i)}=0
\end{cases}$$
Where each $\alpha_i$ is the Lagrange multiplier associated with the input sample $x_i$.
Furthermore, it can be proved that, once we have determined the optimal values of the Lagrange multipliers, the normal vector of the hyperplane can be computed by:
$$ \mathbf{w}=\sum_i^m \alpha_i y^{(i)}x^{(i)}$$
Now we can see that only the vectors with an associated value of $\alpha_i\neq0$ will contribute to the computation of the hyperplane. This vectors are support vectors.
Now the question is: When $\alpha_i \neq 0$?
As explained in the notes linked above, the values of $\alpha_i \neq 0$ are derived from the KKT conditions which are needed to be satisfied in order to find the values of $\alpha_i$ that minimize our cost function. These are:

$\alpha_i = 0 \implies y^{(i)}(\mathbf{w}^Tx^{(i)}+b)\geq 1$
$\alpha_i = C \implies y^{(i)}(\mathbf{w}^Tx^{(i)}+b)\leq 1$
$0<\alpha_i < C \implies y^{(i)}(\mathbf{w}^Tx^{(i)}+b)= 1$

So, in conclusion the vectors (samples) that lie on the margins (condition number 3 and condition number 2 when $=1$), the vectors that are correctly classified but lie between the margins and the hyperplane (condition number 2 when $<1$) and the vectors that are misclassified (also condition number 2 when $<1$) are the ones with $\alpha_i \neq0$ and therefore contribute to the computation of the hyperplane $\rightarrow$ they are support vectors.


If you are capable of optimizing the weighting hyperparameter γ, then the relative importance of the loss functions becomes an empirical question. That data will help guide the mixture of losses.
Normalizing the gradients is ad hoc. Thus simplifier but could over simplify the problem.


Gap leaderboard score and model scoring on a Competition

As you have commented, you are concerning about over-fitting.

In fact, cross validation will help to weaken over-fitting, but it can't eliminate over-fitting. Model scoring on TRAIN dataset sometimes exceeds scoring on TEST dataset. Here are some examples I can find:

Deep Residual Learning for Image Recognition (the famous ResNet paper). Checkout figure.1
This kernel of Two Sigma Financial Modeling Challenge on Kaggle says:

  we are getting a public score of 0.0169 which is slightly better than the previous one.
  Submitting this model to the LB gave me a score of 0.006

In my practice, I am more willing to concern the scoring gap between TRAIN, VALIDATION(DEVELOP) and TEST dataset. Personally speaking, TRAIN>VALIDATION=TEST is better than TRAIN=VALIDATION>TEST.

Edit: For class imbalance problem, there are some resources:

8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset. This blog shows a common workflow dealing with imbalanced class issue.
Class Imbalance Problem in Data Mining: Review. This paper compares several algorithms created for solving the class imbalance problem.


memory error in matrix cosine_similarity

This is talking about RAM. There are a few factors that will decide how many rows/columns you can use. Instead of rows/columns, it is maybe easier to just think in total number of elements: num_rows * num_cols. The memory you will require is going to have a relationship to this number.

There are ways that might take less working memory to solve the problem - usually memory and speed are part of the trade-off. If you use a lot less memory while computing the result, you can do fewer at a time and so it takes longer.

If you have floating point numbers (those with decimals), then Pandas usually uses the data type float64 by default. You could try using float32 instead. It offers about half of the accuracy, but also only uses half of the memory. You can do this by simply adding this line before you compute the cosine_similarity:

import numpy as np

normalized_df = normalized_df.astype(np.float32)
cosine_sim = cosine_similarity(normalized_df, normalized_df)

Here is a thread about using Keras to compute cosine similarity, which can then be done on the GPU. I would point out, that (single) GPUs will generally have less working memory available than your computer itself.

Here is a blog that speaks about how to scale up your computation to use tools like Spark, for distributed computing. That would allow you to deal with much larger matrices, provided you have several computers available.


Why might several types of models give almost identical results?

This results means that whatever method you use, you are able to get reasonably close to the optimal decision rule (aka Bayes rule). The underlying reasons have been explained in Hastie, Tibshirani and Friedman's "Elements of Statistical Learning". They demonstrated how the different methods perform by comparing Figs. 2.1, 2.2, 2.3, 5.11 (in my first edition -- in section on multidimensional splines), 12.2, 12.3 (support vector machines), and probably some others. If you have not read that book, you need to drop everything RIGHT NOW and read it up. (I mean, it isn't worth losing your job, but it is worth missing a homework or two if you are a student.)

I don't think that observations to variable ratio is the explanation. In light of my rationale offered above, it is the relatively simple form of the boundary separating your classes in the multidimensional space that all of the methods you tried have been able to identify.


K-means does not use labels.

The example that you looked at uses labels to compare the clusters to the labels. That part obviously requires labels; but that is not part of k-means.


How can causal inference be used with machine learning?

Your understanding is correct. Finding correlations between variables is simple but turning them into causal assertions needs an extra effort. Causal inference is used mostly to reach a "prescription" in the form of "do X so that Y happens".

When not to use Causal inference:

If it is possible to do experiments, causal inference can be avoided. For example, A\B tests let you study the effect of a change in two groups and reach a causal conclusion. For example, result of an A\B test would be "users in group A which see a button with color intensity 50 clicked 10% more than group B with color intensity 40", so do X='increase color intensity of the button' so that Y='more click' happens. With larger, more uniform groups, your assertion would be more reliable.

Causal inference in Machine learning:

In most machine learning projects these type of experiments are possible and mostly cheap, therefore why bother? Moreover, specially in predictive projects, value comes from correlated relations. Knowledge of causal relations, which are a subset of correlated relations, does not add value.

Causal inference:

When you work with historical data or you can only "observe" the data without affecting it, causal inference comes into play. Generally, causal inference is a controversial topic as it tries to extract causal relations from observational data (as opposed to experimental data in A\B tests).

To best of my knowledge, the main contributor to causal inference is Prof. Judea Pearl. His underlying tools are probabilistic graphical models (PGMs) and do-calculus. These tools let us explicitly encode our assumptions about the mechanics of data generation, and reach causal conclusions. So when an assertion like "do X, so that Y happens" goes wrong, we can track the problem in our assumptions in a principled way. For example, we may have ignored an important hidden variable that by including it our conclusion would change. He fundamentally says, anyone who reaches a "prescriptive" conclusion is doing causal inference, so it is better to lay out your assumptions explicitly, to prevent correlation-causation problems going unnoticed.

Some interesting resources:

Simpson's paradox is an entry point to become interested in causal inference,
This paper of Judea Pearl connects the paradox with causal inference.


Plotting multiple precision-recall curves in one plot

Try using Matplotlib gca() method in this way you can indicate what axis you want to plot in
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import plot_precision_recall_curve
import matplotlib.pyplot as plt

plot_precision_recall_curve(ada, X_test, y_test, ax = plt.gca(),name = "AdaBoost")

plot_precision_recall_curve(ada_sm, X_test, y_test, ax = plt.gca(),name = "SMOTE")

plot_precision_recall_curve(ada_rus, X_test, y_test, ax = plt.gca(),name = "RUS")

plt.title('Precision-Recall curve')


Features selection: should I mix features?

Let LASSO pick the best ones. If the features are highly correlated and you want them picked as a group, add some L2 regularization too. This is called Elastic Net regularization, and it is a generalization of L1 and L2 regularization. Other than that, do not feel obliged to artificially group features.


Is it possible to change the sequence of convolution layer and pooling layer?

In crude terms, a single convolution layer tries to capture to what degree certain feature is present in given region of figure. Max pooling layer is a downsampling strategy: instead of capturing presence of certain feature to a very fine-grained degree, the layer downsamples to certain classes like almost present, almost not present. (In crude terms because what exactly each layer learns is highly specific to dataset and network architecture used).

If you think of convolution as signal extraction and max pooling as a downsampling/smoothing strategy, it makes sense to use smoothing after signal extraction and not before.


Problem importing CNTK in Azure jupyter notebook

Keeping your latest edit in mind, the error in your screenshot is actually just a recommendation to upgrade pip. As suggested, please run pip install --upgrade pip, and it shall upgrade pip package in your environment and shall not re-appear unless a new version of pip get released later on.


input_dim for Dense Layer after LSTM layers Keras

You do not need to specify the input_dim for the later layers, the model can infer the shape of those input layers from the output shape of the previous layer.

In addition, the input_dim values currently specified don't match the output dimensions of the prior layers- for example, the output dimension of the LSTM layer with return_sequences = False will be (200,). The network should fail to compile with an expected input size error with this code.

You can fix this by correcting or removing the input_dim values from the final three layers.


How can i add weights in a bag of words model in text analysis?

One possible solution is to introduce prior counts for words (higher counts for words that are more important) that could be added to the term-document matrix.

An alternative solution is to compute tf-idf features (weights that modify word counts based on frequency) and apply additional weighting to tf-idf with higher weights corresponding to important words.


Implementing a custom hard sigmoid function

Based on this post, hard-sigmoid in Keras is implemented as
max(0, min(1, x*0.2 + 0.5)). To obtain the graph you like you have to tweak the shift and slope parameters, i.e. leave them out in your case:

$$
max(0, min(1, x))
$$

This will generate following graph:

For Keras' TensorFlow backend you can find the implementation here.
This would be the corresponding changed "hard-sigmoid", for your case:

zero = _to_tensor(0., x.dtype.base_dtype)
one = _to_tensor(1., x.dtype.base_dtype)
x = tf.clip_by_value(x, zero, one)
return x


Website navigation path using graph theory

I've done a lot of web analytics work and while we did do path analysis, we never focused so much on whether the user took the absolute shortest path to the page.

Unlike building a road system, a website should be built to provide different paths to the same content depending on the intent of the user. For example, some users know exactly what they want and will do a search. Other users are exploratory and want to browse.

Finally, sometimes you specifically want the user to step through a handful of pages to sell them on a conclusion. This is called a conversion funnel. If the user landed directly on the conversion point without seeing the info leading up to it, they'd bounce off the site and we'd lose the lead.


It is actually depends on the algorithm you are using.
For example, for random forests the ranges don't matter, since one feature is never compared in magnitude to other features. It's only the range of one feature that is split at each stage.

But on the other hand SVM or Logistic regression will probably do better if your features have roughly the same magnitude, unless you know apriori that some feature is much more important than others, in which case it's okay for it to have a larger magnitude.


Machine learning with partially defined labelling using existential quantifiers

It's probably possible.  I'll suggest a few plausibly-practical methods, starting from very crude (but probably not super-effective) to more complex (might be more effective but might be computationally expensive).  I expect that many of these approaches might be relatively slow to train the classifier but doable and worth a try.  But first, let me try to formalize the  problem more precisely.

Theory

In machine learning it's often helpful to formulate your problem by identifying a loss function.  Then you can formulate your problem as finding $f$ that minimizes the loss function.  In your case, I think a natural loss function would be that $L(f,g)$ counts the number of partitions $p$ such that conditions (2) or (3) are violated.  You could of course add a regularization term as well.

Once you've defined a loss function, your problem becomes: given $g$, find $f \in \mathcal{H}$ that minimizes $L(f,g)$, where $\mathcal{H}$ is the hypothesis space (the space of valid/legal models).

Let's start with a theoretical thought experiment.  Suppose you didn't care about the computational time to train a classifier, and just want to find a valid classifier $f$.  Let $\mathcal{H}$ be the hypothesis space, i.e., the set of allowable functions.  (For instance, for a linear SVM, $\mathcal{H}$ would be the set of SVM weights -- i.e., the set of linear decision boundaries.)  If you didn't care about computational time, you could just enumerate all $f$ to find one that minimizes $L(f,g)$ -- so this proves that the problem is well-defined.

More generally, suppose we have a space of classifiers $f$ that output probabilities instead of hard labels (think: logistic regression).  Then we can define a loss function based on the cross-entropy loss:

$$L(f,g) = \sum_{p : g(p)=0} \max \{-\log(1-f(x)) : x \in p\} + \sum_{p : g(p)=1} \min \{-\log(f(x)) : x \in p\}$$

You could add a regularization term here, too.

Of course, other loss functions are possible as well; but the point is that once you have a loss function, then classification becomes well-defined as an optimization problem.

Plausible method: one-class classifier

One crude approach is to use a one-class classifier.  For each partition $p$ such that $g(p)=0$, you obtain many instances $x$ where you want to have $f(x)=0$.  So, you could add each one of those to a training set, and then train a one-class classifier on it.

This is simple and easy to implement.  However, I suspect its accuracy might be low.

Plausible method: active learning

Suppose you have a way to obtain labelled instances if you need, through manual training.  In particular: let's assume that given a partition $p$, if you really needed to, you could construct hard labels for instances $x \in p$, and specifically, if $g(p)=1$, you could find an instance $x \in p$ that should receive the label $1$.  This manual-labelling process might be difficult and tedious and time-consuming/expensive, so not feasible to do on a large scale, but let's suppose you could do it on a small scale if needed.

Then you could use active learning:

Let $f$ be an arbitrary classifier (maybe trained using a one-class classifier).  Let $T$ be an initial training set, initially empty.
Repeat until convergence:

a. Pick a partition $p$ such that $f(x)$ is wrong for some $x \in p$.

b. If $g(p)=0$, add find $x \in p$ such that $f(x)=1$, and add $(x,0)$ to $T$.

c. If $g(p)=1$, use the manual-labelling process to find some $x \in p$ that should be labelled $1$ and add $(x,1)$ to $T$.

d. Train a new boolean classifier $f$ using standard supervised learning methods on the training set $T$.

There are various optimizations you could use to break ties in step 2c (in hopes of reducing the amount of manual labelling); e.g., if you're using the cross-entropy loss defined above, you could choose the $p$ that contributes the most to the total loss for $f$.  Or, you could choose the $p$ that is most "different" from all previously considered $p$ with $g(p)=1$.  Or, you might be able to adapt other active learning methods.

This might work.  However, it does require the ability to do some manual labelling of individual instances, which is more than you promised us in the original problem statement.

Plausible approach: direct optimization

Another possible approach would be to choose a differentiable classifier, and the optimize the (cross-entropy) loss $L(f,g)$ defined above directly using gradient descent.  For instance, you could choose $f$ to be a logistic regression or neural network classifier, as these are differentiable.  Now, you can compute the gradient of $L(f_\theta,g)$ with respect to the parameters $\theta$ of the model (e.g., for logistic regression, $\theta$ are the weights) and then apply gradient descent to find the model parameters $\theta$ that minimize $L(f_\theta,g)$.

This may get a little messy, as loss function $L$ contains $\min$ and $\max$ functions, which have points where they are non-differentiable.  If you don't do anything special, this can cause oscillation around the non-differentiable boundary.  One possible trick is to replace each $\min$ by a softened version of the min, and replace each $\max$ by a softened version of the max, so that the loss function becomes differentiable.

Alternatively, you could adjust ordinary gradient descent to behave well around them.  One standard technique is to reduce the step size when you get near the boundary to avoid stepping across the boundary, or to use projected gradient descent.  Thus, if we have a term

$$\ell(\theta) = \min(\ell_1(\theta),\ell_2(\theta))$$

and we compute the gradient $\nabla \ell(\theta)$, the gradient will be based on whichever of $\ell_1(\theta),\ell_2(\theta)$ is smaller for this particular value of $\theta$.  Suppose the gradient is $g = \nabla \ell(\theta)$.  Ordinary gradient descent would take a step in the direction $g$, say adjusting $\theta$ to $\theta - \lambda \cdot g$ for some small constant $\lambda$.  You could additionally require that the step avoid crossing the boundary line $\ell_1(\theta)=\ell_2(\theta)$.  For instance, if $\ell_1(\theta)<\ell_2(\theta)$ for this particular value of $\theta$, we'll have $g = \nabla \ell(\theta) = \nabla \ell_1(\theta)$.  So, take a step $- \lambda' \cdot g$, but constrain $\lambda'$ to be in the range $0 \le \lambda' \le \lambda$ and to satisfy $\ell_1(\theta - \lambda' \cdot g) \le \ell_2(\theta - \lambda' \cdot g)$ -- i.e., don't step across the other side of the discontinuity.

Note that using ordinary gradient descent will probably be very slow, as each evaluation of the loss function requires evaluating $f$ on every point in $X$: $O(|X|)$ time.  You'll probably do better by using stochastic gradient descent: in each iteration, pick a single partition $p$, compute the gradient of the term in $L(f,g)$ for $p$, and move in the direction of that gradient.  Or, pick a minibatch of 20 partitions $p$ and compute the gradient of the sum of those 20 terms.

Plausible method: boosting

Another reasonable method would be to use boosting with a very simple classifier, such as a decision stump.

A decision stump is a decision tree with a single level: you pick one feature, compare it to some threshold, and make a binary decision based on whether the value of that feature is above or below the threshold.  Decision stumps are convenient because you can enumerate all candidate stumps -- all possible combinations of a feature and a threshold -- and evaluate the loss function for each.  (In practice, as an optimization, we don't enumerate all possible stumps.  Rather, for a particular feature, we randomly pick 100 candidate thresholds by sampling 100 times from the training set, and we compute the loss for each candidate threshold; we do this for each feature, and take the best overall combination.)

You could use AdaBoost directly with decision stumps and minimize the loss function defined above, and you might get a reasonable classifier.

For even better results, you could use gradient boosting.  This is a slight generalization where each leaf of each stump outputs a continuous value rather than a boolean; the classifier's output is obtained by summing the value from each stump, and then checking whether the sum is positive or negative.  The general methodology can be applied with any differentiable loss function, so you could apply that methodology to the cross-entropy loss function defined above.

Plausible method: incremental optimization

Here's another approach that can be applied to classifiers that aren't differentiable.
In my algorithm, I will maintain a tentative training set $T \subseteq X \times \{0,1\}$ that assigns a tentative label to some of the elements of $X$.  The training set $T$ will evolve throughout the algorithm, but it will always remain consistent with $g$, defined as follows:

We say that $T$ is consistent with $g$ if, (a) for every $p$ such that $g(p)=0$, we have $(x,0) \in T$ for all $x \in p$, and (b) for every $p$ such that $g(p)=1$, there exists at least one $x \in p$ such that $(x,1) \in T$.

The training set $T$ specifies one possible labelling that would be consistent with $g$.  It's not necessarily the right one; it's just one possibility.

At each step, the algorithm uses $T$ as labels to train a classifier $f$ using standard supervised learning methods and computes $f$'s loss on $T$, then adjusts $T$ to reduce the loss of $f$ on $T$.  Define $\text{train}(T)$ to be the classifier $f$ obtained by training on the training set $T$, using some boolean classifier of your choice.  Here is the algorithm:

Set $T$ to be any training set that is consistent with $g$.  Compute $f := \text{train}(T)$ and $\ell = L(f,g)$.
Repeat until convergence:

a. Enumerate all training sets $T'$ that differ from $T$ in only one element yet remains consistent with $g$, and for each train a classifier and compute the loss.  Pick the $T'$ that minimizes $L(\text{train}(T'),g)$.

b. Set $T := T'$.

Step 2a can be done by enumerating all partitions $p$ such that $g(p)=1$. Suppose $T$ includes $(x,1)$ where $x \in p$.  Then enumerate all $x' \in p$ and consider $T' = T \setminus \{(x,1)\} \cup \{(x',1)\}$.  There are at most $|X|$ such possibilities, and you have to train the classifier on each, so each iteration of the loop might be pretty slow... but one could hope that it will eventually converge on a reasonable solution.

Basically, this tried to maintain some tentative labels that are consistent with your $g$, and that also are consistent with the regularity conditions imposed by the classifier $f$ (e.g., linear separability, or whatever), incrementally adjusting it at each step of the iteration.

For your particular parameter values ($|X|=10^5$, $|P|=10^3$), I suspect this will be too slow.  But you could try experimenting with the approach for scaled-down versions of your problem, with a smaller feature space ($|X|$ is smaller) and see if it seems to lead to useful results.

If this algorithm gives useful results, there many heuristics and optimizations you could use to speed up convergence:

You could probably make a smarter choice for the initial value of $T$ and speed up convergence, by applying a one-class classifier to the set of 0-labelled instances (since the instances of the form $(x,0) \in T$ never change throughout the algorithm) and using that to choose tentative initial label.
You could use something akin to stochastic gradient descent to speed convergence: in step 2a, pick a single $p$ (or a small minibatch of $p$'s) and  consider just training sets $T'$ that differ from $T$ only in $p$ (but not in other partitions.
You could try using an online/incremental training method for training $f$.  Since $T'$ is so similar to $T$, rather than training a new classifier from scratch, you might be able to reuse much of the effort from training on $T$ to more quickly compute $\text{train}(T')$ given $\text{train}(T)$.
Depending on the nature of the feature space and the partition function, you might be able to solve the following problem more efficiently than brute force: given a training set $T$ and a partition $p$ such that $g(p)=1$ and an $x \in p$ such that $(x,1) \in T$, find some $x' \in p$ such that minimizes the loss when training on $T' = T \setminus \{(x,1)\} \cup \{(x',1)\}$.  For instance, with logistic regression, you might be able to train a classifier $f_0$ on $T \setminus \{(x,1)\}$, then classify each $x' \in p$ using $f_0$ and rank them by the probability score that logistic regression outputs.  This is a heuristic and not guaranteed to find the optimal $x'$, but it's also a lot more efficient than enumerating all candidates for $T'$ and training a classifier on each.

A useful subroutine

Many of the above schemes require solving the following subproblem:

Given a partition $p$ and a classifier $f$, find $x \in p$ such that $f(x)=1$, or report that none exists.

If the classifier outputs confidence/probability scores, then the subproblem is:

Given a partition $p$ and a classifier $f$, find $x \in p$ that maximizes $f(x)$.

The naive way to solve this subproblem is to enumerate all $x \in p$.  Depending on the particular classifier you choose and the structure of the partitions, there may be more efficient algorithms.

For instance, if we have a linear classifier, maximizing $f(x)$ becomes equivalent to maximizing $x \cdot w$ for some known vector $w$.  If $p$ has a simple structure, there might be faster ways to solve this optimization problem.  For instance, if $p$ is a convex region defined by linear inequalities, you can maximize $x \cdot w$ subject to $x \in p$ using linear programming.

And you can use a fast algorithm for this subproblem to speed up the training process, for many of the methods sketched above.  Whether that is possible will depend on the specific type of classifier you choose and the structure of the partitions, but if you're trying to do this in practice, I recommend you ask a separate question about how to solve this subproblem for the particular type of classifier and partition structure that arises in your application -- that might enable significant computational optimizations.


Formatting multi-level variable width data for model

Looking back at this, I've ended up finding an answer to my own question.

What I am trying to do here is a pivot, which converts categorical data into multiple columns, and then sets the value to that of another column.

Additionally, I've found that it is very common for models to have thousands of columns, if not more.


Training textblob with 16k rows of labeled data won't work (only few are working)

Oh Wow, textblob's default setting can't deal with even a reasonable amount of data.

TL;DR: You do need better control over what the library does.

The class NaiveBayesClassifier, or more exactly it's superclass NLTKClassifier uses a default for of feature_extractor.  In this case it uses basic_extractor, which, apart from stemming the words in some way (not relevant for us here) it does the following:

features = dict(((u'contains({0})'.format(word), (word in tokens))
    for word in word_features))

Once per each document (sample).

Ouch!  That will create a python dictionary for every single document.  And store in these dictionaries the term frequecy for each document.  On 16k rows (documents/samples) that is very likely to be way too much memory.

On Ubuntu you likely have swap space, which is extra disk space used as memory, your OS figuring out it is out of memory starts swapping pages of memory onto this swap space.  Yet, swap space if super slow compared to actual memory, therefore the computation takes forever.  And, in the end, it is this swap space and this slowness that keeps the algorithm running forever, instead of overheating your CPU or making the machine completely frozen and out of memory.

Better approach to the problem

ML techniques do not understand words or strings in general, they only understand numbers.  textblob hides this fact from you because it performs the feature extraction behind the scenes.  That works for toy problems but will likely impair the solution of any more complex problem (be it because of problem complexity or amount of data, or both).

You need to take control of the feature extraction yourself.  A Naive Bayes classifier is a rather simple algorithm and will work well with a plethora of extraction techniques.  Term Frequency (simply counting the number of each distinct word in every document), will work well enough as long as you do not use expensive python dictionaries to hold the frequencies in memory.  But once you get control of your feature extraction there are many more techniques to use: TF-IDF, n-grams.  I find the sklearn's explanation of text features very enlightening.  But I digress.

Unfortunately nltk (the library under texblob) does not make things any easier.  It requires its training sets to have a dictionary interface.  In other words, without a good deal of hacks nltk cannot train on big amounts of data.

Option 1: Hook into nltk directly

First lets assume that your file is named data.csv and we will try to extract features reading the file line-by-line.  First let me generate a data file:

import nltk
nltk.download('gutenberg')  # just in case
from nltk.corpus import gutenberg
corpus = gutenberg.words('melville-moby_dick.txt')
with open('data.csv', 'w') as f:
    for i in range(10**4):  # 10k
        f.write(' '.join([w for w in corpus[i*3:i*3+3] if w != ','])
                + ',' + str(i%2) + '\n')
f.close()

This was only generation of some data, nothing too relevant to the problem at hand.  I do not have your file so I just generate a random 10k rows CSV using the words from Moby Dick text at random.

Onto the actual code.  We will still use texblob's Naive Bayes but will not allow it to construct its huge list of dictionaries.  Instead, we will hook into it directly:

from textblob.classifiers import NaiveBayesClassifier
import csv

cls = NaiveBayesClassifier('')

class FeatureDict(object):
    def __iter__(self):
        with open('data.csv') as f:
            for row in csv.reader(f):
                yield {w: w for w in row[0].split()}, row[1]

cls.train_features = FeatureDict()  # this is passed to nltk
cls.classify('bang Boom!')

It is a hack but runs in constant memory.

Option 2: switch to sklearn

sklearn's vectorizers will perform the operation within NumPy (or within
the NumPy arrays inside pandas' data frames).  Which are memory efficient - especially if compared with thousands of python dictionaries.

With sklearn one will need to do this in two steps: one for data vectorization and another for the prediction.  make_pipeline joins the steps.  Assuming the same data prepared in data.csv we can do:

from sklearn.feature_extraction.text import tfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
import pandas as pd

data = pd.read_csv('data.csv', names=['doc', 'label'])
# some words are badly formatted, sklearn have issues with them
data = data.dropna()
model = make_pipeline(TfidfVectorizer(), MultinomialNB())
model.fit(data['doc'], data['label'])
model.predict(['buggy ho!'])

The sklearn solution is slightly more elegant but may blow memory at some point (500k samples perhaps).  In that case, one would need to hack sklearn itself to do things in constant memory.


Which Graph Properties are Useful for Predictive Analytics?

Using Community detection you can build a recommendation system. The most commonly used algorithm in this field is Blondel Algorithm which u have probably seen in SNAP. Blondel is almost the fastest Community Detection algorithm among widely accepted ones and its result is pretty acceptable (at least according to modularity score). As a side comment, you may also want to have a look at NetworkX and igraph libraries for more graph algorithms and visualizations.

Hope it helps, good luck.


The notation of $splits(label)$ under Random Forest

You correctly mentioned the definition of impurity which is
$$I_G(P) = 1 - \sum _{i=1}^k p_i^2$$
This can be written as
$$I_G(P) = \sum _{i=1}^k p_i*(1 - p_i)$$
At any split, for any branch $T_i$, you calculate the probability using the classical definition i.e.,
$$p_i = \frac{|T_i|}{|T|}$$
Using this, you can derive the impurity definition $I(T_i)$ given in the paper.
Once you are able to calculate the impurity of a branch you can calculate the impurity of a node using the weighted sum of each branch, where weights are the probability of that branch. Hence,
$$I_{node} = \sum_{\forall T_i  \in splits(a)} \frac{|T_i|}{|T|} I(T_i)$$
Using the above impurity, you can calculate the change in impurity or the information gain as given in the paper.
I hope this solves your doubt.


Adding to the answer above,

The labeling totally depends on how you define it. You can define 0 as negative or as positive. However, for the sake of understanding and ease of readability, keep it meaningful.
The instances that are correctly predicted are given by the diagonal.
Here, '1' is True Negative or for the class labelled as 0 and '5' is True Positive or for the class labelled as 1.

If you find it difficult to interpret by a simple confusion matrix, you could plot it.
Check out plotConfusion by MathWorks.


Is this the way to obtain the same individuals for x_test and y_test?

train_test_split has this capability built-in.  Just pass all of the data in the first go like:

Code:

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=250)

Test Code:

from sklearn.model_selection import train_test_split

x = list(range(100))
y = list(range(100, 200))
x_train, x_test, y_train, y_test = train_test_split(
    x, y, test_size=0.3, random_state=250)

assert(all(xx == yy - 100 for xx, yy in zip(x_train, y_train)))
assert(all(xx == yy - 100 for xx, yy in zip(x_test, y_test)))


BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

You just stumble over one big problem in the NLP field : finding the perfect metric..

Most traditional metrics (BLEU, ROUGE, ...) simply does not take into account the distance in terms of semantics between barking and crying.

So according to these metrics, The dog is crying is as similar as The dog is salmon to the reference, the dog is barking.
From a human viewpoint, this is not correct, the first sentence is closer to the reference, because for example the second sentence makes no sense.

People recently tried to provide better metrics in this sense. You might be interested in BERT score.

The idea is simply to use a BERT model (that have been pretrained, therefore have some linguistic knowledge) to compute how similar 2 sentences are.


what is criterion in flcuster of scipy package?

Welcome to the community!
You may want to refer to a tutorial on Agglomerative Hierarchical Clustering before reading this answer. My explanation is more practical.
Assume the data below:
from scipy.cluster.hierarchy import ward, fcluster
from scipy.spatial.distance import pdist
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.text import TextPath

X = [[0, 0], [0, 1], [1, 0],
     [0, 4], [0, 3], [1, 4],
     [4, 0], [3, 0], [4, 1],
     [4, 4], [3, 4], [4, 3]]
X = np.array(X)
Z = ward(pdist(X))

plt.plot(X[:,0],X[:,1],'o')

We have 12 points in 2 dimensions. This is how they are distributed:

Now let's see how maxclust affects the clustering. Please note that I named clusters by numbers and plot the name of cluster each data belongs to, instead of its point.
for tt in [1, 3, 5, 9]:
    plt.figure(figsize=(8,8))
    plt.title('t={}'.format(str(tt)))
    plt.xlim((-1,5))
    plt.ylim((-1,5))
    memberships = fcluster(Z, t=tt, criterion='maxclust')
    for ii in range(len(memberships)):
        path = TextPath((X[ii,0],X[ii,1]), str(memberships[ii]))
        plt.plot(X[ii,0],X[ii,1],marker=path,c='b',markersize=50)

    plt.show()

Results are shown below. I explain each:
t is the parameter which limits number of clusters i.e. you can have maximum t clusters in output. If t=1, any other criteria will be ignored as all data points have to be in the same cluster.

So far so good. Let's say t=3. Now the fun starts. maxclust finds an optimal distance between each pair of points which are going to be in the same cluster. If t=3, without that distance, you get unbalanced clusters. In the figure below we have 4 obvious clusters and if I want 3 clusters out, two of them will merge and cause a cluster which is "wrong" (let's say wrong for simplicity). The maxclust distance threshold, prevents the algorithm to do so as it rejects far points being in the same cluster. So you see the data is clustered to 2 clusters instead of 3 which makes perfect sense. Please note that, yes, we have bigger clusters. But they are "right". It is like we zoomed out and see the clusters from a higher perspective.

Let's go with t=5. Now the algorithm is allowed to find 4 clusters. Topology tells you that finding "right clusters" i.e. clusters in which each pair of points have a relatively small distance (in compare with distances to other clusters), is possible. This means that maxclust can find an optimal distance threshold and t lets us have 4 clusters. So it works!

The last but not least is t=9. No we can have clusters more than the actual number needed (which was 4). Now maxclust tries to find its minimum distance between pairs so that points in same clusters are closer to each other than to other points in other clusters. As you see in each real cluster, The distance between center points to other two is smaller than the distance between those two diagonal points (as this distance is identical, clustering algorithms choose members randomly). So you see that each real cluster is divided to two, due to the effect of combination of maxclust and t

Hope it helped. Good Luck!


How to keep a subsetted value for calculating mean

You can answer this using tapply, which will give you the mean for all your subsets, e.g.

dummy data

df = data.frame(mon=rep(month.abb, times=10)

calculate mean for all months

tapply(df$temp, df$mon, mean, na.rm=T)


Is there any existing algorithm or formula that can calculate the complexity of a dataset?

well ... "if n elements each element is composed of m sub elements" sound like you have a set of sets from discrete objects. If this is the case you can go through Combinatorial Analysis like Graph Theory. You can model it in two ways:

1) Using Hypergraphs: You have your elements as hyper-edges of a Hypergraph. This model lets you compute distances, diversities, sizes, and many more graph theoretical measures e.g. look at this, this and this.

2) Simple Graphs: You can create simple complete graphs from each sub-element and connect them to other complete graph (forming cliques) of other sub-elements. See the example bellow:

element1 = [[a,b,c],[b,c,d],[e,f,g]]
Graph = [(a,b),(b,c),(a,c),(b,c),(c,d),(b,d),(e,f),(e,g),(f,g)]

using this you will have a huge graph (network) of your object on which you can calculate tones of structural and statistical measures.

I assume my thought were naive as your question is not clear and you need to provide more detailed example to get a more accurate answer.

Good Luck!


I would suggest training your own custom Dialog Act Classification Model. Detecting a dialog act is a Natural Language Understanding (NLU) problem. Using a dialog act classification model you can detect if a sentence is used for "greeting", "question", "opinion", "promise", etc.

Detailed approach:

Use a pre-trained language model (for example BERT), to train your own Dialog act classification model. You can create your own custom Dialog act class based on your requirement (as you want to remove specific types of sentences). You can class them as junk or use "greeting", "question", "opinion", etc as dialog class.

After you are done with the training you can loop through the list of sentences and filter them based on their predicted classes.

  You might have to research if there are similar corpus which you can
  use or have to manually label and then train the model. It will be a more reliable approach as compared to the word2vec or rule-based POS approach.

For more details on state-of-the-art approaches and dialogue act corpus follow this link. If you have no idea how to use a language model you can use the transformers library from huggingface. I hope you find this helpful.


Where exactly are the weights updated? Is it between the initial convNet and the RPN. All updateable weights are updated during backpropagation.

I assume you were trying to ask when are the weights updated?, well according to the original paper there are 3 way to train Faster R-CNN, each has it's own when to update weight time:

Alternating training aka 4-step alternating training, which is the one used by the author in the original faster r-cnn paper. It's explained quite well in the original paper
Approximate joint training, the one you mention in the question, after a full forward pass of both networks, a backward pass is done. I'm not quite sure what you mean by "In this case how are both considered simultaneously?"
Non-approximate joint training, I don't really understand about this one

Back to the Where exactly are the weights updated? question, well, some to be mention are the RPN kernerls, RPN Cls and Reg FC Layer, the detection (Fast R-CNN) Cls and Reg FC layer also the CNN kernels (if you didn't freeze them)

I hope this help since I'm also learning about Faster R-CNN too, also sorry for the bad English.


How to access an embedding table that is too large to fully load into memory?

There are a couple of options:

Incrementally parse JSON with something like ijson. Munge and append each part to tf.Variable.
Reduce the precision of the numbers dtype=tf.float32 to
dtype=tf.bfloat16


Does it make sense to train a CNN as an autoencoder?

Yes, it makes sense to use CNNs with autoencoders or other unsupervised methods. Indeed, different ways of combining CNNs with unsupervised training have been tried for EEG data, including using (convolutional and/or stacked) autoencoders.

Examples:

Deep Feature Learning for EEG Recordings uses convolutional autoencoders with custom constraints to improve generalization across subjects and trials.

EEG-based prediction of driver's cognitive performance by deep convolutional neural network uses convolutional deep belief networks on single electrodes and combines them with fully connected layers.

A novel deep learning approach for classification of EEG motor imagery signals uses fully connected stacked autoencoders on the output of a supervisedly trained (fairly shallow) CNN.

But also purely supervised CNNs have had success on EEG data, see for example:

EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces

Deep learning with convolutional neural networks for brain mapping and decoding of movement-related information from the human EEG (disclosure: I am the first author of this work, more related work see p. 44)

Note that the EEGNet paper shows that also with a smaller number of trials, purely supervised training of their CNN can outperform their baselines (see Figure 3). Also in our experience on a dataset with only 288 training trials, purely supervised CNNs work fine, slightly outperforming a traditional filter bank common spatial patterns baseline.


How does bagging help reduce the variance

High Variance  - Model varies a lot on small changes
High Bias - Model doesn't vary so much but predict quite away from the truth

Let's check a Decision Tree on 5 values -
\begin{array} {|r|r|}
\hline
1 &5 &10 &15 &20\\
\hline
\end{array}
In this tree split,
Value of 9.9 will be 7.5
Value of 10.1 will be 12.5.
Showing a very high variance.

Let's create 4 Random Tree of 3 elements each -
\begin{array} {|r|r|}
\hline
Tree-1 &5 &10 &15\\
\hline
Tree-2 &1 &15 &20\\
\hline
Tree-3 &1 &05 &20\\
\hline
Tree-4 &5 &15 &20\\
\hline
\end{array}
Value of 9.9 = (7.5 + 7.5 + 12.5 + 10)/4 ~ 9.375
Value of 10.1 = (12.5 + 7.5 + 12.5 + 10)/4 ~ 10.625
Variance is reduced a lot.
In bagging, we build multi-hundreds of the Tree(Can build other models too which offers high variance) which results in a large variance reduction


getting error while scrapping Amazon using Selenium and bs4

Most likely the element you are trying to find (I suppose it is a link or a button with class a-last) is not on the page. When the error appears, you should look at your chrome window and see what it is showing you. Check if the element is on the page.

It could be that amazon.com just showed a captcha and therefore all the usual elements disappeared from the screen. For example, if you try to do while True without waiting time on Google it will show you a captcha after several requests.


How to obtain the word vectors optimally

Use nlp.pipe() to process texts in larger batches, which is much faster, especially for a lot of short texts:

for doc in nlp.pipe(sent_list):
    # averaged doc vector
    print(doc.vector)
    # token vectors
    print([token.vector for token in doc])


PAC stands for Probably Approximately Correct.
It was a very common research area in computer science looking for proof of learnability of certain hypothesis sets.

The usual hypothesis sets were not distributions (like in statistics) but more logical formulas like DNF, CNF, DFA, etc.

In order to prove learnability, you need to show that for every sample distribution, for every concept $h$ in the hypothesis class $H$,  for every $\lambda$ (the probability part) and for every $\epsilon$ (the approximately part) you can find an hypothesis $h^*$ such that with probability at least $1- \lambda$ the disagreement between $h$ and $h^*$ will be smaller than $\epsilon$.

This bar for learning is very high, and some of the research proved negative results about many simple hypothesis sets.

When you try to learn to differentiate between cats and dogs, you don't have a mathematical definition of hypothesis set. Most publications run an algorithm on a dataset and show their result. Almost nobody proves that the algorithms are correct (even given some assumptions). I find the lack of proofs quite sad.

The math behind PAC is actually probability. Let's say that you have two cubes (classifier, concept) and you want to know how correlated they are. So, you toss them together $m$ times and you check the level of disagreement, epsilon. The higher the $m$ and the lower the $\epsilon$, the more confident you are that this is due to correlation and not due to error, $\lambda$. Given this intuition, PAC learning usually works in the other direction. Once you state the desired lambda and epsilon, the algorithm provides the number of samples $m$ needed to reach them.


How can i test the performance of a model when the test data contains seen and unseen data

The test of your model should always be done exclusively on unseen data. That is fundamental to asses your model's capacity to generalize and predict observations it has never seen.
If you test it using already seen data, that is like cheating, and overfitting will be 100% guaranteed.

Repeat the test only on unseen data, and check the difference between train and test performances. If differences are big, then you have an overfitting problem. There are several ways to fight overfitting, but they depend on the nature of your model. Could you please provide some information on it?


How to rotate the plot and find minimum point?

Please refer to the original post.

The problem was the np.radians(angle). I corrected that.

Edit: The problem was not the radian conversion but the different scales of the x- and y-axis.
45 degrees rotation was simply not enough.

Let's use a modified version of the code you posted:

def rotate_vector(data, angle):
    # make rotation matrix
    theta = np.radians(angle)
    co = np.cos(theta)
    si = np.sin(theta)
    rotation_matrix = np.array(((co, -si), (si, co)))
    # rotate data vector
    rotated_vector = data.dot(rotation_matrix)
    # return index of elbow
    return rotated_vector

Now, we define a square as test-data:

data = np.array([
    [1, 0], [2, 0], [3, 0], [4, 0], [5, 0],
    [0, 0], [0, 1], [0, 2], [0, 3], [0, 4],
    [5, 1], [5, 2], [5, 3], [5, 4], [5, 5],
    [0, 5], [1, 5], [2, 5], [3, 5], [4, 5],
])

plt.scatter(data[:, 0], data[:, 1])

Plotting the data gives us the following figure:

Now, we rotate that vector using the function from above:

rotated_data = rotate_vector(data, 45)

plt.scatter(rotated_data[:, 0], rotated_data[:, 1])

If we plot the rotated data, we get the following figure:

So it does work. However, if we want to rotate the graph that you posted to find the minimum, we need to remember that the scales of the two axes are different. That means the graph may look like a 45° rotation would be enough while actually, it is not.

I corrected the original post again. Also, I added a method to find an approximate rotation angle to account for the different axis scales.


In batch normalization, shouldn't using DropConnect harm test accuracy?

TL;DR

This is an interesting idea and probably best to be tested with your specific problem; however it is generally understood that you will get better results by not using both Dropout (including DropConnect) and Batch-Norm together, given their overall effects during training. Recent evidence/tests. Having said that, I think it would balance out in the end anyway, as you propose.

More considerations

Looking at the explanation of the base implementation, you could be correct in saying that batch normalisation computations of mean and variance could be influenced - but surely it depends in which order you compute your batch statistics?

There is an ongoing debate (see e.g. this question) as to which order to apply layers such as batch-norm, dropout and activations themselves to the weights. There is the argument (pointed out above) that using both together isn't a good idea. There are some more great points made here.

If you have a look at the three implementations summarised here, you can see that there isn't a big difference between Dropout and DropConnect... it is just about scaling the values correctly, so that the (expected) sum remains consistent.

To each their own!

It almost gets philosphical! I believe it depends on your own personal perspective on why we use each of these layers. If you see Dropout as a means of restricting the network and preventing the co-adaptation of neurons, we are trying to block flow of information through certain paths of the network, so it might make sense to also remove that information from the batch-norm computations.
If you see batch-norm as being a surgically precise method to man-handle the input distribution to a layer, you might want to compute batch-norm stats before the DropConnect sets any weights to zero and use those on the entire outbound batch.


Is it faster and better to train a GAN on just one digit as opposed to the whole mnist dataset?

After some further research and experimentation, I think I am able to answer my own question.

1. Yes it is feasible!

The quickest way to answer my first question was to try it out, which I did. Training a GAN on only a single digit (3 in my example) worked and produced the expected results:

Besides the obvious practical example, I was able to find some further articles discussing GAN which mentioned that reducing the classes in the training sample to reduce variance is advised (e.g. in CIFAR-10 concentrate on frogs OR trucks not both). So while I suspected as much it is still nice to confirm that this approach is valid and feasible.

2. I'm not sure it is faster

The second part of my question was whether such an approach is better (i.e. more accurate) and faster.

I was unable to confirm this because mnist is a very simple example and accuracy is very good, very fast in either case. I learned that speed of model fitting depends on the model complexity and number of epochs, which do not change by focusing on just one digit! So naturally covering all digits individual takes a lot longer than just training one whole model.

It might be the case that for more complex classes/image types more complex models and epochs are needed and that cutting down on variance (i.e. just frogs, no trucks) can help cutting down those two main drivers of computation duration but this was not the case for mnist like images.

The only direct benefit I was able to find is the greater utility of individual models. Due to the clear separation I am able to generate a specific number on demand (by calling the relevant model) instead of generating only a random number from 0-9.


Recently practitioners are representing categorical variables as embeddings for  ML models. I can see a solution to your problem there.
As your problem is having a two-level hierarchy you can consider two embeddings, one set of embeddings for modules, and another set for files. For every document, you can take their combination and pass it as input to the model. In this way, both the module and file-level information will be captured.
References:
https://towardsdatascience.com/deep-embeddings-for-categorical-variables-cat2vec-b05c8ab63ac0


clarification on splitting individual trees in extra trees?

So K would be the number that determines how many attributes to consider for a random split. Then to split, a random attribute from that set will be chosen?

No, a random split is made for each of the randomly selected features, and these $K$ potential splits are scored and the highest scoring one is made.

  If K > 1, in a given set of attributes [1,2,3,4,...,n], is it always a contiguous subset of size K? Or is it K random attributes chosen from those n attributes? And once you choose a random attribute from that subset, it is replaced or left out?

We choose $K$ random attributes, not necessarily (and probably not) contiguous.  I think the original algorithm selects without replacement, and the implementation in sklearn definitely does it without replacement.  I could imagine another implementation deciding to do it with replacement though: still the split point for that feature could be different between the two selections.  (One more caveat is when a selected feature is constant for the population in the current node: these should be left out, and probably it is implementation-specific whether to select a new feature or just have fewer than $K$ in the end.)

  It also says:

  For a given problem, the smaller K is, the stronger the randomization of the trees

  I'm confused as to why this is.

The smaller $K$ is, the fewer features you consider for a split, and so the more the algorithm depends on these random choices (and is less likely to latch on to noisy, spurious correlations).  Compare to a full decision tree model, in which every split point of every feature is considered.


Split Dataset in two Explainable Subsets

Most classical machine learning algorithms assume targets are labeled completely correctly. If you think your labels are noisy, it would make sense to apply techniques to handle that issue.
As far as splitting a dataset into subgroups to improve performance, you are describing any tree-based machine learning algorithm. The goal of any tree-based learning algorithm is learn to split the data into more homogenous groups in the feature space based on target labels.


How to read the labeled enron dataset categories?

Looking at the first link from your post there is an in-depth overview of the different categories. Cat_#_level_1 denotes the top level category (12 in total) whereas Cat_#_level_2 denotes the second level category (up to 19 categories, depending on the level 1 category). E.g. an email with the labels Cat_3_level_1 and Cat_6_level_2 has the label california energy crisis / california politics (category 3.6). You therefore need to combine the two levels to get to the label that belongs to the second level category.


What kind of "vector" is a feature vector in machine learning?

I'm having trouble understanding the use of Vector in machine learning
  to represent a group of features.

In short, I would say that "Features Vector" is just a convenient way to speak about a set of features.

Indeed, for each label 'y' (to be predicted), you need a set of values 'X'. And a very convenient way of representing this is to put the values in a vector, such that when you consider multiple labels, you end up with a matrix, containing one row per label and one column per feature.

In an abstract way, you can definitely think of those vectors belonging to a multiple dimensions space, but (usually) not an Euclidean one. Hence all the math apply, only the interpretation differs !

Hope that helps you.


Sentiment analysis of the target in articles

I'm not sure I completely get the idea, but it looks to me like what you're actually interested in is the sentiment of a word in a particular context: a content word like "car" might not carry a stable sentiment by itself, but its usage in a specific context might.
So I'd suggest a method like this: for any target word you extract either the sentence or a context window, i.e. N words on the left and N words on the right of the target word. Then you could use predefined sentiment analysis tools to extract a sentiment value for this instance. From there you could:

measure the mean sentiment for a word by averaging over the instances
compare the distribution of sentiment or average sentiment for two different words


How to use sklearn train_test_split to stratify data for multi-label classification?

The error you're getting indicates it cannot do a stratified split because one of your classes has only one sample. You need at least two samples of each class in order to put one in the training split and one in the test split. You should examine what your class breakdown is to find the culprit.


How to compute the prediction interval for unseen data?

What you are referring to is a problem of estimation regression model uncertainty.
The uncertainty estimation method depends on the model that you are using.
Take a look at this tutorial, it provides a detailed explanation of what to do when you are using Linear Regression. It also points to the more sophisticated paper, which describes what to do when your model is more complicated.

If you are working with Regression Trees (xgboost) this tutorial provides a code that you can use.


How to use cluster analysis with grouped data so one cluster may only have not more than one item from each group?

You can write your own algorithm.  I drafted something up quickly. It can be significantly optimized.

Let's make some random data

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

n = 9
x = np.random.rand(n,2)
y = np.zeros((n,))
y[n//3:2*n//3] = 1
y[2*n//3::] = 2

plt.scatter(x[:,0], x[:,1], c=y)
plt.show()

Now let's get the interdistance of each of these points.

dists = np.asarray([np.linalg.norm(i-j) for i in x for j in x]).reshape(n,n)

plt.imshow(dists)
plt.show()

We will make a list which will hold the radius of each circle for each point. For each point we will iterate over the other points by their relative nearness. If an associated label is not yet seen, add it to the temporary list. Otherwise, we end the function and take the average of the last distance and the current illegal point.

radii = []

for row in dists:
    labels = []
    dis = []
    for i in np.argsort(row):
        if y[i] not in labels:
            labels.append(y[i])
            dis = row[i]
        else:
            dis = (dis + row[i])/2
            break
    radii.append(dis)

Now we can plot these circles by

fig, ax = plt.subplots(figsize=(10,10))

for ix, i in enumerate(radii):
    circle = plt.Circle((x[ix, 0], x[ix, 1]), i, color='b', fill=False, alpha = 0.5)
    ax.add_artist(circle)

plt.scatter(x[:,0], x[:,1], c=y)
plt.xlim([-0.2,1.4])
plt.ylim([-0.2,1.4])
plt.show()

If we increase the number of datapoints $n$ we get


Imbalanced Data how to use random forest to select important variables?

15K features is a lot of features. I'd bet your 0.01 F1 score is a direct result of having so many features. Without knowing more details about your data it is hard to suggest a more sensible alternative. I'd want it to be closer to 100 features rather than 15,000 though. Are these features extracted from text? N-grams?
Regarding your feature selection problem, the first step is to eliminate any features with no variance (all samples the same), then one can utilis various statistical methods for selecting features to retain those with high variance (sklearn has great libraries for this: Feature selection).
One could also utilize Ranked Guide Iterative Feature Elimination (RGIFE) for this purpose, which uses random forests: RGIFE - Ranked Guided Iterative Feature Elimination
P.S. Your class ratio is very skewed as well. Once you're all settled with a more reasonable dataset in feature space I'd consider either undersampling the larger class, oversampling the smaller class or a combination of both, and keep an eye on the resulting confusion matrices when performing your evaluation.


Data augmentation: ImageDataGenerator vs openCV

Keras' ImageDataGenerator doesn't offer much support by itself for data augmentation. However it has a parameter called preprocessing_function which allows you to use custom augmentors with it.

I personally use imgaug which offers virtually any augmentation you can think of and works well with ImageDataGenerator like I said.


For each state of continuous HMM, an emission probability distribution is defined. It can be a Gaussian distribution or a Gaussian Mixture Model (GMM). After computing the state variable, you can get the emission probability distribution of observed variables using corresponding state distribution. Note that there is no deterministic value for observed variables and just a probability distribution conditioned on the predicted state variable. In hmmlearn, you can get the means, covars and priors of the GMM using means_, covars_ and weights_ attributes of the model, respectively.


How word2vec can handle unseen / new words to bypass this for new classifications?

One way to do this is to use context information to represent each word along with the w2v vector. You can choose to represent this information in any way you like: add another 600 dimensions (100D w2v vectors for 3 left and 3 right context words), just another 100D as the sum of context vectors or any other fixed length representation for your context.

When you're training, you can use a version of 'word dropout' that will utilize this information. 20% of the time, set your w2v vectors to zero, forcing your classifier to use the context dimensions to represent the word.

When you encounter a new word, the hope is, the classifier learned to use the context information as well as it learned to use the w2v information.


How (and why) are random forests able to represent both linear and non linear data

Decision Trees are able to create both linear and nonlinear boundaries and so are Random Forests.

This is because of how they cluster the data based on nested "if-else" statements. These statements draw vertical/horizontal lines between the samples and cluster them in rectangles. Consecutively, rectangles of the same class can be far away from each other (with other class rectangles in between) but still they belong to the same class. This is how nonlinear relations are modeled.


What is the difference between and Embedding Layer and an Autoencoder?

Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data)

Autoencoder is a type of neural network where the inputs and outputs are the same but in the hidden layer the dimensionality is reduced in order to get a more dense representation of the data.

Word2vec contains only 1 hidden layer but the inputs are the neighborhood words and the output is the word itself (or the other way around). So it cannot be an autoencoder cause the inputs and outputs are different.

Embedding layer is only a "simple" layer in a neural network. You can imagine it as a dictionary where a category (i.e word) is represented as a vector (list of numbers). The value of the vectors are defined by backpropagating the errors of the network.


In multiple linear regression why is it best to use an $F$-statistic when evaluating predictors?

The example given in your textbook proposes a multiple linear regression with 100 predictors, all of which have a "true" regression coefficient of 0. In other words, your independent variables have no statistical association with your dependent variable.

When you calculate the $p$-value of an individual coefficient, you're looking at the magnitude of the coefficient, the standard error of the coefficient, making some distributional assumptions about it, and asking the following question: "what is the probability of seeing a coefficient value this extreme if the true value is actually zero?"

If our distributional assumptions are correct, any given coefficient with a true value of 0 will report a <0.05 $p$-value approximately 5% of the time. That's not so much of a problem if we only have one predictor, but by the law of large numbers, if we have lots of predictors, we'd expect 5% of them to report a $p$-value this low. This makes the $p$-values in high-dimensional regressions hard to interpret.

The $F$-test is different. Instead of evaluating every single coefficient for statistical significance, it applies a single test to the entire regression. So instead of having 100 chances to throw up an erroneous $p$-value, it only has one chance. This makes the $F$-test useful for evaluating whether or not there is a regression effect for high-dimensional regressions.


You can use pandas groupby function to group your rows according to  accounts and then perform your desired operation on them.


how to sort a multi-level pandas data-frame by a particular column?

TO BE EDITTED
There would definitely be more optimal ways for addressing this problem. But this is a simple approach.
import pandas as pd
df = pd.DataFrame({'first': ['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux'],
                    'second': ['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two'],
                    'value': [0.361041, 0.476720, 0.565781, 0.848519, 0.405524, 0.882497, 0.488229, 0.303862]})

    first   second  value
    0   bar one 0.361041
    1   bar two 0.476720
    2   baz one 0.565781
    3   baz two 0.848519
    4   foo one 0.405524
    5   foo two 0.882497
    6   qux one 0.488229
    7   qux two 0.303862

import itertools
#storing unique values of the first column.
unique = list(df['first'].unique())
#unique
#The following line, gives the row indices of the dataframe with specific value on the index ('first') column.
indices =[df.index[df['first'] == unique[i]].tolist() for i in range(len(unique))]
#Then this line sorts each pair (it can be more than 2) of the elements from the value column.
sorted_values = []
for [i,j] in indices:
    sorted_values.append(sorted([df['value'].iloc[i],df['value'].iloc[j]]))

#The following line returns the row index of a specific value of 'value' column.
#df['value']
[df['value'] == 0.361041].index.tolist()
#This ravels the sorted_list and gives the goal order of the values in 'value' column
values = []
values = list(itertools.chain(*sorted_values))
#values

#output
[0.361041, 0.47672, 0.565781, 0.848519, 0.405524, 0.882497, 0.303862, 0.488229]

Now that we have the actual order of elements in the value column, we can rebuild the whole dataframe based on this column.
However, I've done ascending ordering instead of your desired descending one.


How to add more theta parameters into my logistic regression?

Your pass your initial_theta into Logistic_Regression where it defines how the cost function and its derivative are evaluated. Just make initial_theta the same width as X. If you want your code to be fool-proof, check variable sizes within your function like

def Logistic_Regression(X,Y,alpha,theta,num_iters):
    assert len(theta) == X.shape[1], 'theta should have one coefficient per each column of X'
    ....

A more convenient solution would be to define initial_theta inside the function, based on X:

def Logistic_Regression(X,Y,alpha,num_iters):
    theta = np.zeros(X.shape[1])
    ....

In this case, you can be sure that the shape of initial $\theta$ is correct.


Algorithm for gesture classification in a wearable

It might be really useful to provide a bit more details regarding the actual data, produced by your sensor. Can you provide a simple example for any simple gesture?

Anyway, let's try to address the problem in the following way:

sensor can be represented as a function f(t)
range of the function will be {1,2,3,..100}

So we can draw something like the following (can we?):

You can extract various high-level features out of that data, but I would suggest to go with inflection points as the most straightforward option:

Those points, basically, define a pattern. I'll use a figure from the different answer to illustrate the idea. For example, the naive pattern below is used to identify a circle.

In your case situation is a bit simpler, because sensor output is the only data you can work with. Naturally, you can combine multiple sensors in order to define a bit more complex interaction scenarios:

Also you can take into account f'(t) in order to distinguish between slow and fast movements, etc.

Update

Thanks for the data. It does look like these three gestures looks pretty much the same from the data perspective (see visualization below).

It really looks like several features are required to identify the gesture.


Speed decay proof for L2 regularization and non-normalizied weight initiation

Ok, I have found the source of confusion, thanks to @Emre for the insight.
The task talks about epochs of training, while I was trying to solve one batch update. I think this confusion is statistically possible enough to validate existence of this question :)


That's correct, you need a NVIDIA GPU compatible with CUDA 8, 9 or 10: https://pytorch.org/get-started/locally/

NVIDIA has a list of compatible cards here: https://developer.nvidia.com/cuda-gpus#compute

Alternatively you could work on a GPU equipped cloud instance (or install pytorch without GPU).


Can neural networks be considered a method for data analysis?

There's a lot to say on this topic. First of all, outside of academia, there is an unofficial classification (more slanted toward business types) of machine learning systems (and thus neural networks), which splits them into three categories in order of increasing utility/autonomy: descriptive, predictive and prescriptive.
A descriptive system summarizes data, a predictive one predicts future trends (or anything else not explicitly present in the data) and a prescriptive one recommends courses of action. Neural networks certainly do more than summarize data, so most of them fall in the predictive category, but with additional techniques they can and do reach the prescriptive category.
To support decision making, as you mentioned, there are a number of desirable properties and capabilities a system should have, but the one I will focus on since you emphasized it is model interpretability, which will bring us further down the line to neural networks as methods of data analysis.
Neural networks, as is, are eminently uninterpretable, black-box models: often, this leads people to adopt simpler models, such as decision trees or random forests. However, this isn't a fact of life: a lot of useful information can be extracted from what a neural network has learned, information that humans can understand.
In the case of networks for simple tabular data (e.g. house prices), [https://eli5.readthedocs.io/en/latest/blackbox/index.html] offers permutation important to estimate the predictive power of features (e.g. surface area matters, color of walls not so much). It also offers a more complex method, LIME, which builds a surrogate linear model of the neural network (which is much more easily interpreted). LIME can also be used for CNNs. Other interpretability methods include partial plots and SHAP.
Since we're on the topic of CNNs, other methods tailored specifically for them exist, such as the (basically industry standard by now) Grad-CAM, which highlights the contribution of parts of an image to the output of a CNN.
If you consult literature review papers on explainable AI (XAI), you will notice that one of the main goals is knowledge extraction from trained models.
On top of all that, in my opinion, plain neural networks, without any fancy explanation method applied, can be considered data analysis methods as is. Image segmentation, semantic segmentation, autoencoding, for instance, sound like the definition you cited for data analysis, and they are all done with neural networks.


Decision Tree - Preprocessing for very sparse features

There are many feature engineering options for sparse categorical data. A couple of common options:

Remove features - Drop columns that are sparse
Feature reduction - One example is singular value decomposition (SVD)
Reduce number of categories - Using domain knowledge, group together related low frequency categories
Use sparse representation - only stores nonzero elements


Cross Entropy vs Entropy (Decision Tree)

Are both measures usable? Is only cross-entropy used?

They both could  be used for this special case. However, I personally prefer "entropy" because it requires less mental gymnastics.

Let's first review the definitions. The most agreed upon and consistent use of entropy and cross-entropy is that entropy is a function of only one distribution, i.e. $-\sum_x P(x)\mbox{log}P(x)$, and cross-entropy is a function of two distributions, i.e. $-\sum_x P(x)\mbox{log}Q(x)$ (integral for continuous $x$).

The formula used in The Elements of Statistical Learning [Page 308, 9.2.3 Classification Trees] can be written as
$$-\sum_k P_{m}(k) \text{log}P_{m}(k)$$
where $P_{m}(k)$ is the ratio of class $k$ in node $m$. This could be interpreted as a function of only one (data) distribution, i.e. an entropy, that measures the impurity of node $m$. Nonetheless, it can also be interpreted as a cross-entropy between data distribution and model estimation (based on @DrewN nice explanation), i.e.
$$-\sum_k P^{\text{data}}_{m}(k) \text{log}P^{\text{model}}_{m}(k)$$
where we, hypothetically, set the model estimation to match the data distribution in node $m$, i.e.
$$P^{\text{model}}_{m}(k)=P^{\text{data}}_{m}(k) = P_{m}(k)$$
to minimize the cross-entropy.
Accordingly, cross-entropy would be the same as both data entropy and model entropy in value, i.e.
$$\begin{align*}
\overbrace{-\sum_k P^{\text{data}}_{m}(k) \text{log}P^{\text{model}}_{m}(k)}^{\text{data-model cross entropy } H(P^{\text{data}}_{m}, P^{\text{model}}_{m})}&=\overbrace{-\sum_k P^{\text{data}}_{m}(k) \text{log}P^{\text{data}}_{m}(k)}^{\text{data entropy } H(P^{\text{data}}_{m})}\\
&=\overbrace{-\sum_k P^{\text{model}}_{m}(k) \text{log}P^{\text{model}}_{m}(k)}^{\text{model entropy } H(P^{\text{model}}_{m})}
\end{align*}$$

but it is different in meaning and rightfully has a different name.  I say "hypothetically" because in practice, classifier only chooses the class with maximum probability, i.e.
$$P^{\text{classifier}}_{m}(k)=\left\{\begin{matrix}
 1 & k=\underset{k'}{\text{argmax }}P_m(k')\\
 0 & \text{o.w.}
\end{matrix}\right.$$

From another perspective, when cross-entropy is equal to entropy, it means KL divergence is zero
$$\text{KL}(P^{\text{data}}_{m} \parallel P^{\text{model}}_{m}) = H(P^{\text{data}}_{m},P^{\text{model}}_{m}) - H(P^{\text{data}}_{m})=0$$

All in all, we can still confidently use "entropy" for decision trees when we talk about node splitting and node impurity. For example, a split occurs when entropy of class distribution in parent node is higher than the weighted-average of class entropies in left and right children (i.e. positive information gain).

As an extra note, cross-entropy is mostly used as a loss function to bring one distribution (e.g. model estimation) closer to another one (e.g. true distribution). A well-known example is classification cross-entropy (my answer). Also, KL-divergence (cross-entropy minus entropy) is basically used for the same reason.


Python library to implement Hidden Markov Models

For another alternative approach, you can take a look at the PyMC library.
There is a good gist created by Fonnesbeck which walks you through the HMM creation.
And if you become really eager about the PyMC, there is an awesome open-source book about Bayesian Modeling. It does not explicitly describe Hidden Markov Processes, but it gives a very good tutorial on the library itself with plenty of examples.


How to download BAIR action free robot pushing dataset?

BAIR dataset can be downloaded here
https://sites.google.com/berkeley.edu/robotic-interaction-datasets

Additionally, here is the code to extract data from the dataset

import datetime
import os
import time

import cv2
import numpy as np
import skvideo.io
import tensorflow as tf
from PIL import Image
from tensorflow.python.platform import gfile

def get_next_video_data(data_dir):
    filenames = gfile.Glob(os.path.join(data_dir, '*'))
    if not filenames:
        raise RuntimeError('No data files found.')

    for f in filenames:
        k = 0
        for serialized_example in tf.python_io.tf_record_iterator(f):
            example = tf.train.Example()
            example.ParseFromString(serialized_example)
            # print(example)        # To know what all features are present

            actions = np.empty((0, 4), dtype='float')
            endeffector_positions = np.empty((0, 3), dtype='float')
            frames_aux1 = []
            frames_main = []
            i = 0
            while True:
                action_name = str(i) + '/action'
                action_value = np.array(example.features.feature[action_name].float_list.value)
                if action_value.shape == (0,):      # End of frames/data
                    break
                actions = np.vstack((actions, action_value))

                endeffector_pos_name = str(i) + '/endeffector_pos'
                endeffector_pos_value = list(example.features.feature[endeffector_pos_name].float_list.value)
                endeffector_positions = np.vstack((endeffector_positions, endeffector_pos_value))

                aux1_image_name = str(i) + '/image_aux1/encoded'
                aux1_byte_str = example.features.feature[aux1_image_name].bytes_list.value[0]
                aux1_img = Image.frombytes('RGB', (64, 64), aux1_byte_str)
                aux1_arr = np.array(aux1_img.getdata()).reshape((aux1_img.size[1], aux1_img.size[0], 3))
                frames_aux1.append(aux1_arr.reshape(1, 64, 64, 3))

                main_image_name = str(i) + '/image_main/encoded'
                main_byte_str = example.features.feature[main_image_name].bytes_list.value[0]
                main_img = Image.frombytes('RGB', (64, 64), main_byte_str)
                main_arr = np.array(main_img.getdata()).reshape((main_img.size[1], main_img.size[0], 3))
                frames_main.append(main_arr.reshape(1, 64, 64, 3))
                i += 1

            np_frames_aux1 = np.concatenate(frames_aux1, axis=0)
            np_frames_main = np.concatenate(frames_main, axis=0)
            yield f, k, actions, endeffector_positions, np_frames_aux1, np_frames_main
            k = k + 1

def extract_data(data_dir, output_dir, frame_rate):
    """
    Extracts data in tfrecord format to gifs, frames and text files
    :param data_dir:
    :param output_dir:
    :param frame_rate:
    :return:
    """
    if os.path.exists(output_dir):
        if os.listdir(output_dir):
            raise RuntimeError('Directory not empty: {0}'.format(output_dir))
    else:
        os.makedirs(output_dir)

    seq_generator = get_next_video_data(data_dir)
    while True:
        try:
            _, k, actions, endeff_pos, aux1_frames, main_frames = next(seq_generator)
        except StopIteration:
            break
        video_out_dir = os.path.join(output_dir, '{0:03}'.format(k))
        os.makedirs(video_out_dir)

        # noinspection PyTypeChecker
        np.savetxt(os.path.join(video_out_dir, 'actions.csv'), actions, delimiter=',')
        # noinspection PyTypeChecker
        np.savetxt(os.path.join(video_out_dir, 'endeffector_positions.csv'), endeff_pos, delimiter=',')
        skvideo.io.vwrite(os.path.join(video_out_dir, 'aux1.gif'), aux1_frames, inputdict={'-r': str(frame_rate)})
        skvideo.io.vwrite(os.path.join(video_out_dir, 'main.gif'), main_frames, inputdict={'-r': str(frame_rate)})
        skvideo.io.vwrite(os.path.join(video_out_dir, 'aux1.mp4'), aux1_frames, inputdict={'-r': str(frame_rate)})
        skvideo.io.vwrite(os.path.join(video_out_dir, 'main.mp4'), main_frames, inputdict={'-r': str(frame_rate)})

        # Save frames
        aux1_folder_path = os.path.join(video_out_dir, 'aux1_frames')
        os.makedirs(aux1_folder_path)
        for i, frame in enumerate(aux1_frames):
            filepath = os.path.join(aux1_folder_path, 'frame_{0:03}.bmp'.format(i))
            cv2.imwrite(filepath, cv2.cvtColor(frame.astype('uint8'), cv2.COLOR_RGB2BGR))
        main_folder_path = os.path.join(video_out_dir, 'main_frames')
        os.makedirs(main_folder_path)
        for i, frame in enumerate(main_frames):
            filepath = os.path.join(main_folder_path, 'frame_{0:03}.bmp'.format(i))
            cv2.imwrite(filepath, cv2.cvtColor(frame.astype('uint8'), cv2.COLOR_RGB2BGR))
        print('Saved video: {0:03}'.format(k))

def main():
    data_dir = '../softmotion30_44k/test'
    output_dir = '../ExtractedData/test'
    frame_rate = 4
    extract_data(data_dir, output_dir, frame_rate)
    return

if __name__ == '__main__':
    print('Program started at ' + datetime.datetime.now().strftime('%d/%m/%Y %I:%M:%S %p'))
    start_time = time.time()
    main()
    end_time = time.time()
    print('Program ended at ' + datetime.datetime.now().strftime('%d/%m/%Y %I:%M:%S %p'))
    print('Execution time: ' + str(datetime.timedelta(seconds=end_time - start_time)))

References:
https://github.com/edenton/svg/blob/master/data/convert_bair.py


Can Anyone Explain this code piece by piece?

def pd_centers(**featuresUsed**, centers):

inputs of the function are featuresUsed that are the features(column names) you already have and centers.

colNames = list(**featuresUsed**)

names of columns (featuresUsed) are saved here as a list named colNames

colNames.append('prediction')

Here a new column name rather than those already existing is added to column names (features) named prediction

# Zip with a column called 'prediction' (index)
Z = [np.append(A, index) for index, A in enumerate(centers)]

A list of elements of centers with their indices are saved as Z
For instance, if we have centers equal to ['CA', 'TEX', 'FLO', 'ORE']
 then Z would become and finally these indices would become the values of prediction column.

[array(['CA', '0'], dtype='<U21'),
 array(['TEX', '1'], dtype='<U21'),
 array(['FLO', '2'], dtype='<U21'),
 array(['ORE', '3'], dtype='<U21')]

# Convert to a pandas data frame for plotting
P = pd.DataFrame(Z, columns=colNames) #save Z as a dataframe with columns colNames
P['prediction'] = P['prediction'].astype(int) #converting the values in Prediction column to int type
return P

here Z is converted to dataframe named P and the elements in the prediction column is converted to int type and P as the ultimate dataframe is returned. this prediction column is the cluster number.

if centers is a list of tuples, and it should be I assume, then instead of increasing indices, then the corresponding elements in each tuple are saved as A and index.


Overcome memory limitation when downloading from database into Orange

You can use "download data to local memory" checkbox in SQL Table widget, it should allow you to work with up to 1000000 rows.


Create Custom Python Model in Azure ML Studio?

https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/python-language-modules This article describes the modules provided in Azure Machine Learning Studio that support running custom Python code in a machine learning experiment.


Logistic regression is just a generalized linear model, so there is a linear regression lurking in here whose errors are supposed to be normally distributed, and from which you can get confidence intervals in the usual way.

That is you have a linear predictor (the -theta' * x part), transformed by the inverse of a link function (the logistic function). You can transform prediction +/- confidence interval using this inverse link to get your +/- in probability terms. It won't be symmetric.

Nice writeup of how to do this in R: https://stackoverflow.com/a/14424417/64174


Transformer seq2seq model and loading embeddings from XLM-RoBERTa

It is indeed possible, but the question is if it is a good idea.

FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R.

Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as they show in their blog post.

Now, why I don't think it is a good idea. Recent papers show that:

Improving MT with pre-trained Transformers requires a lot of effort and it is still questionable whether it pays off, see Incorporating BERT into Neural Machine Translation
.
The encoder of MT capture different structures: see [What does BERT look at?](What does BERT look at?) and Analyzing Multi-Head Self-Attention:
Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned.
In an end-to-end trained translation system, a large part of the translation process is happening already in the encoder, see Analyzing Word Translation of Transformer Layers.

I assume you only have little training data and therefore you hope that pre-trained representation might help, but I believe you should put more effort in optimizing an end-to-end model to work in low-resource setup (have a look at Revisiting Low-Resource Neural Machine Translation: A Case Study) and data augmentation technique like iterative back-translation that it used in unsupervised machine translation.


Why is patsy used to prepare data for logistic regression?

Mostly convenience. In this particular case, it takes care of one-hot encoding categorical variables e.g. C(occupation).

patsy takes care of other things under the hood as well, like dropping rows with missing values and adding a constant intercept variable.