What is the best option to test the data set of images using weka?

Data Preparation

Weka uses the arff file type.

A modified example arff file content from the previous link:

% 1. Title: Iris Plants Database
% 2. Creator: R.A. Fisher, Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
@RELATION iris
@ATTRIBUTE petallength  NUMERIC
@ATTRIBUTE petalwidth   NUMERIC
@ATTRIBUTE class        {Iris-setosa,Iris-versicolor,Iris-virginica}
@DATA
1.4,0.2,Iris-setosa
1.4,0.2,Iris-virginica
1.3,0.2,Iris-setosa
1.5,0.2,Iris-versicolor
1.4,0.2,Iris-setosa
1.7,0.4,Iris-setosa

The above example can be broken down into sections:

 Comments

Lines beginning with "%" are comments and are not read by the WEKA data readers

@RELATION

dataset name (use quotes around the name if there are spaces)

@ATTRIBUTE

each attribute defines a feature of one of the following types:

NUMERIC: real number
NOMINAL: discrete value from a predefined list (see example @ATTRIBUTE class feature)
STRING: a string where quotes must be used if spaces are included
DATE: @ATTRIBUTE `name` date [`date-format`] where the default date-format is "yyyy-MM-dd'T'HH:mm:ss"

@DATA

Delimiter specifying the start of the CSV data

Weka Explorer

After creating your arff file, you have a few options when using the Weka explorer:

Load entire dataset into Weka explorer and use the built-in k-fold cross validation option which randomly partitions your dataset and automatically performs CV for your given a classifier
Load entire dataset into Weka explorer and use the built-in train/test split option with a user-specified percentage to randomly partition your dataset.
Manually partition your arff file into a train file and test file and use the supplied test-set option in the Weka explorer. To manually partition your arff file, simply copy your original arff file and and keep non-intersecting sets of CSV rows (i.e. delete instances from one file and retain those instances in the other).

If this is for educational purposes, it might be easiest to manually partition your dataset so that you know which images are in training and which are testing. If this is for research, of course k-fold CV is preferred.

If you are using the Weka Java library rather than the desktop application, there are plenty of online resources that explain the library architecture: Waikato Github io site


If you want to use arbitrary length input and outputs, there are two common options: train a single model for each length or use a recurrent neural network (RNN) architecture. A non-RNN neural network cannot handle arbitrary length because the shape of the weights depend on the length of the input. From your example, it seems like you want to keep different copies of weights based on the length. You could do that, but in theory, it is the same as training separate models. The weights from one length will only be updated by inputs of the same length, so why not save yourself the hassle and have separate models?

Although RNNs are typically used for natural language processing and understanding, it does not mean they cannot be used to solve other deep learning problems. Specifically for your problem, you should use a many-to-many architecture.

Source: http://karpathy.github.io/2015/05/21/rnn-effectiveness/

In Keras, you can use the LSTM layer and set the parameter return_sequences to True, which will return an output the same length as your input sequence. Documentation here: https://keras.io/layers/recurrent/#lstm.


How to predict tf-Idf vector with Neural Network?

I hope you do understand, that trying to predict some abstract TF-IDF vector based on read article you are trying to teach a neural network to produce a new article for a given user. It would be a major step in artificial intelligence if achieved, so keep us posted on your progress.

However, if you would think of alternative approaches, I would suggest following steps to make your neural network to work:

You need a classification/clustering model for your articles (which can be based on your TF*IDF, or on some tags assigned to articles). The more classes you could squeeze from your model, the better would your resulting recommender be. On that step, you should be at least able to provide user with articles similar by topic.
You will need at least two-hop data set of user reading current article and hopping to the next one which would be used as a training set for actual predictor. Here, you will train a neural network (or any other model) which will try to predict a next-article class (from step one) based on current article - that would be a recommender system.

Of course, you may, given you have data from (2), train a neural network that would answer the question of "If user reads article A, will the next article contain word B", but that would require you to have the size of output variables of your TF*IDF matrix (for any reasonable dataset that would be at least 20-30k words unless you work with books for toddlers) which makes your model overcomplicated (that would be, actually, a TF*IDF matrix size number of models for every word).


What does it mean that classes are mutually exlcusive but soft-labels are accepeted?

What does it mean that classes are mutually exclusive but soft-labels are accepted?

As it can be seen from here, tf.nn.softmax produces just the result of applying the softmax function to an input tensor. The softmax "squishes" the inputs so that sum(input) = 1; it's a way of normalizing. The shape of the output of a softmax is the same as the input - it just normalizes the values. The outputs of softmax can be interpreted as probabilities. In contrast, tf.nn.softmax_cross_entropy_with_logits computes the cross-entropy of the result after applying the softmax function (but it does it all together in a more mathematically careful way). It's mathematically careful due to the fact that $y_i$ in $log(y_i)$ can be zero. As you can read from here, A randomly-initialized softmax layer is extremely unlikely to output an exact 0 in any class. But it is possible, so worth allowing for it. First, don't evaluate log(yi) for any y′i=0, because the negative classes always contribute 0 to the error. Second, in the practical code you can limit the value to something like $log( max( y_predict, 1e-15 ) )$ for numerical stability - in many cases, it is not required, but this is sensible defensive programming. I encourage you to take a look at the answers to this question.

  NOTE: While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

The first sentence means that your classifier may not be able to classify the labels exactly as they are, one-hot-encoded. What it does is to find the chance that how likely it is that the input belongs to each class. And this won't make a problem if it does not have mutually exclusive output vector as the algorithm implies. It just needs a vector that the sum of its entries is equal to one. If they are not, the computation of the gradient will be incorrect. I guess this line is added to announce that the outputs of this differentiable component will not be one-hot-encoded and it's because of the nature of these nets. The first layers of convolutional networks are like bases vectors and each instance of classes share these bases. All inputs are made up of these bases.

  I'm wondering if, in a multi-class exclusive case where the only constraint on the labels is that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...

Basically, for multi-label classification, your input may have different labels. Consequently, your classes won't be mutually exclusive anymore. Moreover, in those cases, we don't use softmax as the last layer. You have to have sigmoid for each output and your cross-entropy cost function will be slightly different 1. Consequently, the output of each entry should be a valid probability, that is why sigmoid is used, and the label vectors for such tasks is a not one-hot-encoded anymore. different classes have different entries and based on the existence of instances of each class, the corresponding entry should be one.


What techniques are used to understand call patterns?

When you said "a broader topic", did you mean what algorithms to use to examine the event log with the goal of reducing future calls from the same customer on the same topic? In other words, a customer may call for help for a different topic. If a customer calls in for the same topic repetitively, something can be improved.

You may get ideas from a Coursera class Processing Mining since the issue you're solving is similar to the example of a spaghetti process in lecture 6.7:

"Spaghetti process describing the diagnosis and treatment of 2765 patients in a Dutch hospital. The process model was constructed based on an event log containing 114,592 events. There are 619 different activities (taking event types into account) executed by 266 different individuals (doctors, nurses, etc.)."

By the way, you can click the drop down menu under "Sessions", choose "April 1, 2015 to May 19, 2015" then you can register and view the lectures. On the right of each lecture, there are some icons. The first icon is to download slides for the lecture. You may find reading the slides is faster than listening to the lecture.

Suppose a customer called for installation of software release 1.5 then called a day later for running the new features of software release 1.5. Are these two issues logged as two of the 200 call categories? If so, we can use a time period (say one week) to judge whether it is about the same topic. Within a short time period, a customer is likely to work on the same topic, especially with the same key words such as "software release 1.5". We can coach the call center employees on solving possible follow up questions by saying something like "now that we finished installation, let me show you a couple of new features. It'll save you time." This will reduce the number of calls on the same topic from the same customer.


Predict probability when model was trained in balanced dataset

Daniel,
What you did goes under the name of "oversampling." There is a sample of some "real" population, and you replace it with a sample from a "manufactured" population. The problem that makes sense in application is the estimation of

$$P_r(Y=1|X) = \text{probability of response=1 in the $\mathbf{real}$ population given the predictor $X$} $$

but by using an oversample you are estimating
$$P_m(Y=1|X) = \text{probability of response=1 in the $\mathbf{manufactured}$ population given the predictor $X$}$$

The two probabilities are related. I'll worked the details. I'll pretend the predicor $X$ is discrete. If $X$ takes numerical values one has to replace some probabilities by probability densities.
$$\dots\dots\dots$$

To simplify the notation, let $\pi_1 = P_r(Y=1)$ and $\mu_1 = P_m(Y=1)$ be the probabilities of response in the real and manufactured populations, let
$$
L_r =
\frac{P_r(X=x|Y=1)}{P_r(X=x|Y=0)} = \frac{\frac{P_r(Y=1|X=x)}{P_r(Y=0|X=x)}}{\frac{\pi_1}{1-\pi_1} }$$
be the odds ratio of $Y=1$, i.e.: the ratio of the odds among cases with $X=x$ and the odds in the general $\mathbf{real}$ population. Finally, let $L_m$ be the corresponding ratio in the $\mathbf{manufactured}$ population.

By Bayes' Theorem:
$$ P_r(Y=1|X=x) = \frac{P_r(Y=1,X=x)}{P_r(X=x)} =   \\
=\frac{P_r(X=x|Y=1)\space \pi_1}{P_r(X=x|Y=1)\space\pi_1 + P_r(X=x|Y=0)\space (1 - \pi_1)} = \\
=\frac{L_r\space \pi_1}{L_r\space\pi_1 + \space (1 - \pi_1)} \tag{1}
$$
In a similar way, we get an analogous result for the manufactured population:
$$
P_m(Y=1|X=x) = \frac{L_m\space \mu_1}{L_m\space\mu_1 + \space (1 - \mu_1)} \tag{2}
$$
Since the manufactured sample is a random sample, stratified by $Y$, the conditional distribution of X within responders is the same as in the real population. Same as for non responders, i.e.:
$$
 P_r(X=x|Y=j) = P_r(X=x|Y=j) $$
for $j=0,1.$ If the sample stratified by values of Y was anything other than random sample these would not be true.
It follows that $\boxed{ L_r = L_m }$. Next we solve for $L_m$ in terms of $P_m(Y=1|X)$ from (2) and replace in (1).

$$\dots\dots\dots$$
$\mathbf{Digression}$:
Here is my easy way to carry the steps, without mess. Two non-zero vectors $\mathbf{v_1}$, $\mathbf{v_2}$ are parallel iff there is $\lambda \ne 0$ such that $\mathbf{v_1}=\lambda \mathbf{v_2}.$ Below I will use this idea, and I will not care about the exact value of $\lambda$, so I will be using "$\lambda$" as a short-hand for "$\mathbf{\text{some non-zero mumber}}$."
$\mathbf{\text{End Digression}}$:
$$\dots\dots\dots$$

The easy way to solve is to observed that for non-zero messy values of $\lambda$ (not the same in each occurrence!) one has:

$$
\begin{bmatrix}
 P_r(Y=1|X) \\
 1 \\
\end{bmatrix} = \lambda
\begin{bmatrix}
       \pi_1 &0\\
      \pi_1      &1-\pi_1
\end{bmatrix}
\begin{bmatrix}
L_r\\
1
\end{bmatrix} ,
$$
and
$$
\begin{bmatrix}
 P_m(Y=1|X) \\
 1 \\
\end{bmatrix} = \lambda
\begin{bmatrix}
      \mu_1 &0 \\
      \mu_1      &1-\mu_1
\end{bmatrix}
\begin{bmatrix}
L_m\\
1
\end{bmatrix}
. $$

Therefore,

$$
\begin{bmatrix}
L_m\\
1
\end{bmatrix}
 = \lambda
\begin{bmatrix}
      \mu_1  &0 \\
      \mu_1      &1-\mu_1
\end{bmatrix}^{-1}
\begin{bmatrix}
 P_m(Y=1|X) \\
 1 \\
\end{bmatrix} ,
$$

and so (remember that here $\lambda$ stands for "some non-zero number")

$ \space
\begin{bmatrix}
  P_r \\
   1
\end{bmatrix} = \lambda
\begin{bmatrix}
       \pi_1 &0 \\
      \pi_1      &1-\pi_1
\end{bmatrix}
\begin{bmatrix}
L_r \\
1
\end{bmatrix} = \\
\text{              }=\lambda
\begin{bmatrix}
      \pi_1      &0    \\
      \pi_1      &1-\pi_1
\end{bmatrix}
\begin{bmatrix}
       \mu_1     &0     \\
      \mu_1      &1-\mu_1
\end{bmatrix}^{-1}
\begin{bmatrix}
 P_m \\
 1 \\
\end{bmatrix} = \\
\text{              }=\lambda
\begin{bmatrix}
      \pi_1 (1- \mu_1)        &0 \\
      \pi_1  - \mu_1     & \mu_1 (1- \pi_1)
\end{bmatrix}
\begin{bmatrix}
 P_m \\
 1 \\
\end{bmatrix} =
\lambda
\begin{bmatrix}
      \pi_1  (1-\mu_1)  P_m \\
      (\pi_1  - \mu_1) \; P_m   +  \mu_1 (1- \pi_1)
\end{bmatrix}.
$
Thus,
$$ P_r = \frac{\pi_1 (1- \mu_1) P_m}{(\pi_1 - \mu_1) \; P_m   +  \mu_1 (1- \pi_1) } $$

$$\dots\dots\dots$$
Example: Let's work the details of a Binomial model,
$$P_m(Y=1|X) = \frac{e^{\beta_0 + \beta X}}{1+e^{\beta_0 + \beta X}} $$
or in the "where $\lambda$ is some non-zero scalar" notation (I would not had digressed before if I did not had ulterior motive.. :) ):

$$
 \begin{bmatrix}
 P_m \\
 1 \\
\end{bmatrix} =
\lambda
\begin{bmatrix}
e^{\beta_0 + \beta X} \\
1 + e^{\beta_0 + \beta X}
\end{bmatrix}
$$

What is the implied model in the real population?

$ \space
\begin{bmatrix}
     P_r \\
      1
\end{bmatrix} =
\lambda
\begin{bmatrix}
      \pi_1 (1- \mu_1)   &0 \\
      \pi_1  - \mu_1     &\mu_1 (1- \pi_1)
\end{bmatrix}
\begin{bmatrix}
  P_m \\
   1
\end{bmatrix} =
$

$\;$
$
= \lambda
\begin{bmatrix}
      \pi_1 (1- \mu_1)   &0 \\
      \pi_1  - \mu_1     &\mu_1 (1- \pi_1)
\end{bmatrix}
\begin{bmatrix}
e^{\beta_0 + \beta X} \\
1 + e^{\beta_0 + \beta X}
\end{bmatrix}=
$

$\;$
$
= \lambda
\begin{bmatrix}
      \pi_1 (1- \mu_1)   e^{\beta_0 + \beta X} \\
      \pi_1 (1- \mu_1)   e^{\beta_0 + \beta X} + \mu_1 (1- \pi_1)
\end{bmatrix} =
\lambda
\begin{bmatrix}
\frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)} e^{\beta_0 + \beta X} \\
1 + \frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)} e^{\beta_0 + \beta X}
\end{bmatrix}
.$

If we let $\tau = \ln(\frac{\pi_1 (1- \mu_1)}{\mu_1 (1- \pi_1)})$, we can absorve this constant in the exponents to get:
$$
\begin{bmatrix}
     P_r \\
      1
\end{bmatrix} =
\lambda
\begin{bmatrix}
e^{\tau + \beta_0 + \beta X} \\
1 + e^{\tau + \beta_0 + \beta X}
\end{bmatrix}
.$$
Taking the ratio and simplifying the non-zero constant in numerator and denominator we get that fitting a logistic model to the manufactured population results in an implied logistic model for the real population, $\mathbf{\text{with the same coefficients for X}}$ and with a difference in the constant (in the logistic model) given by:
$$ \beta_{real} = \tau + \beta_0 $$

$$\dots$$
Note that, according to your reference, the ratio of $\gamma_1 = Pr(Z=1|Y=1)$ and $\gamma_0 = Pr(Z=1|Y=0)$ should come up. Indeed:
$$ \gamma_1 = Pr(Z=1|Y=1) = \frac{P(Z=1,Y=1)}{P(Y=1)} = \frac{P_r(Y=1|Z=1)P_r(Z=1)}{P_r(Y=1)} = \frac{P_m(Y=1)}{P_r(Y=1)} P_r(Z=1)=
\frac{\mu_1}{\pi_1}P_r(Z=1) $$
likewise (i.e. change Y to 1-Y),
$$ \gamma_0 = \frac{1-\mu_1}{1-\pi_1}P_r(Z=1) $$
so
$$ \ln(\frac{\gamma_1}{\gamma_0}) = - \ln(\frac{\pi_1 (1-\mu_1)}{\mu_1 (1-\pi_1}) = \tau $$

$$\dots\dots\dots$$
Notes for full disclosure: I worked with the probability model. When one works with finite samples the example above suggests two ways of estimating the coefficients:
* estimate coefficients using the sample from the real population
* estimate coefficients using the manufactored populations

It terns out that this two estimators are not the same (it is obvious if one consideres one estimator is based on more cases than the other). Both estimators are asymtopically consistent, but it can be shown the one based on the manufactored population is more biased (forgot the reference :( ).

In the data science space we are more concern with the quality of the predictions than the parameters of the parameters used to make those predictions, so as long as you check results properly (e.g.: using a testing set to build models and another to validate them), the bias in the parameters should not deter us from using oversampling.

$$\dots\dots\dots$$


Customize loss function for Music Generation LSTM (?)

That's not a good way to represent symbolic monophonic music.
The biggest problem is that time should be modelled as a discrete and not a continuous variable. The reason for that is that music with notes that are off the beat sounds like crap. E.g if the tempo indicates that one beat is 250 ms long and one beat randomly is only 200 ms long it will sound very bad.
Instead, fix the tempo so that, say one quarter note is 125 ms long and have your time variable be an integer factor of that. You could cut the range to [1, 16] so that the shortest note is 125 ms long and the longest 2 000 ms (125 * 16).
Then I would skip the velocity because it is very hard to model. I don't think even state-of-the-art models incorporates it and most music sound fine even without it.
For pitches, I would cut down their range to three or four octaves. Most MIDI songs doesn't use more scales than that and it makes it much easier for your model to generalize.
So with four octaves, you are left with two variables $t \in [1,16]$ for time and $p \in [1,48]$ for pitch. You can either assume these are independent (which they aren't, but the generated music might still sound good!) and model them separately, giving you a sequence of pairs of discreete values:
$$
(t_1, p_1), (t_2, p_2), ...
$$
Or you could multiply $t$ and $p$ so that each input to the network is an integer in the range [1,16*48]:
$$
t_1p_1, t_2p_2, ...
$$
You could also interleave the input so that every other item in the sequence contains timing and pitch information:
$$
t_1, p_1, t_2, p_2, ...
$$
I don't know which way is the best - you'll have to experimetn.
I'm fairly certain it won't matter which loss function you choose. But I would go with crossentropy because it usually works well with discrete variables.


Methods of building machine learning models

Strictly speaking, those are not methods of "building" models, they are feature selection strategies.

Most of them are implemented in Scikit-learn.


With Stochastic Gradient Descent why we dont compute exact derivative of loss function?

That's because the whole loss is $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ and that number $N$ is the dataset size, it can be very large. It's just too slow to compute the true gradient, thus we compute its unbiased estimate via Monte Carlo. There are some theorems that say that stochastic gradient descent converges under certain conditions, so it's a reasonable method. You just don't to wait long (computing the true gradient), you can converge faster. The speed isn't the only reason. Also, researchers found out that using small batch size can improve the performance of neural networks and it's reasonable as well because the lower the batch size the higher is the variance of the estimate, and the higher variance (i.e. noise) and the higher variance prevents the net from overfitting.


Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide some default recommendations.

Of course, once you have any data, and you can rebuild the model to incorporate the user, you can make recommendations. You can do that in Spark but you already know that. This will take too long if you need to add information about new users at runtime. The technique you want is called "fold-in", used to figure out what the new user vector is (approximately) given the items the user interacts with. It's just some linear algebra and does follow from the equation you give.

I dug out an old slide which might help:

The "Cu" is not really different. I added an 'extension' to handle the case of negative input, but it's the same for positive input.

Here's an implementation of fold-in, although I think it's going to be too dense to be of much value:

https://github.com/OryxProject/oryx/blob/2c01d496f93f2825a076eb7fe492aa39a5290aa6/app/oryx-app-common/src/main/java/com/cloudera/oryx/app/als/ALSUtils.java#L74

Computing the new user vector implied by a user-item interaction is fairly easy linear algebra. The tricky part I've found is deciding how much to weight it.

Hope that is a push in the right direction.


My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?

In reinforcement learning, you must set rewards so that they are maximised when the agent achieves the goals of the problem. You should avoid trying to "help" the agent by setting interim rewards for things that might help it achieve those goals.

For card rejection, if that is part of the game (i.e. it is valid to play a "wrong" card, and you lose your turn), then either no reward, or a negative one might suffice. Probably you should go with no reward, because the punishment would be in not winning that round anyway.

If an invalid card cannot actually be played according to the rules of the game, and there is no "pass" move or equivalent, then you should not allow the agent to select it. Simply remove the action from consideration when making action selection. It is OK for the agent/environment to enforce this in a hard-coded fashion: A common way to do that, if your agent outputs a discrete set of action probabilities or preferences, is to filter that set by the environment's set of allowed actions, and renormalise.

What if you want the agent to learn about correct card selection? Once you have decided that, then it becomes a learning objective and you can use a reward scheme. The action stops being "play a card" and becomes "propose a card to play". If the proposal is valid, then the state change and reward for the round's play are processed as normal. If the proposal is not valid, then the state will not change, and the agent should receive some negative reward. Two things to note about this approach:

Turns in the game and time steps for the agent are now separate. That's not a problem, just be aware of the difference.
This will probably not encourage the agent to play better (in fact for same number of time steps, it will probably have learned less well how to win, because it is busy learning how to filter cards based on the observed features), but it will enable it to learn to propose correct cards without that being forced on it in a hard-coded fashion.

For winning a round, then you  might want to reward the agent according to the game score it accumulates. Assuming that the winner of the overall game is the player with the highest score, this should be OK.

However, there is a caveat to that: If by making certain plays the agent opens up other players to score even higher, then simply counting how many points the agent gets is not enough to make it competitive. Instead, you want very simple sparse rewards: e.g. +1 for winning the game, 0 for drawing, -1 for losing. The main advantage of using RL approach in the first place is that the algorithms can and should be able to figure out how to use this sparse information and turn it into an optimal strategy. This is entirely how AlphaGo Zero works for instance - it has absolutely no help evaluating interim positions, it is rewarded only for winning or losing.

If you go with +1 win, -1 lose rewards, then you could maybe make players' current scores part of the state observation. That may help in decision making if there is an element of risk/gambling where a player behind in the scores might be willing to risk everything on the last turns just for a small chance to win overall.


Is PCA (by eigendecomposition) or SVD better in decorrelating the predictors of a machine learning model?

To the best of my knowledge, the answer to your question is no. Regarding finding the correlations of different variables, they work the same. They both capture linear associations and do not capture nonlinear ones. The difference between them is mostly about numerical computation which makes SVD more handy than traditional PCA. I recommend having a look at this answer and this explanation.

  As a final remark, let’s discuss the numerical advantages of using SVD. A basic approach to actually calculating PCA on a computer would be to perform the eigenvalue decomposition of $X^TX$ directly. It turns out that doing so would introduce some potentially serious numerical issues that could be avoided by using SVD.


Why does a belief network need to be represented using a directed acyclic graph (DAG)?

Yes, we use DAGs to represent dependency relationships.

We need directed graph because condition probability P(A|B) is not same as P(B|A).

We assume it to be acyclic to get certain properties and ease calculations. You can create cyclic graph and you are going to get a lot of contradictions.

If you ease both the restriction(directed and acyclic), we get more general model called Markov Network.


I think maybe you are looking for:

receipts_by_name_x_cat = df.groupby(['name','category']).count()

Or, if you just want the total across all categories:

receipts_by_name = df.groupby(['name']).count()

Then, you can search those who have more than one:

receipts_by_name[receipts_by_name['receipt']>1]

And, you can find the length of an index by typing:

len(df.index.get_level_values(0))

Assuming the name was the first index column (otherwise substitute 1, 2, etc.)


Prioritized Replay, what does Importance Sampling really do?

DQN suffers intrinsically from instability.  In the original implementation, multiple techniques are employed to improve stability:

a target network is used with parameters that lag behind the trained model;
rewards are clipped to the range [-1, 1];
gradients are clipped to the range [-1, 1] (using something like Huber Loss or gradient clipping);
and most relevant to your question, a large replay buffer is used to store transitions.

Continuing on point 4, using fully random samples from a large replay buffer helps to decorrelate the samples, because it's equally likely to sample transitions from hundreds of thousands of episodes in the past as it is to sample new ones.  But when priority sampling is added into the mix, purely random sampling is abandoned: there's obviously a bias toward high-priority samples.  To correct for this bias, the weights corresponding to high-priority samples are adjusted very little, whereas those corresponding to low-priority samples are relativity unchanged.
Intuitively this should make sense.  Samples that have high priority are likely to be used in training many times.  Reducing the weights on these oft-seen samples basically tells the network, "Train on these samples, but without much emphasis; they'll be seen again soon."  Conversely, when a low-priority sample is seen, the IS weights basically tell the network, "This sample will likely never be seen again, so fully update."  Keep in mind that these low-priority samples have a low TD-error anyway, and so there's probably not much to be learned from them; however, they're still valuable for stability purposes.
In practice, the beta parameter is annealed up to 1 over the duration of training.  The alpha parameter can be annealed simultaneously, thereby making prioritized sampling more aggressive while at the same time more strongly correcting the weights.  And in practice, from the paper you linked, keeping a fixed alpha (.6) while annealing the beta from .4 to 1 seems to be the sweet-spot for priority-based sampling (page 14).
As a side note, from my own personal experience, simply ignoring the IS weights (i.e. not correcting at all) results in a network that trains well at first, but then the network appears to overfit, forgets what it's learned (a.k.a. catastrophic forgetting), and tanks.  On Atari Breakout, for example, the averages increase during the first 50 million or so frames, then the averages completely tank.  The paper you linked discusses this a bit, and provides some charts.


Compare similarities between two data frames using more than one column in each data frame

The last df is your required result, 2nd last is mine.
I think 3, item_f is not possible as its absent in df1.


Different feature importance results between DNN, Random Forests and Gradient Boosted Decision Trees

Few factors that can cause such differences -

A common issue with the default RF FE approach.
It's not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories
Read

Correlated Features
 When you have correlated features,
RF Tree-based FI approach will divide the importance depending upon how the Tree used the two correlated Features
Permutation based method will assign negligible importance to both as skipping anyone will have no impact on the result

Feature Interaction
 If we have two Features with Interaction,
Tree based approach will catch the Interaction in successive splits and accordingly divide the importance into two Features
Permutation based method will assign high importance to both the Features as dip will be high when anyone is skipped(Assuming additive Interaction)
In this case, this is a question to ponder how the FI should be divided when two Features are interacting. Ideally, a third feature(e.g. X1*X2) should be created and should get all the FI.

You should try -

Identity correlated Features using respective tests e.g.Pearson, Crammers, Spearman etc.
Use Permutation approach for all cases. Sklearn has got the option

#https://scikit-learn.org/stable/modules/permutation_importance.html
from sklearn.inspection import permutation_importance
r = permutation_importance(model, X_val, y_val, n_repeats=30)

Then if needed, look for any interaction effect (See suggested reading)

Suggested reading
Beware Default Random Forest Importances
Discovering-interaction-effects-in-ensemble
Chapter#7 - Feat Engg
Interpretable-ml-book


Breaking a DataFrame based on Specific Column Values

One way to separate these values out of your pd dataframe is to use Pandas indexing. There are many different routes that you could go, for this, I would recommend:
new_data = pd.data[data['salary'] >= 45000]

Should separate your data out nicely.
You can read further about this here


Absolute-value max pooling in 2D convolutional neural networks

If you need to implement such an absolute value max pooling you can convert the output of the convolutional layer to an absolute value and then apply a maxpooling.
Moreover such thought is not necessary as maxpooling works with the output of the convolutional layers and is not influenced by the images directly. By which I mean that the convolutonal layers extract the most important features from the image be it positive or negative that plays a role in predicting a correct output
So for example if a low intensity pixel affects the identification more then it will be given a positive large weight compared to a pixel value that is less influential.


How do I get the feature importace for a MLPClassifier?

The short answer is that there is not a method in scikit-learn to obtain MLP feature importance - you're coming up against the classic problem of interpreting how model weights contribute towards classification decisions.

However, there are a couple of great python libraries out there that aim to address this problem - LIME, ELI5 and Yellowbrick:

LIME (or Local Interpretable Model-agnostic Explanations, blog post here, arxiv paper here) which "explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction", or basically to explain model outputs by approximating the results of a classifier with a model that can be explained. It's on pypi and they have examples on their github page - very straightforward to get started.

ELI5 (or Explain Like I'm 5) is a "Python library which allows to visualize and debug various Machine Learning models using unified API". Although not all scikit-learn integration is present when using ELI5 on an MLP, Permutation Importance is a method that "...provides a way to compute feature importances for any black-box estimator by measuring how score decreases when a feature is not available", which saves you from trying to implement it yourself.

Yellowbrick is "a suite of visual diagnostic tools called “Visualizers” that extend the Scikit-Learn API to allow human steering of the model selection process" and it's designed to feel familiar to scikit-learn users. Compared to the other two libraries here it doesn't offer as much in the way for diagnosing feature importance, but it's still worth mentioning for more general use cases.


Welcome to the site! As Media has mentioned it is a classification problem not prediction problem. If you meant Logistic Regression then it makes sense.

You need to remember one thing if the outcome is only one value then algorithm would also return the same value irrespective of anything. if you don't have negative cases then you cannot use Logistic regression and it doesn't know when to say 0 as the model wasn't trained on it.

If possible using Business Matter Experts advice populate some dummy data for 0's and balance the data. Train your model and see what the model is going to say.

Else you cannot use any kind of classification algorithm.


Adding a bias makes Q-learning algorithm ineffective

Having a bias term with the one-hot encoding prevents each state’s Q values from being independent

Any ideas why this is the case?

The usual point of using linear regression or other function approximation in Q-learning is to generalise, and thus prevent the Q values from being independent deliberately. So this is not a general statement about Q-learning and linear function approximation. In general yes you can add bias terms - also, in general when using linear regression or neural networks to estimate action values, then you should have bias terms.
Having said that, I would expect the bias should make little difference here, since the one hot coding of state makes it redundant - you can already express any mapping of state to Q value of each possible action using just the weights. Although you are using what looks like a linear regression model, essentially this is just the tabular form of Q learning.
Q learning with function approximation can be unstable. The combination of off-policy, bootstrap updates and function approximation is known as the "deadly triad", and you need to add tricks like experience replay to keep it stable. Although in theory the linear regression should learn a correct bias, probably adding bias puts the algorithm into that unstable zone.


After anaconda installation spyder not working

According to this bug report, you can solve this with:

pip install --upgrade pip setuptools


Do you only need to differentiate between one or more spaces?
If so, maybe you can make use of a regular expression separator as listed in the pandas docs.
Something like this:
pd.read_csv(... sep='\s+', ...)

Also, consider adding the parameter delim_whitespace as such:
pd.read_csv(..., delim_whitespace=True, ...)


What are h(t-1) and c(t-1) for the first LSTM cell?

Assuming that the LSTM is going to be used for sequence generation (e.g. in a language model or the decoder part of an encoder-decoder NMT architecture), we have the following:

In supervised learning setups:

In a language model, the LSTM's  $h_{-1}$ and $c_{-1}$ are initialized to zeros, for all the layers. If the LSTM is the decoder part of an encoder-decoder NMT architecture, they will be initialized with the hidden state and context vector of the LSTM from the encoder part, either from the last position of from a combination of all positions by using some attention mechanism.

During training, the inputs $x$ are not fed with the LSTM predictions for the previous tokens because this presents convergence problems; instead, you use teacher forcing, which consists in feeding the gold tokens (the tokens from the training data) as inputs.

During inference, you have to use the network's own predictions as inputs for the next tokens.

The mismatch between the training and inference regimes is said to present exposure bias, that is, during training you make the network see only "good" previous tokens and it never learns to recover from its own bad predictions. However, according to recent research, exposure bias may actually not be a serious problem.

In more exotic learning setups:

In textual GANs, $h_{-1}$ and $c_{-1}$ are sometimes initialized with a latent vector $z$ that follows some prior distribution.

Also, in textual GANs and other learning setups where there are no gold tokens, teacher forcing is not an option, so you need to use the LSTM predictions as previous token inputs.


I don't know your data, so I can't say if your approach is good or not. But your approach does not cover weekdays. So I would apply time series analysis methods predict the service date.


What are the 2 classes of categories to help define a problem

I never saw a name for these categorifications. However, it seems to categorize by type how target variable exists or what is the type of target variable (in supervised and semi-supervised) or input variable (in unsupervised)

[wrong] 2. I think in data mining we call "Reinforcement Algorithms" semi-supervised.

I think Ranking will be a regression problem. just after defined number you can change them to ranks based on their order.
There is some correlation between the first and second type of categorifications you named, you can see that in this picture:

https://towardsdatascience.com/supervised-vs-unsupervised-learning-14f68e32ea8d


Predict future value with time period using non linear regression model

If your data is time series data then you should analyze it accordingly, with a time series model. Alternatively, you could build a regression model with polynomials or splines, depending on your data and how "curvy" it is. Another option would be a GAM model (Generalized Additive Model), which works very well with curves of various shapes.


What method/algorithm for constrained multi-target regression

After some further research and inspiration I have figured out the following method:
I am using the vector notation of my problem with
$\underline{y}=\begin{pmatrix}y_1\\y_2\\y_3\end{pmatrix},~\underline{x}=\begin{pmatrix}1\\x_1\\x_2\\x_3\end{pmatrix},~\underline{\underline{B}}=\begin{pmatrix}\beta_{10}&\beta_{11}&\beta_{12}&\beta_{13} \\ \beta_{20}&\beta_{21}&\beta_{22}&\beta_{23} \\ \beta_{30}&\beta_{31}&\beta_{32}&\beta_{33}  \end{pmatrix}$
This results in the optimisation problem with $N$ examples
$\underline{\underline{\hat{B}}}=\underset{\beta \in C}{\operatorname{argmin}}L(\underline{\underline{B}})=\underset{\beta \in C}{\operatorname{argmin}}\sum_{i=1}^{N}\lVert \underline{y}_i - \underline{\underline{B}}\cdot\underline{x}_i \rVert^2,~~~~\text{with}~~~~C= \lbrace \underline{\underline{B}}\in \mathbb{R}^{3\times4} \mid \beta_{11}+\beta_{22}+\beta_{33}=0\rbrace$
For solving this, I am using projected gradient descent where $\underline{\underline{B}}$ for every iteration is projecet onto $C$. Normally the Projection operator would require another optimisation process $\operatorname{proj}_C(x)=\operatorname{argmin}_u \big\lbrace \chi_C(u) + \frac{1}{2}\lVert u-x\rVert^2\big\rbrace$ with $\chi_C$ the characteristic function.
Now comes the neat part: Since the constraint $C$ describes a flat subspace in a 12-dimensional vector space, the projection can be described as a simple linear map.
Using a reduced version of $\underline{\beta}=\begin{pmatrix}\beta_{11}\\\beta_{22}\\\beta_{33}\end{pmatrix}$ (in the projection step $\underline{\underline{B}}$ will be flattened to a vector to avoid working with 3D tensors), the projection $\underline{\beta}'=\operatorname{proj}_C(\underline{\beta})$ can be represented as
$\underline{\beta}' = \underline{\underline{P}}\cdot\underline{\beta}~~~~\text{with}~~~~ \underline{\underline{P}}=\frac{1}{3}\begin{pmatrix}2&-1&-1\\-1&2&-1\\-1&-1&2\end{pmatrix}$
The vector $\underline{\beta}$ and projection map $\underline{\underline{P}}$ can then be arbitrarily augmented with the remaining entries of $\underline{\underline{B}}$, as they will not be affected by the projection.
With the above method, my constrained gradient descent is only slightly more expensive than a usual gradient descent method by computing one extra linear operation for each iteration.
The results I have obtained so far with the method seem promising


Prime model with mock data, good idea or not?

This is not the answer you'll want to hear, but I would say, "no." Priming is not a good idea. A model is only as good as its inputs. If you're making up the inputs yourself, then your model isn't learning real patterns and you might as well just hard code a set of rules that represent how you would have created the primed data.

A fancier way might be to make a bayesian model where you create priors based on your assumptions. The bayesian model would then evolve as real data becomes available.


Similarity score: Can Sklearn SVR predict values greater than 1 and less than 0?

I am using svm.SVR() from scikit-learn to apply Logistic Regression on my training data to solve similarity problem.

Wait a second, if you're using support-vector regression, then you're not using logistic regression.  These two are very different algorithms.  They aren't even applicable to the same type of problem.  Support-vector regression is used when you are predicting a continuous target, whereas logistic regression (despite the name) is a classification algorithm.

  However, for some of the test data, the prediction contains value in negative (less than 0) and for all the same vs same comparison it returns the value as 1.09469178.

There's nothing unusual about this if you're using support-vector regression.  A support-vector machine can (in theory) output any real number.

Logistic regression, on the other hand, is a sigmoid function.  It will take as input any real number and output a result between 0 and 1.  Maybe you meant to use scikit's LogisticRegression model rather than SVR?

  The reason I am using SVM-Regression is to find similarity between two inputs.

Could you expound on your use-case a bit more?  Support-vector regression isn't really meant to be used to compute similarity.

I'm speculating that you have a training set of (X, y) pairs (where y is a label between 0 and 1).  You are training a model to output $\hat{y}$, a prediction of y.  To find the similarity of two inputs, $X_i$ and $X_j$, you pass them both through the model and measure the difference in the $\hat{y}_i$ and $\hat{y}_j$.  Is that right?

If so, I think this is a really roundabout way of computing similarity, and is unlikely to yield better results than a more straightforward method.  I'm not sure that you need a machine learning algorithm at all.

Have you looked into other similarity metrics that might be suitable for your problem?  For similarity based on Euclidean distance, you can compute $\frac{1}{1 + d(X_i, X_j)}$ (where $d$ is the euclidean distance function).  Cosine similarity might be a good choice if you care more about the similarity in direction of two input vectors.


DBSCAN clustering on document [updated]?

Bilgin!

Anony-Mousse puts right questions and gives good suggestions. Before you use the self-implemented DBSCAN code - write it on paper. Perhaps it is not the best algorithm at all for your database so try sci-kit learn implementation first to see the results.

Here are the Python implementation https://github.com/chrisjmccormick/dbscan/blob/master/dbscan.py and here is the theory https://github.com/chrisjmccormick/dbscan/blob/master/dbscan.py

Good luck!


Are there any learner-specific optimizers?

I would argue and say that different machine learning algorithms use different optimization algorithms. Linear regression has a direct solution $(X'X)^{-1}X'Y$. For logistic regression sklearn python library uses a Liblinear C++ library. The library uses "a trust region Newton method". SGD (Stochastic Gradient Descent) with Backpropagation is mainly used in neural networks and I did not hear about both of them before implementing my first neural network. LightGBM and XGBoost use their own invented heuristics to build decision trees.

Generally speaking most of ML algorithms are fast. If the training time is in seconds there is no need to optimize the time of algorithm execution. If the training time is actually in years (most recent papers on neural networks by Google where they use hundreds of GPUs and CPUs) then you definitely start thinking about speed optimization.

SGD is supposed to be inferior to other algorithms because it uses only first derivatives. The problem with the neural networks is that the gradient changes too fast and the calculation of the second derivative appears too costly (There are papers that benchmark it). SGD was specifically optimized for neural networks with momentum calculations and reducing learning rate after the networks stopped training. Constant attempts are made to improve it, for example ADAM. But ADAM is still considered to perform worse on validation sets (There was a big benchmark paper about it).

There are some papers that try to use neural networks to choose the direction of the descent instead of using SGD. Papers show some marginal improvements in speed but the algorithms are still not used as the main method of optimization.

Lots of other algorithms are tried for neural networks. For example, genetic algorithms. There are papers that use the idea of "stochastic" in SGD for genetic algorithms. According to their benchmarks they achieve 1/10th of the speed of SGD. They argue that the best neural networks are influenced a lot by the ability of SGD to find a minimum (like ResNet with residual links). But if you choose genetic algorithms then maybe it is possible to use more complicated neural networks where SGD actually fails.

In term of how to call the algorithms.

I prefere the name "optimization algorithm" or "algorithm used for training". But for some of the ML methods "optimization algorithm" sounds wrong. LightGBM and XGBoost have pretty complicated algorithms of building trees where they balance between optimization of a score (logloss or sum of squares) and using other algorithms to fight overfitting. Decision trees could be trained to zero error but then they are one of the worst predictors for the validation set. LightGBM and XGBoost do not reduce their errors to zero and thus allow to win most of machine learning competitions with structured data on Kaggle.


What is the intuition behind using LSTM for classification tasks?

I'll go for you questions one by one:

  What is the rationale behind using it in classification tasks?

LSTM Networks for classifcation tasks are mainly due to NLP. In particular, Sentiment Analysis. What is the sentiment/polarization of a tweet? By "understanding" natural language, a model can guess whether it is positive or negative towards a given issue.
Since the sequence of words is fundamental in order to understand natural language, you need to capture the sequence of words in order to understand the text, that you then need to then estimate the sentiment.

--

  Can the classification problem be reduced to sequence prediction
  problem in some way?

I think NLP made the two tasks very related to each other, even though in my opinion they can still be analytically separated, so I wouldn't say that one can be reduced to the other.

However let's say that, once you have implemented a model that is very good at estimating the sentiment of a text, you can then make it generate positive/negative texts. In other words, they are closely related (thanks to NLP) but they are not the same.

Hope this helps, otherwise let me know.


Any pytorch tools to monitor neural network's training?

I am using tensorboardX. It supports most of the features of TensorBoard. I am using the Scalar, Images, Distributions, Histograms and Text. Haven't tried the rest like audio and graph. But the repo also contains examples for those usecases. Installation can be done easily with pip. It's all explained in the readme.

There are also other software which implement a wrapper for PyTorch (and other languages/frameworks) of TensorBoard. As far as I know, they support fewer functionalities. But have a look at Crayon and Tensorboard-Logger.


correlation between categorical(ordinal) and discrete(continuous) value

You can juse bin them to numerical bins [1 - 5] as long as you are sure you're doing this to ordinal variables and not nominal ones. Bivariate analysis should be easier for you.

If you still want to see how to get correlation of categorical variables vs continuous , i suggest you read more about Chi-square test and Analysis of variance ( ANOVA )


Dealing with issues in "test" predictons for single "items" (null values, standardization in place, etc)

You need to save the instructions for performing these preprocessing steps, not necessarily the dataset that you extracted them from.

See
Obtaining consistent one-hot encoding of train / production data
Binary Classification - One Hot Encoding preventing me using Test Set
one-hot-encoding categorical data gives error

In particular, sklearn preprocessors can be pickled, then used with their transform method in production, if you can use sklearn in deployment.  PMML also cam translate most transformers.  Or you can write your own simple transformer.

As to using newer data to rework the transformers, that's getting closer to retraining; in most settings, I would keep it in the same place as refitting the model: either both offline or both online.


pandas data frame doesn't show any thing ,when view as data frame in pycharm

Welcome to the site! Instead of describe, try print(dataSet.head()) and that should show you some of the data in your dataframe.


Variable Importance in unsupervised anomaly detection algorithms

There have been workshops dedicated to "outlier detection and description" (ODD), but there came out nothing from them that convinced me, unfortunately. but YMMV.
It definitely won't be enough to just use some library! You'll need to read and implement papers. There are subspace outlier detectors and correlation outliers, for example, that will tell you which features were relevant for a particular outlier.

But in general I believe you'll quickly run into multiple testing problems: in real data, every point is anomalous if you just try attribute combinations hard enough. Donald Trump is anomalous because of his orange skin, fake hair, and small hands, for example. If you only look at his xenophobia, he probably is pretty normal, unfortunately.


Feature Selection with one-hot-encoded categorical data

Sequential forward selection appears to be a greedy search algorithm if I am not mistaken? It appears you initially fit all possible one variable models, and then choose the one variable model that is highest performing. You then attempt to add a second variable to the highest performing one variable model from the prior step, by fitting all possible two variable models. You choose the highest performing two variable model, and if it is superior to the best one variable model, you move to the two variable model. The process continues, trying all three variable models, then four (if the best three variable model is superior to the best two)... until you choose the best k features or until the model stops improving.

The best k seems quite arbitrary and if there is a way to let the algorithm continue until there is no improvement then that is what I would do. However, here are some initial impressions of this approach that I have:

I suppose that a problem with this is that you have a large chance of not finding the most optimal model (local solution). A candidate variable may work particularly well with other variables that have yet to be included in future steps, or you may not even get the chance to use potentially useful predictors if the algorithm terminates beforehand.

However, the largest problem with this method as I see it is just how expensive it is. You state that you have 400+ columns. Say you have 415 variables. Then, in the first pass you will have to fit 415 models, then on the second pass, 414, etc...and this is not even including possible hyperparameter tuning and cross validation. This is a huge amount of models, and to be honest this is often the problem with wrapper based methods of feature selection in particular. The majority of them end up fitting a large amount of models for potentially marginal, if any, gain in model performance. The problem is made worse when you have hyperparameters that need to be tuned, and so I find the tradeoff between performance gains and the computational time spent using these methods to not be worthwhile, unless you want the absolute best model.

In your case, Peter's advice in the comments is the route I would take purely because it is way faster and often good enough. Almost every modern ML method offers regularization in some way that will explicity feature select for you aka embedded feature selection (by not using unhelpful predictors at all) or by strongly limiting their influence via. shrinkage. Ridge/LASSO/Elastic Net mentioned by Peter is a terrific suggestion. Other methods such as those based on trees also have embedded feature selection and may work well in your case, considering the dimension of your dataset.


CNN Design for Counting on Simple Images

This doesn't seem like a Deep Learning problem, this could be solved by classic Computer Vision techniques, so maybe you could inspire your architecture by those.

Check is your model convolutional layers receptive field is large enough for the shapes you trying to count.

You probably won't be able to do counting without some kind of post processing

Check Yolo strategies, you probably will be able to do a counter with it.

This is a object detection problem, you will often have poor performance trying to use a architecture designed for image classification such as AlexNet while trying to face this as a regression problem.

Your output size should probably be a varying length vector from which the length is the desired count.


So there two ways of doing this, IMHO,

By creating a well balanced target variable by choosing the right threshold. As I suggested in the comments above. In doing so we are simply taking care of values which should be treated as positive which would otherwise become negative if we take lower threshold.
By using the mean threshold and that will generate imbalanced target variable and when you perform the modelling, you can see the ROC and PRC curves and decide the threshold based on that. But keep in mind, it also depends what kind of problem you are solving.


Are non-relu activations better for small/ dense datasets?

There is no reason to to predict tanh would be better if for a smaller, dense dataset.
The effect of activation function varies from dataset to dataset. You could it treat like an empirical question - find the best activation function for a specific data set like any other hyperparameter.


In short PCA, returns an orthogonal set of basis features that best represent the variance in the data.

Intuitively, imagine you want to identify whether we are talking about a dog or a cat. Your features are: size, weight, color, fur type, etc... but you also have features like weather, owner name, etc...

It should be evident that the first set of features obviously explains the variance between a dog and a cat much better than the second set. Thus, you should only consider those and completely disregard the second set.

This is what PCA does however, it adds the extra restriction that all features must be orthogonal. It then assigns a metric to each component based on the amount of variance that feature explains. You can then sort the components and extract the top $n$ components (feature reduction).


Is the early stopping of xgboost using correct

That's not cheating. If your test set is a representative sample of the future data you'll want to make predictions on, you'll want to have the lowest possible error there!

If you maximized performance on the training set, instead, you might overfit. It's fairly easy for a boosted algorithm to inadvertently memorize its training data rather than learn a meaningful mapping of inputs to output.

Per the comment below, the "test set" you describe is actually functioning like a validation set here.


In general, generating independently training and test sets is a legitimate option. The crucial aspect is that both the generating processes are equal. You can check this looking at this example from the author of the caret R package and the Applied Predictive Modeling book.
However, it is something that can be easily proven with simulations. In what follows, both generating independently training and testing data or splitting the same data in training and testing subsets give the same results. The glm has a median accuracy of 92%.

# simulations with training and test data generating at the same time
n <- 100
accuracy <- vector("numeric")
for (i in 1:1000){
  #create data
  x <- rnorm(n)           # generate X
  z <- 1 + 4*x + rnorm(n) # linear combination with error
  pr <- 1/(1+exp(-z))     # inv-logit function
  y <- pr > 0.5           # 1 (True) if probability > 0.5
  df <-  data.frame(y = y, x = x)
  train <- sample(x = 1:n, size=n%/%2, replace = F) # sampling training data units
  glm.fit <- glm(y ~ x, data = df[train,]) # fit on the training data
  predicted <- predict.glm(glm.fit, newdata = df[-train,]) # predict on the other data units
  accuracy=c(accuracy, sum(diag(table(predicted>0.5, df[-train,]$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy

# simulations with training and test data generating independently
n <- 100 # dataset size
accuracy <- vector("numeric")
for (i in 1:1000){
  #create data
  x <- rnorm(n%/%2)           # generate X
  z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error
  pr <- 1/(1+exp(-z))         # inv-logit function
  y <- pr > 0.5               # 1 (True) if probability > 0.5
  df.train <-  data.frame(y = y, x = x)
  glm.fit <- glm(y ~ x, data = df.train) # fit on the training data
  # generating independent test data
  x <- rnorm(n%/%2)
  z <- 1 + 4*x + rnorm(n%/%2) # linear combination with error
  pr <- 1/(1+exp(-z))         # inv-logit function
  y <- pr > 0.5               # 1 (True) if probability > 0.5
  df.test <-  data.frame(y = y, x = x)
  predicted <- predict.glm(glm.fit, newdata = df.test) # predict on the test data
  accuracy=c(accuracy, sum(diag(table(predicted>0.5, df.test$y)))/(n%/%2)) # collect accuracy
}
quantile(accuracy, probs = c(0.025, 0.5, 0.975)) # glm accuracy


You might want to check out this article on Neural Networks, Manifolds, and Topology for a "geometric interpretation of MLP output".
Here is an excerpt that you might be most interested in:

There are a variety of different kinds of layers used in neural
networks. We will talk about tanh layers for a concrete example. A
tanh layer tanh(Wx+b)  tanh(Wx+b)  consists of:
1.A linear transformation by the “weight” matrix W
2.A translation by the vector b
3.Point-wise application of tanh.

and here is an animated gif of the geometric description he gives:

At the beginning of this animation, you will see the lines are straight, and undergo a linear transformation (weights are multiplied by inputs, Wx) where the lines move, but remain parallel, with the origin also remaining at (0,0); this is what happens in all linear transformations.
Then, you will see the transformation translated by the bias vector b.  That is, you will see all the lines move together in the direction of the bias vector b.  In this case, you will see the entire picture move down and to the right.
Finally, you will see the "kernel trick" which transforms the inputs to the next layer in a nonlinear way, causing the nice parallel lines to warp.  This non-linear warping allows the neural network to solve non-linear problems.  That is, if the space is distorted, and the points you are trying to classify move apart under this non-linear transformation, it may then be possible to create a separating hyperplane between points which should have different classifications, thus allowing the neural network to differentiate between your classes.
Hopefully, this will give you a geometric interpretation of what is happening at each layer of a neural network.  From there, you can extrapolate as you add more and more layers to your network.
In the example shown above, he is using tanh for the "kernel trick", but a number of other nonlinear functions could also be used (e.g. Sigmoid, ReLU, etc.)
If you find this animated gif hard to follow, I would strongly recommend you check out 3Blue1Brown's Essence of linear algebra series where he has a number of videos with intuitive explanations of Linear Algebra and great geometric animations.


General approahces for grouping a continuous variable based on text data?

Since it is general methodological question, let's assume we have only one text-based variable - total number of words in a sentence. First of all, it's worth to visualize your data. I will pretend I have following data:

Here we see slight dependency between age and number of words in responses. We may assume that young people (approx. between 12 and 25) tend to use 1-4 words, while people of age 25-35 try to give longer answers. But how do we split these points? I would do it something like this:

In 2D plot it looks pretty straightforward, and this is how it works most of the time in practise. However, you asked for splitting data by a single variable - age. That is, something like this:

Is it a good split? I don't know. In fact, it depends on your actual needs and interpretation of the "cut points". That's why I asked about concrete task. Anyway, this interpretation is up to you.

In practise, you will have much more text-based variables. E.g. you can use every word as a feature (don't forget to stem or lemmatize it first) with values from zero to a number of occurrences in the response. Visualizing high-dimensional data is not an easy task, so you need a way to discover groups of data without plotting them. Clustering is a general approach for this. Though clustering algorithms may work with data of arbitrary dimensionality, we still have only 2D to plot it, so let's come back to our example.

With algorithm like k-means you can obtain 2 groups like this:

Two dots - red and blue - show cluster centres, calculated by k-means. You can use coordinates of these points to split your data by any subset of axes, even if you have 10k dimensions. But again, the most important question here is: what linguistic features will provide reasonable grouping of ages.


Can I use Keras to give fitness scores to a genetic algorithm?

Yes, you can get a neural network to predict real numbers instead of for classification. This type of problem is called regression.
You can find dozens of tutorials for doing regression with Keras. The official one is here.


How is an ASR's output compared to ground truth for validation?

The answer I was looking for is Word Error Rate. It is the most standard way of comparing ASR transcriptions wrt ground truth. It is less granular than what I had in mind, it is basically Levenshtein Distance on a word level rather than a character level.
jiwer in python has a few other metrics and is easy to use.


How is DS used in the case of Payment Gateways?

Common use cases include:

Fraud detection
Transactions volume prediction
Next transaction date

Fraud detection

This is usually tackled with anomaly detection.

It requires information on the two transaction parties and using machine learning to figure out when a transaction is out of the norm and flagging as a potential case of fraud.

Transactions volume prediction

This is usually tackled with time-series forecasting.

The idea is to predict the amount of transaction that will be done in the next day/week/month for a large amount of users. It requires seasonal information and, in case you have a lot of C2B transactions, analysis on major business.

Next transaction date

This is also usually tackled with time-series forecasting. However, this analysis can be done on individual levels and not necessarily on a group of users.

The idea is to predict the date that the next transaction will be executed. This can be useful because there are cases where users will perform a lot of transactions in a short period of time, which could may be grouped together.


How Transformer is Bidirectional - Machine Learning

It is the encoder part of the Transformer model that is bidirectional in nature, not the whole model.
The full Transformer model has two parts: encoder and decoder. This encoder-decoder model is used for sequence-to-sequence tasks, like machine translation.
There are other tasks, however, that do not need the full model, but only one of its parts. For instance, for causal language modeling (e.g. GPT-2) we need the decoder. For masked language modeling (e.g. BERT), we need the encoder.
The decoder is designed so that each predicted token can only depend on the previous tokens. This is achieved with self-attention masking, and it is what makes the decoder unidirectional.
The encoder does not have self-attention masking. Therefore is designed not to have any dependency limitation: the token representation obtained at one position depends on all the tokens in the input. This is what makes the Transformer encoder bidirectional.


Learning resources for Data Science for applications in road traffic data?

The best way to learn data science is through problem solving. I suggest you to head over to Kaggle and work through the for-knowledge problems.

To get a good start on Machine Learning problems, acclimate yourself with the tree package in R. This will help you understand how decision trees work, and building upon that, how random forests, gradient boosting machines and other sophisticated tree based algorithms work.

Then, there are SVMs and deep learning models.

To get an understanding of unsupervised learning problems, learn k-means and employ it for clustering.

Other general concepts/ ideas to understand are:

cross-validation
overfitting, regularization
bias-variance trade off
dimensionality reduction/ variable selection
generalization error
ensemble learning

For books, the most common recommendation to anyone who is familiar with statistics and wants to get into Machine Learning is "The Elements of Statistical Learning" by Hastie, Tibshirani, and Friedman.


The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the selection of a dictionary. If the word is not in the dictionary we have chosen, then we are saying we have no valid representation for that word (or we are simply not interested).

Other tags are commonly used to groups such thing together, not only (j)unk.

For example <EMOJI> might replace any token that is found in our list of deined emojis. We are keeping some information, i.e. that there is a symbol representing emotion of some kind, but we are neglecting exactly which emotion. You can think of many more examples where this might be helpful, or you just don't have the right (labelled) data to make the most of the contents semantically.


Making sense of indices in 2D convolution operations in convolutional neural networks

From http://www.deeplearningbook.org/contents/convnets.html,

  The only reason to ﬂip the kernel is to obtain the commutative property. While the commutative property is useful for writing proofs, it is not usually an important property of a neural network implementation. Instead, many neural network libraries implement a related function called the cross-correlation, which is the same as convolution but without ﬂipping the kernel.

The discrete cross correlation function with 1-based indexing for $$I = U*V\ and\ K=M*N$$ is given by  $$C(x,y)=\sum_{m=1}^M\sum_{n=1}^NI(x+m-1,y+n-1)K(m,n)$$

where $$x\in(U-M+1)\ and\ y\in(V-N+1)$$

When applying 2-D convolutions, many neural network implementations will reduce the size of the output. To retain the original size in output and to retain information at the borders, many practical use cases add a padding of zeros to the input image.

Coming back to your specific question, in the case where indices go out of bounds, either on the positive side or on the negative side, (which happens when you try to retain the size of the input in your output), values corresponding to them are usually taken as zeros, which is akin to zero-padding in implementation.

Discrete convolutions are actually given by $$(f * g)[n] = \sum_{m=-\infty}^\infty f[n-m] g[m]$$ In practice the limits are made finite because of the same assumption of zero amplitude signal for out of bound indices.


Convolutional Neural Networks - When do I adjust parts of it, and how?

If you wanna get the optimal parameters, you can try using grid search. A grid search or parameter sweep does an exhaustive searching through a specified subset (you have to define this subset) of the hyperparameter space of the learning algorithm which in your case is the Convolutional Neural Network. Although this may take a long time but this is one of the ways you'll know the good parameters for your model.

Try searching for optimization algorithms or parameter tuning techniques.


Clustering big data by reducing data accuracy?

You have n=20000000 data points and d=20 attributes.

Approach 1

The hierarchical agglomerative clustering has a time complexity of O(n^3)
and requires O(n^2) memory. Therefore it is infeasible for large data sets (not to mention big data)

What you can do is first use some nearly-linear-time clustering algorithm to cluster your data to, let as say, 2000 clusters. This algorithm can be, for example, K-Means. See this stackoverflow answer for its time complexity. This is actually an alternative to your discussion of 'reduce accuracy and group'. Then you apply hierarchical clustering to the found 2000 clusters.

Approach 2
Use the algorithm, which is specially designed for large data.
For example, Birch algorithm


understanding linear algebra of a forget gate

Note that $$[h_{t-1}, x_t]$$
is the concatenation of two vectors.
In your example, it would be: $$[h_{t-1}, x_t] = [-1, 2 , 3, 0]$$
and then the dimensions of $W_f$ would be $2 \times 4$, where $2$ is the dimension of the output of the LSTM cell, i.e. the activation $h_t$, that you defined to be of dimension $2$.

Hence, $$W_f \cdot [h_{t-1}, x_t] $$ is a multiplication of a matrix of dimension $2\times4$ by a vector of $4$, which will return a vector of dimesion $2$. And then the sigmoid function will be applied point wise on each of the two elements of the result.

Hope it makes sense.


Does CART algorithm takes into account in the order of the set of attributes?

A possible answer.

Check to see if you have nominal variables with more than a reasonable number of levels. If that is the case then a heuristic is applied to select the best split. One such heuristic is to group levels on left to right, and that might affect the order. Anyway, it is reasonable to expect that the order of levels should not affect the output.

See this link for detalied explanation on how this is implemented on matlab
matlab documentation.


You could compute mean and standard deviations in sliding windows, and use those to remove outliers.

For example, taking windows of, say, length 100, you can compute the mean and std for for these 100 successive observations, and see whether any point falls above the 3 sigma rule. In this case, the circled outlier would still be recognized as such, while the others should not, as they are not so outlying w.r.t. neighboring data (i.e. within the window that includes them and their neighboring observations).


Advantages of Binary Rating System for Collaborative Filtering Recommender Systems

This article from Business Insider answers your question from the business perspective.

They mention that the original problem is that the way Netflix used the 5-star rating was not standard within the industry:

  Netflix’s Cameron Johnson, who oversaw the shift, told Business
  Insider that it all came from the realization that Netflix had always
  used star ratings differently than the rest of the internet, but that
  this distinction wasn't clear to users.

And the way ratings were presented to the user discouraged them to contribute. Their solution was the thumbs up/thumbs down approach because it was more clear to the user that they were training an algorithm:

  So when looking for a replacement, Netflix wanted to make sure that
  was clear. That’s why Netflix settled on “thumbs up/down," which is
  widely understood to imply that you are training an algorithm to know
  what you like, Johnson said.

I suspect there is more than that thought, and maybe binary data is simply "good enough".

This chapter from Mining of Massive Datasets has some interesting considerations on ratings versus boolean utility matrices.

From page 339:

  If the utility matrix is not Boolean, e.g., ratings 1–5, then we can
  weight the vectors representing the profiles of items by the utility
  value. It makes sense to normalize the utilities by subtracting the
  average value for a user.

Specially when treating rating matrices one need to be aware that different users may have different understanding of ratings, hence you will most likely normalise the matrix before calculating the recommendations, while for binary data this is not required.


Why are autoencoders for dimension reduction symmetrical?

There is no specific constraint on the symmetry of an autoencoder.

At the beginning, people tended to enforce such symmetry to the maximum: not only the layers were symmetrical, but also the weights of the layers in the encoder and decoder where shared. This is not a requirement, but it allows to use certain loss functions (i.e. RBM score matching) and can act as regularization, as you effectively reduce by half the number of parameters to optimize. Nowadays, however, I think no one imposes encoder-decoder weight sharing.

About architectural symmetry, it is common to find the same number of layers, the same type of layers and the same layer sizes in encoder and decoder, but there is no need for that.

For instance, in convolutional autoencoders, in the past it was very common to find convolutional layers in the encoder and deconvolutional layers in the decoder, but now you normally see upsampling layers in the decoder because they have less artefacts problems.


Mean Absolute Error increasing with more correlated factors

Any reason why you are using MAE instead of MSE? The reason for using MSE is that you get a parabola which would be more accurate for the optimizing algorithm to find the local minima

Did you check the relation between predictors and target variable? did you check multi collinearity?


The way I see it is that your 10 sources of data, they all refer to the same set of people. Depending on the attributes, some can be expanded, some can be merged ...

Attributes such as age should be unique, so it doesn't make sense to expand it to Profile 1 age, profile 2 age ... One simple way is merge them is by using the average or use max. Expanding age only add redundant data to your feature matrix, and increase its dimensionality, in most cases, this doesn't help generalization performance of your model.

On the other hand, number of followers can be expanded. Depending on the data source, a guy has 10 followers on Twitter but 1000 followers on Google+ might simply mean that he barely uses Twitter.

That being said, the way you pick your features or engineer new features should increase your model performance, so if expanding number of followers actually decrease Cross Validation or Test performance, compared to the one using sum of followers then you can simply use sum of followers.


Effect of adding extra unrelated features to linear perceptron

If you add a new feature to the perceptron, the perceptron actually gets one more parameter. So in some sense it is not the same model anymore. But lets ignore that.

Correlation is the key problem. Assume the new feature is a random number which happens to have positive correlation with the desired output for the training dataset, but negative correlation with the test dataset. Then you would expect this new feature to influence the test error negatively, right? At least I would.

But your question was about the training error. In that case I actually don't see a reason why it should be the case. I'd suggest to try it.

And anyways, ask the person/organisation, where you found this question. (And post the answer back)


How to handle like meaning sentences when working on text summarization

In case by 'summarize' you mean in fact 'classify', e.g. 'bad', 'good', etc, then an LSTM/GRU will do the job, just input the whole (truncated probably) sequence, and take the last state:
out, _ = lstm(embedding)
out_state = out[:,-1,:]

Here embedding is whatever word embedding you are using, out is the whole output of LSTM, and batch_first=True (in pytorch), so you take the last state as the input in the fully connected layer.
If you need to make predictions about parts of the text, just add a loop and truncate sentences rather than the whole text.


What does this sentence mean?

It means that the next vector should be perpendicular to all the previous ones with respect to a matrix. It's like how the natural basis vectors are perpendicular to each other, with the added twist of a matrix:

$\mathrm {x^T A y} = 0$ instead of $\mathrm{x^T y} = 0$

  And what is line search mentioned in the webpage?

Line search is an optimization method that involves guessing how far along a given direction (i.e., along a line) one should move to best reach the local minimum.


When can AUC and accuracy rate be equal?

AUC (or most often AUROC = "area under receiver operating characteristic") and accuracy are different measures, but used for same purpose - to objectively measure performance of a simple binary classifier.

The two measures can be equal at extreme values of 0 and 1 for perfect classifiers - or inverse perfect classifiers (you can just invert the output to get a perfect classifier). It might be possible for the values to be numerically equal at other times, but if so it would be a coincidence with no specific meaning.

Both these metrics can be used with a simple classifier that only outputs true or false values for class membership. However, the AUROC metric requires some kind of parameter you can vary in order to plot the ROC curve. Usually this is a threshold for classification, used against the class probability output of a classifier.

There are other possible metrics. For example, F1 score and cross entropy. The F1 score again has 1.0 for a perfect classifier and 0.0 for a bad classifier, but cross entropy scores lower for better classifiers - 0.0 for perfect and no upper limit for bad output. Again, the values of these might be equal to other metrics at some points, but if so it is not meaningful.

If you are comparing two classifiers for a particular task, then it is important to compare them using the same metric, on the same test data. The metric and test data you choose should relate to your original problem.

If you are reading other people's published results, and want to compare them, it is not really possible if one person has used AUC and the other accuracy.


Recommendation systems eg. Netflix movie recommendations, I assume its pretty straight forward to get. Recommendations systems can be in either of content based filtering, collaborative filtering and combination of these two.
Now, we can use matrix factorization for solve this learning problem (problem here is the recommending movies). cut short, it turns out to be an optimization problem and there are method available such as gradient decent (slow and costly in this case) and another is ALS algorithm.
So, ALS is an algorithm which you can use to solve the learning problem a.k.a build recommendation systems.
Please read here more-
http://stanford.edu/~rezab/classes/cme323/S15/notes/lec14.pdf


in binary classification where class labels are {-1, 1} is preprocessing needed?

You need {0,1} or {-1,1} labels depending on the output of your model. If you have a Sigmoid output use {0,1}, while TanH outputs work with {-1,1}.
No label choice is inherently right or wrong, as long as it's compatible with your model architecture and gives you good results.

EDIT:
In case of logistic regression you must use {0,1}, that is because this class of models has a Sigmoid output. Sigmoid function is always bounded in [0,1] and it can't take values outside of that range. It could never reach -1 and training won't work.


is learning rate schedule a hyperparameter?

Hyper-parameters are essentially parameters that you have to tune manually and aren't learned.

The schedule can be considered a hyper-parameter, as well as all the scheduler's parameters (patience, decay, etc.) and even the decision to use or not a scheduler can be considered a hyperparameter.


How to properly represent a tic tac toe board to a CNN?

I used a tic tac toe board to explain the value of convolutions in my master's thesis (https://apps.dtic.mil/dtic/tr/fulltext/u2/1046500.pdf checkout page 21-23 if you are interested).  I think this illustration (extracted from my thesis) might answer the question on how you could represent the board to the CNN and what you hope your filters might learn through training. Presumably you could train the network to recognize playable spaces and predict best X/O placement as you add more layers to your network.  While this doesn't tell you how to do this, I think my diagram will be useful in developing what type of input you should provide your network and some of the dimensions in order to answer your question.


Which algorithm can i use for predicting length of stay in coming year based on historical claims data?

If you want to do prediction using 2011 features, the answer is yes, you can do that.

However, as you don't want to use these features, the answer might be no.

Without using 2011 features, your dataset will have only 2 samples(2009 and 2010) under the assumption that every memberID is different. Prediction from two samples is neither reliable nor feasible.


Ways of calculating the area of colored regions in a map

Finding the size of colored regions might be better framed as a convex hull  problem since the borders are distinct. After each convex hull is defined, the region can be defined in pixels.


How can one can aim for a career in machine learning?

Machine Learning is a really big field. Depending on what exactly you want to do, there might be huge differences.

Having said that, the following skills are helpful:

Programming, especially Python / C++
Frameworks like TensorFlow, sklearn, Torch, ...
Algorithms like neural networks (especially gradient descent), SVM, decision trees, clustering algorithms, Q-Learning, ...
Mathematics (e.g. probability theory / statistics, linear algebra, analysis for calculating gradients

Knowing how to program the GPU directly (CUDA) is probably not always necessary, but certainly a big bonus.

In case you're interested what you learn at KIT (Karlsruhe, Germany) in an introductory machine learning course, see my blog post for exam preparation (I'm sorry, it is mostly in German).

Online Courses

There are lots of online material for machine learning. For example an Coursera ML course


Let $x \in \mathbb{R}^{d}$ be a feature vector (one row of your table).
Your data has the problem that the distribution per dimension is very different, in particular $x_{1} \in \{0,1\}$ and $x_{d} \in \mathbb{R}$.
Using K-Means means to consider each data point in the euclidean space where the similarity (or distance) is measured. However, the main problem arises when using the euclidean space in your context!
If we measure a distance in the euclidean space for $x,y \in \mathcal{X}$, then $||x-y|| = \sqrt{\sum_{i = 1}^{d}(x_{i}-y_{i})^{2}}$. To simplify the examination, let us now consider the squared euclidean loss $||x-y||^{2} = \sum_{i = 1}^{d}(x_{i}-y_{i})^{2} $ and let $e_{i}:= (x_{i}-y_{i})^2$ be the squared deviation per dimension.
Then $||x-y||^{2} =\sum_{i = 1}^{d}e_{i}$. For a column $i$ with binary values we have $e_{i} \in [0,1]$. However, for the last two columns we have $e_{i} \in \mathbb{R}$.
Therefore, the last two columns have already a higher weight on the distance calculation $||x-y||^{2} =\sum_{i = 1}^{d}e_{i}$, which might not represent the true importance of these dimensions. Even more, by using the euclidean space you implicitly assume that the importance of the features are comparable to each other, which might not be the case.
So what are your options:
1.) Min-Max scaling (so that also the last two dimensions are in $[0,1]$).
You can do that, but you should be aware of the impact: You could say from the perspective of the distance measurement, all dimensions have now equal weight, which might be misleading again.
2.) Standardizing your data
You can do that. It is often better than min-max scaling as you maintain the distribution per dimension. Still you should be aware of the impact: You could say from the perspective of the distance measurement, all dimensions still have kind of an equal weight, which might be misleading again.
3.) You can perform L2 normalization and use the cosine similarity.
It results in the same issues..
Therefore, if the importance of some features is very different, or if the importance of different features cannot be determined, e.g. they are not comparable, using K-Means (or measurements in the euclidean space) is  basically a bad idea.
4.)
Recall that Random forest is invariant to these issues: If $\mathcal{X}$ is a set of data points that is classified by an inner node of a decision tree that has been trained by the random forest algorithm, then the algorithm provides a threshold $\theta \in \mathbb{R}$ and a dimension $d' \in \{1,\ldots,d\}$. The inner node splits the data into $\{x \in \mathcal{X} \mid x_{d'} \leq \theta \}$ and $\{x \in \mathcal{X} \mid x_{d'} > \theta \}$.
In other words, it learns at each inner node of a decision tree a weak classifier that is applied to a single dimension, thereby avoiding to define a comparable importance weight for each dimension.
You can use a random forest to build a distance measurement (see Wikipedia), and apply K-Means afterwards.
5.) Use K-Means variants.
6.) Use PCA to project your data to a space $\mathbb{R}^{d'}$ with $d' < d$, then apply K-means.
7.) Use an autoencoder to learn a better embedding. Then apply K-means.


What are some key strengths of BERT over ELMO/ULMFiT?

The reason you're seeing BERT and its derivatives as benchmarks is probably because it is newer than the other models mentioned and shows state-of-the-art performance on many NLP tasks. Thus, when researchers publish new models they normally want to compare them to the current leading models out there (i.e BERT).
I don't know if there has been a study on the strengths of BERT compared to the other methods but looking at their differences might give some insight:

Truly Bidirectional
BERT is deeply bidirectional due to its novel masked language modeling technique. ELMo on the other hand uses an concatenation of right-to-left and left-to-right LSTMs and ULMFit uses a unidirectional LSTM. Having bidirectional context should, in theory, generate more accurate word representations.

Model Input
BERT tokenizes words into sub-words (using WordPiece) and those are then given as input to the model. ELMo uses character based input and ULMFit is word based. It's been claimed that character level language models don't perform as well as word based ones but word based models have the issue of out-of-vocabulary words. BERT's sub-words approach enjoys the best of both worlds.

Transformer vs. LSTM
At its heart BERT uses transformers whereas ELMo and ULMFit both use LSTMs. Besides the fact that these two approaches work differently, it should also be noted that using transformers enables the parallelization of training which is an important factor when working with large amounts of data.

This list goes on with things such as the corpus the model was trained on, the tasks used to train and more. So while it is true that BERT shows SOTA performance across a variety of NLP tasks, there are times where other models perform better. Therefore, when you're working on a problem it is a good idea to test a few of them a see for yourself which one suits your needs better.


Some real important differences to consider when you are choosing R or Python over one another:

Machine Learning has 2 phases. Model Building and Prediction phase. Typically, model building is performed as a batch process and predictions are done realtime. The model building process is a compute intensive process while the prediction happens in a jiffy. Therefore, performance of an algorithm in Python or R doesn't really affect the turn-around time of the user. Python 1, R 1.
Production: The real difference between Python and R comes in being production ready. Python, as such is a full fledged programming language and many organisations use it in their production systems. R is a statistical programming software favoured by many academia and due to the rise in data science and availability of libraries and being open source, the industry has started using R. Many of these organisations have their production systems either in Java, C++, C#, Python etc. So, ideally they would like to have the prediction system in the same language to reduce the latency and maintenance issues.
Python 2, R 1.
Libraries: Both the languages have enormous and reliable libraries. R has over 5000 libraries catering to many domains while Python has some incredible packages like Pandas, NumPy, SciPy, Scikit Learn, Matplotlib. Python 3, R 2.
Development: Both the language are interpreted languages. Many say that python is easy to learn, it's almost like reading english (to put it on a lighter note) but R requires more initial studying effort. Also, both of them have good IDEs (Spyder etc for Python and RStudio for R). Python 4, R 2.
Speed: R software initially had problems with large computations (say, like nxn matrix multiplications). But, this issue is addressed with the introduction of R by Revolution Analytics. They have re-written computation intensive operations in C which is blazingly fast. Python being a high level language is relatively slow. Python 4, R 3.
Visualizations: In data science, we frequently tend to plot data to showcase patterns to users. Therefore, visualisations become an important criteria in choosing a software and R completely kills Python in this regard. Thanks to Hadley Wickham for an incredible ggplot2 package. R wins hands down. Python 4, R 4.
Dealing with Big Data: One of the constraints of R is it stores the data in system memory (RAM). So, RAM capacity becomes a constraint when you are handling Big Data. Python does well, but I would say, as both R and Python have HDFS connectors, leveraging Hadoop infrastructure would give substantial performance improvement. So, Python 5, R 5.

So, both the languages are equally good. Therefore, depending upon your domain and the place you work, you have to smartly choose the right language. The technology world usually prefers using a single language. Business users (marketing analytics, retail analytics) usually go with statistical programming languages like R, since they frequently do quick prototyping and build visualisations (which is faster done in R than Python).


Is there a way to standardize latitude and longitude to be used as predictors in KNN algorithm?

Rather than modifying your data, use great circle distance for those features' conmponent of the cost function.


Can regression trees predict continuously?

Regression trees, particularly gradient boosting (essentially many trees), tend to do very well on continuous predictions, often outperforming models that are truly continuous like linear regression when. This is especially true when there are variable interactions and when you have a sufficiently large dataset (over 10,000 records) so that overfitting is less likely. If your primary objective is simply predictive power, then whether the model is 100% continuous or pseudo continuous should be irrelevant. If making your regression trees more continuous enhances out of sample predictive power, then you can simply increase tree depth or add more trees.


Just tried your cnn_model function and it works correctly for me.
Not sure about what your problem is, but it seems to be your X and Y. X must not have shape (2023, 65), but shape (datasize, 2023, 65). and Y should have shape (datasize, 1) as well. otherwise your network treats X as 2023 inputs of shape (None, 65) instead of treating it as n inputs of shape (None, 2023, 65).
here is my code in case you wanna run it (important part is the definition of X(ims) and Y(lab) :
import keras
from keras.layers import Conv1D, Dropout, MaxPooling1D, Flatten, Dense
import numpy as np

ims = np.random.randn(100, 2023, 65)
lab = np.random.randn(100, 1)

def cnn_model(X, Y):
    model = keras.Sequential()
    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
    model.add(Dropout(0.25))
    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))
    model.add(Dropout(0.25))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(64, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='mse', optimizer='adam', metrics=['mse', 'mae'])
    print('training')
    model.fit(X, Y, epochs=10, batch_size=32, verbose=0)
    print('finished training')
    model.evaluate(X, Y)
    return model

m = cnn_model(ims, lab)
m.summary()


In general the advantage of repeated training/testing is to measure to what extent the performance is due to chance. The most common source of chance comes from which instances are selected as training/testing data. One can use k-fold cross-validation in order to mitigate the effect of chance in this case. Weka performs 10-fold CV by default, as far as I remember, but this is not compatible with providing a specific training/test set.

[edit based on OP's comments]

In the video mentioned by OP, the author loads a dataset and sets the "percentage split" at 90%. This means that the full dataset will be split between training and test set by Weka itself. Weka randomly selects which instances are used for training, this is why chance is involved in the process and this is why the author proceeds to repeat the experiment with different values for the random seed: every time Weka will selects a different subset of instances as training set, resulting in a different accuracy. In other words, the purpose of repeating the experiment is to change how the dataset is split between training and test set. In this case (J48 with default options) there would be no point repeating the experiment with a fixed training set, because there's no chance involved in the process so there's no variation in the result.

It's worth noticing that this lesson by the author of the video seems to be used as an introduction to the more general concept of k-fold cross-validation, presented a couple of lessons later in the course.


How to use a loss functions closed-form gradient even when it does not have a closed-form itself

TensorFlow allows for custom gradient functions with tf.custom_gradient.

You could write a decorator that would return specific gradient values for specific values of x. The loss function would not need to be evaluated.


Having and issue plotting horizontal chart

You have to change one line in your code.

This

plt.barh(bars, percentage, color='lightskyblue', edgecolor='blue')

should become this

plt.barh(range(len(bars)), percentage, color='lightskyblue', edgecolor='blue')

What you are changing is the y position of your bars. If you look at your original bars there are only 3 values there, so bars were overlapping.

Once you change that to  range(len(bars)) you are using the same tick values that you set in plt.yticks(range(len(bars)), new_labels).