How to plot two distribution curves in a faceted way in R/ggplot2?

I would separate the creation of the data from the plotting operation, if I were you. ggplot2::facet_wrap allows you to plot each of the cumulative distributions in each own panel. Here is how I would do it.

pgammaX <- function(x) pgamma(x, shape = 64.57849, scale = 0.08854802)

Gamma <- pgammaX(seq(from=4, to = 8, by = 0.1))
Normal <- pnorm(seq(from=-4, to = 3.9, by = 0.1))

data <- data_frame(
  x = seq(-4,8,by=0.1),
  y = c(Normal, Gamma),
  # use factor to make the plot for Normal to appear first
  distribution = factor(c(rep(c("Normal", "Gamma"),
                          times = c(length(Normal), length(Gamma)))),
                          levels = c("Normal", "Gamma"))
)

library(dplyr)
library(ggplot2)

data %>%  ggplot(aes(x, y)) +
  geom_line() +
  # use facet_wrap to plot each cummulative distribution in each own panel
  facet_wrap(~distribution, scales = "free_x")


"Higher dimensional representation" refers to having more feature maps like you suggested (check the inception module proposed in figure 7).

As to what disentangled features mean, I believe they mean decorrelated: the more features (with different filters) the inception module extracts, the more and the faster the network learns. Because the network will have access to more information leading to detecting which features are salient earlier in the training.


Spectral clustering with heat kernel weight matrix

$\sigma$ represents a typical distance between points. If all points of one cluster of your graph are separated from all points of another by a distance that is significantly higher than $\sigma$, then the spectral clustering will probably use this as a cut.

If you already know the number of clusters (or cuts) that you want to make, $\sigma$ does not need to be tuned very finely. But it becomes really important if you have no idea how many cuts you want to make.

A good approach is to make a parametric study over $\sigma$, and make the decision through the eigen values. Here's an example from one of my study cases, giving eigen values (in ascending order) of the Laplacian matrix, for 4 different values of $\sigma$:

What you are looking for is a break in the increase of eigen values: theoritically, a wide gap between two consecutive values ($n$ and $n+1$) should tell you that it is a good idea to make $n$ clusters.

As you can see, small sigma values (upper left plot) lead to very low eigen values, sometimes even numerically negative. Very small values tend to make very small clusters containing outliers, and a very big cluster with all other points. High sigma values (lower right plot) will make your matrix look like the identity, with high eigen values, this usually won't bring anything interesting.

What I usually do is try a wide window for sigma values, and shorten the range progressively, until I find a satisfying result. This is all graphical, and depends on the prior knowledge about your problem (how many clusters do you approximately expect? 2, 10, 100?).

Eigen values represent the quality of the cut. Small values correspond to very distant clusters. For instance, in my upper right case, I could make 4 very well separated clusters (first 4 eigen values are almost 0). The next cut (to reach 5 clusters) would be less effective than the 3 previous ones. 13 clusters could also have been a good try. But I actually selected 9 clusters, because it was closer to what I actually expected.


Output landscape of ReLU, Swish and Mish

From the article you cite, I have found this figure which looks similar to yours.

On the same page of the article, they say:

Having a smooth profile also plays a role in better gradient flow, as
shown in Fig. 3, where the output landscapes of a five-layered randomly
initialized neural network with ReLUand Mish are visualized.  The
landscapes were generated by passing in the co-ordinates to a
five-layered randomly initialized neural network which outputs the
corresponding scalar magnitude. The output landscape of ReLU has a
lot of sharp transitions as compared to the smooth profile of the
output landscape of Mish.

It seems your figure (and the one above) does not show the loss landscapes (which is the figure 4 in the article) but the landscape of the global neural network using different activation functions. I guess the neural network they used to produce these figures has 2 inputs (the coordinates of the plots), 3 hidden layers and one scalar output. The color in the plots corresponds to the scalar output value.
The authors present these figures because :

Smoother output landscapes suggest smooth loss landscape [...]

This is my understanding of the article, hope it helps.


where to start in natural language processing for a language

@Amar, you can start with NLTK, Spacy, Flair and Polyglot to start with your NLP learning. They are easier to start and understand. You can create your own model for your language and contribute if you have the corpus available with you. The nouns, verbs are easier to capture in English when compared to Chinese, Korean and Arabic. Every language has it own beautiful representation of context. You would a good library that can read the language that you are developing. You can also try IBM Watson and AWS SageMaker on trial basis with check for Language detection.


Can one perform Feature Selection on a subset of training data?

After finding THE best subset doesn't make sense. Your approach is not correct in this way.
If you cannot handle all data you may have to follow one of approaches below:
1- Perform clustering and chose 10% (the more the better, based on what your computer can handle) from each cluster so all 10 subsets may have similar distribution. As a result, the models theoretically should have similar performance. This approach is useful if you can find clusters in your data. Don't forget to split each subset to training/test subsets!
2- If no clusters found, develop 10 models for each subset and use the average of outputs (bagging), and see how it works.
Hyper parameter tuning is an expensive task, even for small datasets; so it may not be a good idea to do it using not 100% but 70-80% of your data as the training set.
Another thing that you should consider is the ML algorithm you choose. Random Forest may or may not be a good choice. So try different algorithms on small subsets to have an idea which algorithm may work better for your problem. Then think about hyper parameter tuning and feature selection.


num units is the number of hidden units in each time-step of the LSTM cell's representation of your data- you can visualize this as a several-layer-deep fully connected sequence of layers in which each layer also has a connection to a memory across the layers,even though that a analogy isn't 100% perfect. num units, then, is the number of units in each of those layers.

For your specific problem, and with length = 1, this reduces to a single layer- your model is not taking advantage of the memory capabilities of LSTM because there's simply nothing to remember beyond a single time step, because there's only a single time-step.

As to why it seems to perform better with a higher num units-- your model doesn't have enough expressive power at num units = 1 to describe the function it's trying to- there is too much information to encode in only a single unit. At num units = 700, I'd be very curious as to what the performance of the model is on out-of-sample data.

It's very likely that your network is simply over fitting, or memorizing your training data, with that much expressive power. If you can verify that that's happening- you'd see very good performance on the training data and poor performance on out-of-sample data- then you'll need to regularize the network in some way. Common ways to do this include implementing early stopping, introducing dropout layers, and/or reducing the number of hidden units in your layers.

Overall, if you don't have more than one time-step observation for a single entity, I would suggest that you change the LSTM layer to a simple fully connected layer to simplify the architecture. If you do or can get those sequential observations, then an LSTM layer might be a good solution for you.


Try content based filtering, you can cluster candidates based on similarity of keywords and by assigning similarity scores. Check tf-idf/cosine similarity measures. Additionally you can customize your algorithm by giving extra weight for section wise similarities since your data is very well structured.

Also check collaborative filtering on Mahout.


What is the job of "RepeatVector" and "TimeDistributed"?

tf.keras.layers.RepeatVector

According to the docs :

  Repeats the input n times.

They have also provided an example :

model = Sequential()
model.add(Dense(32, input_dim=32))
# now: model.output_shape == (None, 32)
# note: `None` is the batch dimension
model.add(RepeatVector(3))
# now: model.output_shape == (None, 3, 32)

In the above example, the RepeatVector layer repeats the incoming inputs a specific number of time. The shape of the input in the above example was ( 32 , ). But the output shape of the RepeatVector was ( 3 , 32 ), since the inputs were repeated 3 times.

tf.keras.layers.TimeDistributed()

According to the docs :

  This wrapper allows to apply a layer to every temporal slice of an input.
  The input should be at least 3D, and the dimension of index one will be considered to be the temporal dimension.

You can refer to the example at their website.

TimeDistributed layer applies a specific layer such as Dense to every sample it receives as an input. Suppose the input size is ( 13 , 10 , 6 ). Now, I need to apply a Dense layer to every slice of shape ( 10 , 6 ). Then I would wrap the Dense layer in a TimeDistributed layer.

model.add( TimeDistributed( Dense( 12 , input_shape=( 10 , 6 ) )) )

The output shape of such a layer would be ( 13 , 10 , 12 ). Hence, the operation of the Dense layer was applied to each temporal slice as mentioned.


Should features be correlated or uncorrelated for classification?

Q1) Should highly correlated features with the target variable be included or removed from classification and regression problems? Is there a better/elegant explanation to this step?

Actually there's no strong reason either to keep or remove features which have a low correlation with the target response, other than reducing the number of features if necessary:

It is correct that correlation is often used for feature selection. Feature selection is used for dimensionality reduction purposes, i.e. mostly to avoid overfitting due to having too many features / not enough instances (it's a bit more complex than this but that's the main idea). My point is that there's little to no reason to remove features if the number of features is not a problem, but if it is a problem then it makes sense to keep only the most informative features, and high correlation is an indicator of "informativeness" (information gain is another common measure to select features).
In general feature selection methods based on measuring the contribution of individual features are used because they are very simple and don't require complex computations. However they are rarely optimal because they don't take into account the complementarity of groups of features together, something that most supervised algorithms can use very well. There are more advanced methods available which can take this into account: the most simple one is a brute-force method which consists in repeatedly measuring the performance (usually with cross-validation) with any possible subset of features... But that can take a lot of time for a large set of features.

However features which are highly correlated together (i.e. between features, not with the target response), should usually be removed because they are redundant and some algorithms don't deal very well with those. It's rarely done systematically though, because again this involves a lot of calculations.

  Q2) How do we know that the dataset is linear when there are multiple variable involved? What does it mean by dataset being linear?

It's true that correlation measures are based on linearity assumptions, but that's rarely the main issue: as mentioned above it's used as an easy indicator of "amount of information" and it's known to be imperfect anyway, so the linearity assumption is not so crucial here.

A dataset would be linear if the response variable can be expressed as a linear equation of the features (i.e. in theory one would obtain near-perfect performance with a linear regression).

  Q3) How to do feature importance for nonlinear case?

Information gain, KL divergence, and probably a few other measures. But using these to select features individually is also imperfect.


How hidden units are conditionally independent in restricted Boltzmann machines

It would be too lengthy to do the derivation here. Would highly recommend watching this video, where Hugo Larochelle has derived the conditional independence of hidden units in RBMs.

Hope this video answers all of your questions, which seem to spur from conditional independence of RBMs. If not, it would be helpful if you could elaborate your question and provide some more context.


sklearn: sklearn.linear_model.HuberRegressor vs sklearn.linear_model.ElasticNet

Yes, many loss functions in regression models are using a combination of L1 and L2 for different purposes. To realize the difference, I start with Ridge regression.

Ridge regression addresses some of the problems of Ordinary Least Squares by imposing a penalty on the size of the coefficients. The ridge coefficients minimize a penalized residual sum of squares.

The HuberRegressor is different to Ridge because it applies a linear loss to samples that are classified as outliers. A sample is classified as an inlier if the absolute error of that sample is lesser than a certain threshold.

ElasticNet is a linear regression model trained with both $\iota_1$ and $\iota_2$-norm regularization of the coefficients. This combination allows for learning a sparse model where few of the weights are non-zero like Lasso, while still maintaining the regularization properties of Ridge. Elastic-net is useful when there are multiple features which are correlated with one another. Lasso is likely to pick one of these at random, while elastic-net is likely to pick both.


One way would be to create 20 features (each feature representing a codon). In this way, you would have a dataset with 3190 instances and 20 categorical features. There is no need to treat the sequence as a Markov chain.

Once the dataset has been featurized as suggested above, any supervised classifier can work well. I would suggest using a gradient boosting machine as it might be better suited to handle categorical features.


How to model manufacturing shift data with irregular production times?

From the information at hand, you could break this down into two problems -

Predicting the production for the shift, and
Finding the probability of breakdown during the shift

For (1) you could either go down the time-series route (ARIMA, Box-Jenkins, Exponential smoothening) or the regression route (provided you have good features)

For (2) you could build a logistic regression model that gives you a probability score

Possible features you could look into are

Time of day, week, month and year
Time between breakdowns
Available labor
Electricity Board data?


Batch Normalization will disrupt multi-threading?

My experience with batch and layer normalisation is that it is around 3 to 5 times slower per step than without it. Although the model may converge in less number of steps (which is what the authors report in the article) it takes much longer to calculate each step, and hence the whole training takes longer.

This has been reported in a few places like:

https://www.reddit.com/r/MachineLearning/comments/4ufmxy/layer_normalization_implemented_in_tensorflow/

https://github.com/cooijmanstim/recurrent-batch-normalization/issues/2

https://github.com/keras-team/keras/issues/1309


Identifying importance of each feature in deep model

This could mean that your features are heavily correlated. As a result, when you set one variable to 0, there is always another one that carries the same information.
Try to set several features to 0 to see if you have the same issue. If yes, then it might be something else (maybe your code).

You might also be interested in this paper, you'll find several methods to assess feature importance.


Language translation with convolutional neural network

I just googled:

A Convolutional Encoder Model for Neural Machine Translation, by Gehring et al., link
Convolutional Sequence to Sequence Learning, by Gehring et al. link
Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction, by Elbayad et al. link

All the implementations I found on GitHub are in PyTorch. I'm not surprised I didn't find much: the application of CNNs to NLP was interesting, but they never beat RNNs at that. After the advent of Attention models, and especially Transformers, these kind of models have not been developed further. In fact the latest paper I linked you is from 2018 - i.e. an Ice Age ago for the pace of NLP.

If you really want to go deeper on this topic of Convolutional NMTs, I suggest you to check the available torch-based code and try to make a replica in Tensorflow/Keras. It's hard work, but a fancy model nonetheless. Good luck!


Add extra term weight when grouping strings by similarity?

When you have a similarity measure fixed already, you can use correlation clustering to turn your similarity matrix into groups as you wanted. Here's a good resource about correlation clustering http://www.francescobonchi.com/CCtuto_kdd14.pdf

For the second step to attempt to make sure volume 1 and volume 2 are not the same, I would add a number of rules that try to find the difference. Look for numerals, if they don't match they have 0 similarity. Play around a bit with this. After you have your new similarity matrix you can use correlation clustering which should seperate volumes 1 and 2 into different groups.


mysql could not recieve table from Citespace

The problem may be is that you created data schema manually. You can erase wos database and just use CiteSpace. It creates dataschema automatically. Hope it will be OK.


Using of 100s of Binary features in regression model

An approach worth exploring for dimensional reduction of these binary features would be Multiple Correspondence Analysis (Python libraries such as Prince and mca are available), which is similar to PCA but deals with nominal categorical variables. Here is an excellent answer on applicability of PCA and factor analysis to binary features and another one on the same topic.


Error while using the pandas_datareader package

BEFORE you import pandas_datareader, run this:

pd.core.common.is_list_like = pd.api.types.is_list_like

where pd stands for your pandas import.

Let me know if this works :)


In XGBoost would we evaluate results with a Precision Recall curve vs ROC?

I am answering my own question, almost two years later.  XGBoost now has a new eval metric aucpr.

https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters


Neural Network written in Kotlin working for simple math problems but not for MNIST classification

I managed to solve the problem on my own. Turns out that the problem was indeed in the hyper-parameters, as I suspected.
I was using the same values as Michael Nielsen so it seemed strange to me that they could be wrong - by a mathematical point of view, my network is exactly the same as his one! - anyway, lowering the $\eta$ and raising the $\lambda$ (in practice, lowering the weights/biases changes on a single batch) lets the network to better understand which direction it should take to minimize the cost function.
I can now get accuracies above 85%, and I'm sure that with a bit of fine-tuning I can reach at least 95%.
I also solved the cost plot problem:

I can't understand why the cost is increasing instead of decreasing [...]

The problem was that instead of doing something like
$C= -{1 \over n} ∑_{xj} [y_j ln(a^L_j) + (1−y_j) ln(1−a^L_j)] + {λ \over 2n} ∑_w w^2$
I was doing
$C= -{1 \over n} ∑_{xj} [y_j ln(a^L_j) + (1−y_j) ln(1−a^L_j) + {λ \over 2n} ∑_w w^2]$
which doesn't make sense, since the last term equals to
$∑_{xj} {λ \over 2n} ∑_w w^2 = n {λ \over 2n} ∑_w w^2 = {λ \over 2} ∑_w w^2$
and so we get a wrong regularization factor for the weights.


Predict items customers would buy in next order

Try association rules, they will help you in finding the right set of rules like what is chance of customer buying X if he purchased A, B, C in the past.

Also from my personal experience, these kinds of tasks are more business need driven. Like sales capture from the item. So even if your model is 60% accurate but captures 90% sales, you should be good to go. Try to give it a thought in this way.


Autoencoders for the compression of time series

Above all, you should take care of the time series.
Despite from that, AEs are thoroughly used for time series, especially LSTM+AE. Did you vary the topology? For the CAE it looks reasonable but the other models lack some layers, or?
Furthermore, some regular advices would be to standardize the input, change the activation functions (tanh worked well for me in the output layer) as well as the number of neurons per layer and the amount of layers in general.
Could you provide the head() of the input data? An AE expects to fit X on X, maybe you missed that?


How to compare number of occurrences in two data frames for a column and extract the similarities

EDIT: Thanks for you clarifying the question. So now the problem is checking the counts of ids in two data frames are the same.
Here is how you could go about it:
d1 = pd.DataFrame(df1[~df1['A'].isnull()].groupby("id").size())
d2 = pd.DataFrame(d[~d['A'].isnull()].groupby("id").size())

d = pd.merge(d1,d2,on="id")

ids_ = d[d["0_x"] == d["0_y"]].index.values

RETURN:
array([1, 3])

This will now give an array of ids where the counts in both df1 and d are the same.


Clustering with groups in data related to cluster label

If you have apriori known labels, then use the labels, not clustering.

Clustering is quite difficulty and fragile, so always prefer a simpler solution if you can.

It's not very clear what you are trying to solve. Select columns to split the data into rooms? Just select the columns then.


Pandas dataframe, create columns depending on the row value

First we use DataFrame.explode to unnest your lists to rows.

Then we use DataFrame.pivot_table to pivot your dataframe from rows to column to get your desired result:

dfn = df.assign(countries=df['countries'].str.split(',')).explode('countries')
dfn['numbers'] = df.assign(numbers=df['numbers'].str.split(',')).explode('numbers')['numbers']

dfn = (
    dfn.pivot_table(index=dfn.index,
                    columns='countries',
                    values='numbers',
                    aggfunc=lambda x: x,
                    fill_value=0)
       .rename_axis(None, axis='columns')
       .add_suffix('_number')
)

Output

    ch_number de_number dk_number fr_number
abc         2         1         0         3
bcd         4         0         5         3


Retrieving date ranges with the r platform

If louisville_weather$DATE is formatted as POSIXct, the month can be extracted, thanks @RockScience, with

lvmonth <- format(louisville_weather$DATE, "%m")

It's a character variable.  So, filtering

louisville_weather <- louisville_weather[lvmonth == "01", ]

should extract your weather for January.

Hope you'll find it cold enough to balance the warmth of datascience.SX !


Why is oversampling outperforming class weight?

It could be your sampling strategy

If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint will be seen over and over.

You could try another oversampling strategy, for example, SMOTE or ADASYN. These techniques create data points that are closets to decision boundaries so you are less inclined to overfit on "easy" data points.

Another things you can try is oversampling the minority class and undersampling the majority class that the same time. When choosing a method to do this, pick one that can oversample near decision boundaries and undersample away from decision boundaries. For example, here is SMOTETomek. Notice how classes purple and green get mainly oversampled and class yellow mainly undersampled.

These images come from imbalanced-learn which is a Python package you can use for all these sampling strategies.

It could be your pipeline

If you use your oversampled data for testing your model performance, you could be (unwillingly) manipulating your results. You need to ensure that you use your augmented data only for training, and not for validation and testing.

          +-> training set ---> data augmentation --+
          |                                         |
          |                                         +-> model training --+
          |                                         |                    |
all data -+-> validation set -----------------------+                    |
          |                                                              +-> model testing
          |                                                              |
          |                                                              |
          +-> test set --------------------------------------------------+


What is an alternative name for "Unstructured Data"?


t-SNE is notoriously fickle.  I would advise you read this short article How to Use t-SNE Effectively before you invest too much time trying to interpret the plots.  In case you don't read it, you'll find that your t-SNE plot will change considerably based on your iterations, and perplexity.  You'll find that distance between clusters aren't accurate and that cluster sizes aren't accurate too.  t-SNE is a cool data reduction too, but often takes multiple runs to get a "good" plot.  Laurens van der Maatan even suggest that running multiple iterations and picking the one with the lowest KL-divergence is perfectly okay.  This is due to the fact that t-SNE is non-convex so you won't necessarily find a global minimum.


Should I scale data before or after balancing dataset?

Scaling, in general, depends on the min and max values in your dataset and up sampling, down sampling or even smote cannot change those values. So if you are including all the records in your final dataset then you can do it at anytime but, if you are not including all of your original records then you should do it before upsampling.


Different representations of dendrograms

This output represents the dendogram as a tree. The innermost parentheses represent the deepest parts of the tree. For instance the top (root) of the tree start with the pair K5 and a subtree, then this subtree is made of another subtree and K4, and so on.
If we ignore the numerical values (distances I assume?) we have this:
(K_5,
  (
    (K_1,
       (
         K_2,K_3
       )

    )
    K_4
  )
)

Which represents this tree:
 --------------------
 |                  |
 |            -------------
K_5           |           |
           -------       K_4
           |     |
          K_1  -----
               |   |
              K_2 K_3

Then it can be converted to the desired format:
[K_1 , K_2 , K_3 , K_4 , K_5]
[K_1 , K_2 , K_3 , K_4] [K_5]
[K_1 , K_2 , K_3] [K_4] [K_5]
[K_1] [K_2 , K_3] [K_4] [K_5]
[K_1] [K_2] [K_3] [K_4] [K_5]


How to build a network that detects common object in images?

You probably just need a deeper network to learn the cross test example. Convolutional layers need to have multiple feature channels and be stacked in order to learn anything more complex that edge/corner counting. Look at example networks for e.g. MNIST and you will usually see typically 2 or 3 CNN layers, with multiple channels (e.g. 32 or 64 channels, each of which is associated with previous layer channels amount of NxN kernels) plus 1 or 2 fully connected layers before classifier.

For a binary choice, yes train on object present/not present classes. There is no major difference between a single sigmoid output or softmax with two outputs. Use cross-entropy loss function, multiclass version if you use softmax.

I suggest just adapt an MNIST example network to your problem by changing input shape and number of outputs. So that would use softmax and multiclass cross-entropy loss.

  Is it a problem to have a large MaxPooling layer

It could work in some situations, but by having a single layer, single channel CNN, you have made the network too simple to classify even moderately complex shapes. Reducing the activations from this layer to just the max is going to limit the network to essentially classifying by strongest matching 5x5 template to the filter in the image. With enough training examples it should be able to create such a filter, but it would not cope very well with noise, rotation or any partial matches.

You might get something more effective with a large pooling layer over say a 2-deep multi-channel CNN, and then a small fully-connected layer over those max channel outputs. It would still be an unusual choice of architecture, but I suspect it could learn problems like your cross test.


Unsupervised clustering of unstructured text by document type

"pathology report, doctor visit notes, prescription orders" these are classes, not clusters. Clustering may as well find "people with headache", "patients on Thursdays", ... you do not get to control this unsupervised.

So I'd rather suggest to do a classification.

For example, put all documents that contain "patient report" in the first lines into a separate set. Inspect the remainder for another keyword. Add another rule. If the data is as explained, 10 such rules may get the leftover documents down to a manageable scale, and they could eventually just be treated as 'others'.

There is nothing wrong with inspecting documents and adding rules if that solves your problem. You probably don't need to make this work on different document collections (say, law texts) ever.


Newton's method optimization for Deep Learning

If you take a look at section 2, it says

  The central idea motivating Newton’s method is that $f$ can be locally
  approximated around each $\theta$, up to 2nd-order, by the quadratic: $$ f(\theta + p) \approx q_\theta(p) \equiv f(\theta) + \nabla f(\theta)^Tp + \frac{1}{2} p^TBp \, \, (1) $$  where $B = H(\theta)$ is the
  Hessian matrix of $f$ at $\theta$. Finding a good search direction then reduces
  to minimizing this quadratic with respect to $p$.

To minimize, you need to take the derivative of (1) with respect to $p$ and set it to zero:

$$\Rightarrow \nabla f(\theta) + Bp = 0$$

which is equivalent to $Bp = -\nabla f(\theta)$.


What is the right algorithm to detect segmentations of a line chart?

One way to do what you're talking about is called "change point analysis." There is an R package for this called changepoint that you might want to check out.

In Python, you could try changefinder.


User profiling with Mahout from categorized user behavior

I think one solution for this might be using Matrix factorization, to perform some kind of collaborative Filtering. This way you will have not have to deal with the two concern points that you have, explicitly.

To do this, create a [User X Action] Matrix W, where rows are users and columns are actions. W(i,j) i.e. the Matrix entry on i row and j column would be the number of times the user i has performed action j. The matrix entries where no data is given are treated as missing data values by most collaborative filtering algorithms.

When you factorize this Matrix, you factorize it into two parts W= A*B. The A matrix will have rows are users and columns as latent dimensions. From this A, you get a feature vector for each User. You can then run clustering on these feature vectors to cluster the users. For more intuition about the factorization approach, look up "Matrix Factorization for collaborative Filtering" on google. Mahout has matrix factorization implementations that you can use.

Using the action categories is another matter altogether. It depends on what a category means. Does it mean that every action inside a category is very similar to each other or that they just have one or two aspect of similarity. A very easy approach would be to factorize a [User X (Action + Category)] Matrix. This can be solved using the same implementation mentioned earlier. You can also have more fancy schemes, like fixing part of the Matrix B to be very similar to the [Action X Category] matrix. But these schemes will require you to write optimization code yourself.


Model Validation accuracy stuck at 0.65671 Keras

I would like to see you data set :)
I am also doing some signal classification.

Unless there is some simple bug in data preprocessing stage: (check what you didn't show here first!)

As correctly pointed to you by @stefanLopez your number of filters is way too low.
Next, filter length is too short to capture anything serious.
Remove batchnorm while testing.
Reduce dropout while testing.
Test with ELU (Exponential Linear Unit) activation.
Last, use more FC layers with more neurons.
Try using glorot (commonly known as Xavier) initializer.

Example model:

model=Sequential()

model.add(Conv1D(filters=24,kernel_size=16,strides=1,padding='valid',activation='elu',kernel_initializer='glorot_normal',input_shape=X_train.shape[1::]))

model.add(Conv1D(filters=16,kernel_size=9,strides=1,padding='same',activation='elu',kernel_initializer='glorot_normal'))
model.add(Dropout(0.1))

model.add(Conv1D(filters=12,kernel_size=9,strides=1,padding='valid',activation='elu',kernel_initializer='glorot_normal'))
model.add(Dropout(0.1))

model.add(Flatten())
model.add(Dense(128,activation='elu'))
model.add(Dropout(0.1))
model.add(Dense(16,activation='elu'))
model.add(Dropout(0.1))
model.add(Dense(1,activation='sigmoid'))

Tell if it helps.


what's the motivation behind BERT masking 2 words in a sentence?

Because BERT accepts the artificial assumption of independence between masked tokens, presumably because it makes the problem simpler and yet gave excellent results. This is not discussed by authors in the article or anywhere else to my knowledge.
Later works like XLNet have worked towards eliminating such an independence assumption, as well as other potential problems identified in BERT. However, despite improving on BERT's results on downstream tasks, XLNet has not gained the same level of attention and amount of derived works. In my opinion, this is because the improvement did not justify the complexity introduced by the permutation language modeling objective.
The same assumption is made by other pre-training approaches, like Electra's adversarial training. The authors argue that this assumption isn’t too bad because few tokens are actually masked, and it simplifies the approach.


Equivalent of numeric encoding when rows can contain multiple values

The binary encoding seems to be the most natural, capturing exactly the present relationships and not introducing nonexistent relationships.  The combinations of names then are thought of as vertices of a hypercube; naturally, if you collapse this down to one dimension, you will be introducing nonexistent relationships.  Indeed, this is what happens with label encoding: Bob isn't really any closer to Alice than Dave is, but your label encoding makes that appear to be the case.

I've usually been happy enough with just the binary encoding, so I don't have much to back up the following, but some ideas:

For your case, you could still project the hypercube to one dimension; there are any number of ways to do it, and every one will add some additional information that shouldn't be there.  You could experiment and try to find one that seems most suitable for your purposes.

That should sound reminiscent of word embedding, and suggests another method: use a neural network to generate "entity embeddings", a projection from your hypercube to some smaller-dimensional space.  In this way, you uncover combinations of persons that are useful for your problem (maybe you end up with a feature that's "at least one of Alice and Dave", etc.).  You could also do something more transparent, and do some feature engineering by hand to pick out interesting combinations and binning to keep the number of features manageable.  For this you could consider target encoding each combination.  (Indeed, that's a very easy solution by itself, though target encoding can easily lead to overfitting if you have very many people or some rare combinations of people.)

Finally, you might be able to split a row containing (Alice, Dave) into two, one with Alice and the other with Dave.  You can now label-encode (or whatever else), and will need either a model that understands multiple rows per sample point, or a postprocessing method to deal with combining rows.


Dealing with a dataset where a subset of points live in a higher dimensional space

Why don’t you just put the data with pay_2 40k and 45k into two additional rows?

Additionally, you could add an indicator variable loc_1_2 and set it to 1 or 2 depending on the location.

In general, dummy encoding might be dangerous. Instead you could convert your addresses or locations into physical coordinates (latitude and longitude) or just distance from, let’s say, centre of the city or center of the country (depending on your data).

Edit

Ok, I see now that putting the data with pay_2 into additional rows will not work.

  But since it is a classification problem, it seems pertinent for the sake of numerical stability to standardise the salary data (pay2) to have values similar to a standard gaussian.

For your information: not all classification algorithms need standardised data, e.g. you may want to consider random forest or GBMs. If you stick to SVMs, regularised logistic regression or similar then I agree, it’d make sense to standardise.

  Thus, I could not set the NaNs to zero for the lower-dimensional observations as zero would be a meaningful value.

The standard approach you asked about, would depend on the properties of your classifier. I’m aware of the following options:

If you classifier can handle NaNs (NAs), e.g. R version of xgboost, then you can just use the data as you show in your table above.
If you have to standardise the data, then you can use mean-imputation. The average and standard deviation that are used for the standardisation should be estimated on non-NaN data, of course, (pay_2 column excl NaNs). In general, in order to prevent data leakage you should not use the data from holdout data subset (or “test”, “validation” or whatever you call it) for calculating the mean and standard deviation.
If your classifier doesn’t require standardisation, you can put some numerical sentinel values instead of NaNs, e.g. -1, 0, or -9999.

If you pick option 2 or 3, you’ll need to add a binary predictor indicating whether pay2 column was modified or not (NaN or not).

Re dummy encoding:
You haven’t asked about it explicitly and it’s difficult to suggest anything concrete without knowing the cardinality of your nominal variables, so I just put here the link to my answer about impact coding


Have you considered that the following steps will be even worse?

The standard algorithm for hierarchical clustering scales O(n³). You just don't want to use it on large data. It does not scale.

Why don't you do a simple experiment yourself: measure the time to compute the distances (and do the clustering) for n=1000,2000,4000,8000,16000,32000 and then estimate how long it will take you to process the entire data set assuming that you had enough memory... You will see that it is not feasible to use this algorithm on such big data.

You should rather reconsider your approach...


You don't need to convert the attributes to numeric data for DBSCAN nor for HAC (hierarchical clustering).

What you need is a distance function.

While there are some such as Gower's, these are just heuristics.

If you want really good results, you need to carefully design a proper distance function that is able to quantify how similar records are for your purpose. This is very much usage dependent, and hence you cannot find the answer already solved for you.


About building database for persian speech to text project

You may want to consider using Persian songs which have the lyrics (i.e. متن) readily available as well. There are also Islamic eulogies, known as noheh (نوحه), which are essentially poems recited somewhat similar to songs and sometimes include the written poem (i.e. نسخه).
Some examples:
https://bit.ly/2FRnI0g
https://bit.ly/3mKpj8W


artificial neural network for vehicular speed prediction based on volume

I think LSTM based RNN network having series of image data with a speed label, can be useful for training such model and you can try if you get some data set. Please do share your result as well.


I think hierarchical clustering would be more time efficient in your case (with a single dimension).
Depending on your task, you may implement something like this:

Having N datapoints di with their 1-dimension value xi:

Sort datapoints based on their xi value.
Calculate distances between adjacent datapoints (N-1 distances). Each distance must be assigned a pair of original datapoints (di, dj).
Sort distances in descending order to generate list of datapoint pairs (di, dj), starting from the closest one.
Iteratively unite datapoints (di, dj) into clusters, starting from beginning of the list (the closest pair). (Depending on current state of di and dj, uniting them means: (a) creating new cluster for two unclustered datapoints, (b) adding a datapoint to existing cluster and (c) uniting two clusters.)
Stop uniting, if the distance is over some threshold.
Create singleton clusters for datapoints which did not get into clusters.

This algorithm implements single linkage clustering. It can be tuned easily to implement average linkage. Complete linkage will be less efficient, but maybe easier ones will give good results depending on your data and task.

I believe for 200K datapoints it must take under second, if you use proper data structures for above operations.


How and Why to rescale image range between [0,1] and [-1,1]

You can use the following scalings

$$x’=\dfrac{x}{255} \qquad (1)$$
$$x’=\dfrac{x-127.5}{127.5} = \dfrac{x}{127.5}-1 \qquad (2)$$

for rescaling to $[0,1]$ or $[-1,1]$.

The rescaling of inputs tries to keep the range of weights in a small range. Theoretically it is not necessary to rescale your inputs because it can be compensated by an appropriate redefinition of your weights. Practically it is important because your weights might occupy a very large range of values.

In order to understand this I will construct a toy example with two inputs $x_1= 1$ and $x_2=1$ and a simple linear regression $y_n=w_1x_{1n}+w_2x_{2n}$. Assume the true values for the weight are $w_1=1$ and $w_2=1$. Now, assume we have the same data set but with different scales for $x_1$ and $x_2$ such that $x_1=10^k$ and $x_2=10^{-k}$. In order to obtain the same inputs we would need to have inverse weights resulting in $w_1=10^{-k}$ and $w_2=10^k$. Theoretically this is not a big deal but practically we see that for large values of $k$ we will need to use variables that store numerical values between $10^{-k}$ and $10^k$.

I don’t know the reason why the authors of the cited paper used specific transformations. Both should lead to the almost same performance as the bias should be able compensate for the additional $-1$ and the reacaling could be compensated by the weights for each input. There might be differences because of the stochasticity of many optimization techniques and because the final solution is very likely not the global optimum. I would choose one over the other if I want to compare my model with a specific model from a paper. In such a case I would use the same transformation as applied in the paper.

You could also use different scales in order to better discriminate the type of input (low resolution vs high resolution) by looking at the transformed values.


My instinct would say to start with a standard dataset and map all addresses ‒ both from your first and second datasets ‒ to that. In the standard dataset, each physical location would have a unique address.

The obvious source for the standard dataset is OpenStreetMap. You can query and dowload data from it using the Overpass API. Its query language, just like the whole OSM project, is well documented in my experience. Overpass Turbo is the user-friendly way with a graphical user interface to get started with the querying. Once you have developed a query, you can download data using e.g. wget, you just have to convert the query into the right URL at the Overpass API Convert Form by entering it into the window Overpass API Convert Form and selecting compact Overpass QL.

  wget --timeout=0 http://overpass-api.de/api/interpreter?data=[your query in compact Overpass QL form] -O filename

In OSM, some addresses are associated with buildings, which are ways (closed polygons), and some are nodes inside the buildings' outlines, so that's a bit of complication.

This all sounds doable. However, the OSM is not complete, so it very much depends on your area of interest whether it's worthwhile to use OpenStreetMap.


How is the "loss" calculated which is supplied by the callback log in Keras?

Yes, it is the loss function you pass to model.compile.

See here for loss metric usage examples. You can also define your own metric (see "Custom metrics" at the bottom of the page from the last link).

You can read about the Model class documentation here. The first method is the compile method with argument descriptions.

If you want to dig deeper into Keras callbacks and the built-in loss functions, check out these sources:

losses source code

Keras callbacks


Logistic Regression Model for categorical features with multiple values in each category

Looks like this is a great problem for catboost. Try it out

I believe, if I use this model on a policy which does not have categorical feature values on which it is trained, it will fail miserably. ANY data set that has big covariate shift will fail miserably.

NOTE read through docs of catboost there is part for feature selection.


First, some clarification: BPE does work with Chinese and Arabic.
The only problem with Chinese is that there are no blanks between words, and therefore there is no explicit word boundary. In order to address that problem, normally you would segment words before applying BPE. For that, the typical approach is to use Jieba or any of its multiple ports to other programming languages. Other languages without blanks, like Japanese, may have their own tools to perform word segmentation.
Now, the answer:
Subword vocabularies are the norm nowadays. The norm also consists of finetuning one of the many pre-trained neural models available (BERT, XLNet, RoBERTa, etc), and the tokenization is imposed by the model you choose.
BPE is, in general, a popular choice for tokenization, no matter the language. Lately, the unigram tokenization is becoming popular also.
I suggest you take a look at the recent tner library, which builds on top of Huggingface Transformers to make NER very easy.


Unsupervised learning for anomaly detection

You can use a density based method such as Local Outlier Factor in order to do this.

If you want to identify outliers in each column independent of other columns, then you would apply this method separately to each column. (Note that this is different from identifying outlier data points considering all columns together.)


Appropriate loss function for multi-hot output vectors

I have also been pondering on this question and trialled loss function on this sort of problem.
For these type of classification tasks, the loss function which seems most appropriate is Binary Cross Entropy Loss : https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a


Usually "eta" means learning rate, but it would be better if you could show an example.


Mask RCNN: Random predictions during inference for the same image

The problem was that some layers were not getting their weights loaded and were being initialised randomly. The random initialisation of those weights were the cause of random predictions during inference.


How best to use the resale transaction year in predicting housing prices?

I think you have identified the two main options:

Model the price trend, i.e. make your model fit to capture the price trend over time
Clean your [price] data so that prices are expressed in "real terms" (not including the price trend).

Option 2 would imply that you "deflate" a standardised price (e.g. price per square meter). Thus the price per sqm in 2018, 2019, and 2020 would be "adjusted" to the price level of (for instance) 2017 so that all prices are "prices of 2017". You need a standardised price (e.g. per sqm) because you need to control for possible unobserved effects in the composition of your data, e.g. when the average house sold in 2020 is "larger" than the average house in 2017. In essence you need make sure that the "deflated" prices are comparable. This can be a problem, for instance when there are changes in the market over time. You could imagine that the willingness to pay for "large" houses change over time, so that one sqm of a "large" house becomes more expensive over time. It can be hard to capture such effects by simply "deflating" (average) prices per sqm.
Option 1 can partly capture the effect(s) described above. Consider the case of a linear regression. Say you have two years (2019,2020) and you want to "control" inflation over time. Your (simplyfied) base model, with price $p$ and $sqm$ as independent variable would look like:
$$ p = \beta_0 + \beta_1 sqm + u. $$
Now you can add a "year dummy" (=1 if year==2020):
$$ p = \beta_0 + \beta_1 sqm + \beta_2 t_{2020} + u. $$
Coefficient $\beta_2$ captures the average effect on $p$ in 2020 compared to 2019. This is sometimes called a "fixed effect" since the variable simply is a "shift" in prices in 2020 compared to 2019 for all levels of $sqm$.
If you think $sqm$ and "time" are somehow related, you can also add interaction terms, e.g.:
$$ p = \beta_0 + \beta_1 sqm + \beta_2 t_{2020} + \beta_3 sqm * t_{2020} + u. $$
In this model you allow for a different intercept (in 2019 and 2020) and for a different slope of $sqm$ in both years. Instead of an interaction of "time" and $sqm$ you could also add and interaction with "size dummies" (e.g. "small" vs. "large" houses).
In essence, option 1 gives you more flexibility since linear regression allows you to "deflate" prices inside the model. Note that linear regression is a parametric approach, so you need to find a proper parameterisation of the model (just as you would need to find the right strategy to deflate prices when you do this outside the model).
When you use regression trees, you don't need to worry about the functional representation of the model. The advantage of linear regression is that the "time dummy" is forced to be fitted on all data. In regression trees the effect of dummies is less prevalent. So in this case "deflating" data outside the model could be worth a try.
However, when you are up to predictions with low variance, you ultimately need to check what approach works best based on test results.
Edit (20-12-30): Dummies
Suppose you have a vector of IDs:
  id
1  1
2  1
3  2
4  2
5  3
6  3

Dummy encoding will look like:
  id1 id2 id3
1   1   0   0
2   1   0   0
3   0   1   0
4   0   1   0
5   0   0   1
6   0   0   1

In a linear regression, dummies generally work as "contrasts", e.g. the effect of id2 vs. id1 and id3 vs. id1, so that you include n-1 of the dummies.


What optimization algorithms are used in scikit-learn?

There are multiple approaches to optimization in scikit-learn. I will focus on generalized linear models, where a vector of coefficients needs to be estimated:

LinearRegression and Ridge use closed-form solution $\beta=(X^TX+I\lambda)^{-1}X^TY$, but Ridge can also use stochastic gradient descent or method of conjugate gradients
Lasso and ElasticNet use coordinate descent
OrthogonalMatchingPursuit uses a greedy algorithm with the same name, that has $L_0$ penalty on coefficients
ARDRegression and BayesianRidge use something like EM algorithm
SGDRegressor and PassiveAggressiveRegressor use guess what! Stochastic gradient descent.
HuberRegressor uses BFGS (a second-order optimization method)


It's not going to be pretty, but you just for-loop your code:
ax, fig = plt.subplots()
for i in range(30):
    ax.plot(X_test[i].T, label='y_true_' + i)
    ax.plot(x_test_pred[i].T, label='yhat_conv' + i)
plt.show()

Because you have so many lines, it would be best to remove the legend from the plot as it will be too much information to interpret. You could use additional subplots e.g. ax,fig = plt.subplots(5,6) and plot one row on each subplot e.g. ax[1].plot(first_plot) etc.
It might be more suitable to generate some summary statistics or differences of the lines (as suggested in the comments). One way would also to plot the error bars/distributions of the lines as shaded regions and just the line of the average for particular groups, similar to the plot below:


Sequential Data is any kind of data where the order matters as you said. So we can assume that time series is a kind of sequential data, because the order matters.
A time series is a sequence taken at successive equally spaced points in time and it is not the only case of sequential data. In the latter the order is defined by the dimension of time. There are other cases of sequential data as data from text documents, where you can take into account the order of the terms or biological data (DNA sequence etc.).
The fact that you have sequential data is important for two reasons. First, you can into account for the representation of the data and also you can take it into account for the data modeling (e.g. Conditional Random Fields, Hidden Markov Models for text or genes and ARIMA Models for time series problems).


Coloring labels using scatterplot3d in R

For the colors  argument you need to assign a vector with the same length as your data points.

You don't explain how colors must be assigned to values of your output columms . In any regard, this would assign 4 different colors randomly:

cols = sample(1:4, nrow(output), replace=TRUE)

plot it with

scatterplot3d(output$X2,output$X6 , output$X7 , color=cols, pch=19, cex=1)

You could create a new column in your dataframe, e.g. rgbcolor and assign that:

scatterplot3d(output$X2,output$X6 , output$X7 , color=output$rgbcolor, pch=19, cex=1)

pch is the plotting character id, usually 1, 19 or 20 (circle or larger dot or smaller dot. (see for yourself: plot(1:25, 1:25, pch=1:25))

Tune the cex param to make it larger or smaller (1 means size = 100%)


Is it ok to interprete PCA plot this way?

Given the information provided, I do not think it's enough to say that your samples are between stages D12 and D19. I admit knowing nothing about your domain, but with what you're showing it is very conceivable that other factors are causing the spread of your samples from the others on PC1 and it's too much of a stretch to say that your samples "project" into the others in between D12 and D19. Why does it have to be a projection orthogonal to the axis described by PC1? Remeber that PC1 is a linear combination of all your inputs. That is, it describes an arbitrary line through your input space that is probably not aligned with any one thing in particular. For example, all three of the following projections could be envisioned for your problem:

Another red flag for me is that you say PC1 is related to differences in experimental protocols. You haven't posted a scree plot or listed the eigenvalues for the different PCs but obviously PC1 soaks up the most variance so it's a pretty big deal. If you're saying that PC1 is correlated with experimental protocol, then that would seem good because hopefully the impact of experimental protocol is minimized in other PCs. However, that's never completely the case, really.

Again, not knowing your domain it's hard to tell what the impact of experimental protocol is on the distribution of your data in the PC plot, but I'd be VERY wary of comparing these data in this way if your data was produced using a different experimental design and, as you noted, "there may be other differences."

My hunch is that you would need to continue minimizing differences in the protocols between your experiments and the published experiments before you could really use this type of analysis. Others might disagree but if you posted the sources of your data, the eigenvectors, and possibly a subset of the data, experts in your field may be able to chime in on this methodology.

The short answer is: you could be right in your interpretation but at this point you don't have enough evidence to suggest your samples are "in between" D12 and D19 in terms of the stage. And I worry about differences in methodology completely swamping your ability to interpret things this way.


The documentation of Apache Spark's MLib library has a neat and clear reference to the implementation of linear SVM's in Python.

It supports two models: SVMWithSGD and SVMModel


The rationale for stacking learners is to combine the strengths of the individual learners.

On the one hand your idea makes sense: if learners have different strengths, adding some features might help the meta-model detect when to give more importance to a particular learner. It's totally possible that overall the meta-model will work better this way in some cases.
On the other hand, generalizing this idea would often defeat the purpose of stacking:

if we know in advance that a particular model is particularly good in some specific identified cases, then it's likely optimal to switch entirely to this model in such cases (no stacking).
if we don't have any particular knowledge, then this would require to add many/all of the original features to the meta-model, just in case they help. But by adding features it's likely that (1) the model will try to use the features directly, hence collapsing into an individual learner itself, and (2) the model will be more complex, therefore more prone to overfitting and likely not to make the best use of the individual learners' output.

To sum up: the stacking approach relies on "making things simple" for the meta-model, so that it only has to make a call about the answers of the individual learners. This way the meta-model can "focus" solely on optimally use these answers. The more features we add to it, the more risk that it will not to its job correctly.


Use text similarity (cosine) instead of machine learning to classify companies into industries

This is an interesting approach.  For me though, it raises a few questions that might impact the effectiveness of the approach:

Were all the descriptions created by the same organization, or performed for the same purpose? e.g. Descriptions are taken from SEC filings.
Have you tried any unsupervised approaches to grouping these companies based on their descriptions?
How did you come up with 15 industries, not 14, not 16?
Have you applied stop words and removed very infrequent words?

Not being critical, but how confident are you that someone else would apply the same label to each of the 2000 instances you labeled?  Could that be one reason why the models are performing poorly?

Ultimately, to answer your question, there is no reason why domain knowledge cannot be used to inform your key words, but it seems like you will struggle when you get to industries that are "similar but different."  The good results that you have received using this method may just be indicative that you created both the labels, and the key words for the industries, which may well be what you want to achieve.

From a defense perspective, your classifier would only represent your opinion.  In the long run, you may be better off giving up a bit of performance to gain objectivity.  Company classification is difficult because of vertical and horizontal integration, and the practice of using holding companies to manage diversified businesses.  If you can take your opinion out of the process, it may be "better".  If someone disagrees with your model, they are just disagreeing with you.  If they disagree with a more traditional ML model, they are disagreeing with the dispassionate similarity of the descriptions.

HTH


How should multiclass classifier performance be measured when one type of error is preferred over another?

Define a scoring table like this (You will need to tweak this table to satisfy your particular use case. I am only using it as an example).

Pred   | True Label
Label  | A    B   C
-------+--------------
A      |  0  -1  -2
B      | -3   0  -1
C      | -4  -3   0

Notice this scoring table has the property of favours over-estimate than under-estimates.

Multiply (element-wise) the scoring table with the number of predictions fall into each of the 9 possible scenarios then sum the scores together will give you a metric with the desired properties.


LSTM-Model - Validation Accuracy is not changing

From the documentation:
inputs: A 3D tensor with shape [batch, timesteps, feature].
Are you saying that you want 1 input and 1 feature, but you want to output 100 neurons? I would consider adding more timesteps.
train_X = train_X.reshape(train_X.shape[0],10,1)
test_X = test_X.reshape(test_X.shape[0],10,1)

Also, I wouldn't add regularization to a ReLU activation without batch normalization. Actually, I probably would use dropout instead of regularization. Consider that with regularization many ReLU neurons may die.


Weighting influence of two neural networks on classification

As neural networks are non-parametric, it would usually be the case that you train the combined model (as you describe it, but perhaps with additional FC-layers at the end) and let the model as a single entity learn/decide how best to combine the output of your two sub-models.

If you want to the hard-coded percentages you mention, then the most straight-forward way would be to do what you mention in the last paragraph.

People often train the same (or slightly differing models) independently on the same dataset, then average the predictions of these models on the test set, which can improve overall performance by integer percentages.


How to annotate labels in a 3D matplotlib scatter plot?

In the following posts [1], [2] the plotting of 3D arrows in matplotlib is discussed.

Similarly Annotation3D class (inherited from Annotation) can be created:

from mpl_toolkits.mplot3d.proj3d import proj_transform
from matplotlib.text import Annotation

class Annotation3D(Annotation):
    '''Annotate the point xyz with text s'''

    def __init__(self, s, xyz, *args, **kwargs):
        Annotation.__init__(self,s, xy=(0,0), *args, **kwargs)
        self._verts3d = xyz

    def draw(self, renderer):
        xs3d, ys3d, zs3d = self._verts3d
        xs, ys, zs = proj_transform(xs3d, ys3d, zs3d, renderer.M)
        self.xy=(xs,ys)
        Annotation.draw(self, renderer)

Further, we can define the annotate3D() function:

def annotate3D(ax, s, *args, **kwargs):
    '''add anotation text s to to Axes3d ax'''

    tag = Annotation3D(s, *args, **kwargs)
    ax.add_artist(tag)

Using this function annotation tags can be added to Axes3d as in example bellow:

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
from mpl_toolkits.mplot3d.art3d import Line3DCollection

# data: coordinates of nodes and links
xn = [1.1, 1.9, 0.1, 0.3, 1.6, 0.8, 2.3, 1.2, 1.7, 1.0, -0.7, 0.1, 0.1, -0.9, 0.1, -0.1, 2.1, 2.7, 2.6, 2.0]
yn = [-1.2, -2.0, -1.2, -0.7, -0.4, -2.2, -1.0, -1.3, -1.5, -2.1, -0.7, -0.3, 0.7, -0.0, -0.3, 0.7, 0.7, 0.3, 0.8, 1.2]
zn = [-1.6, -1.5, -1.3, -2.0, -2.4, -2.1, -1.8, -2.8, -0.5, -0.8, -0.4, -1.1, -1.8, -1.5, 0.1, -0.6, 0.2, -0.1, -0.8, -0.4]
group = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 3, 2, 2, 2, 3, 3, 3, 3]
edges = [(1, 0), (2, 0), (3, 0), (3, 2), (4, 0), (5, 0), (6, 0), (7, 0), (8, 0), (9, 0), (11, 10), (11, 3), (11, 2), (11, 0), (12, 11), (13, 11), (14, 11), (15, 11), (17, 16), (18, 16), (18, 17), (19, 16), (19, 17), (19, 18)]
xyzn = zip(xn, yn, zn)
segments = [(xyzn[s], xyzn[t]) for s, t in edges]

# create figure
fig = plt.figure(dpi=60)
ax = fig.gca(projection='3d')
ax.set_axis_off()

# plot vertices
ax.scatter(xn,yn,zn, marker='o', c = group, s = 64)
# plot edges
edge_col = Line3DCollection(segments, lw=0.2)
ax.add_collection3d(edge_col)
# add vertices annotation.
for j, xyz_ in enumerate(xyzn):
    annotate3D(ax, s=str(j), xyz=xyz_, fontsize=10, xytext=(-3,3),
               textcoords='offset points', ha='right',va='bottom')
plt.show()


No, extremely-random trees does still optimize splits.  It does only pick one random splitting point for each feature (out of those randomly chosen max_features) but then which feature is actually used for the split depends on the criterion chosen.
https://scikit-learn.org/stable/modules/ensemble.html#extremely-randomized-trees


Yes it does, you resample from the whole population without replacement $N_A$ samples, which are associated to group $A$, and the rest of the samples are associated to group $B$. As I understand, you compute the statistic for every possible permutation of the total population and you measure how the actual statistic compares in the distribution of all the permutations.
If your statistic is not extreme in the distribution among all possible permutations, then it's likely that $H_0$ is true. Otherwise, if your statistic is an outlier of that distribution it's likely that $H_0$ is false.


What you are obtaining are different metrics on the predictions you made with a given model (so far I haven't given you any new information).

What you printed are several results from different metrics which measure different things in a model.

As you should know, sometimes the data scientist (you in this case) need to know how the model performs well in positive cases, or how does it perform in negative cases, etc.

The first thing you printed is called the confusion matrix, it crosses the information on Positives/Negatives in the real-world vs your model.

In the first square: When you said positive and it's positive in your data. The last square is negative and you said negative and (depending on how your output is made) the 2nd and 3rd squares are false positives and false negatives.

The precision measures how well your model detects Positives when your model says they are positive, meaning: How your model says the truth about positives $Precision = \frac{True Positives}{True Positives+False Positives}$

The recall measures how well your model detects Negatives when your model says they are Negatives, meaning: How your model says the truth about negatives $Recall = \frac{True Negatives}{True Negatives+False Negatives}$

The f1-score gives you the harmonic mean of precision and recall. The scores corresponding to every class will tell you the accuracy of the classifier in classifying the data points in that particular class compared to all other classes.

The support is the number of samples of the true response that lie in that class.

You can find documentation on both measures in the sklearn documentation.


Ask your grid administrator to add your local machine as a "submit host", and install SGE (which we assume you are using, you don't actually say) so then you can qsub from your machine.

OR....

Use emacs, then you can edit on your HPC via emacs's "tramp" ssh-connection facilities, and keep a shell open in another emacs window. You don't say what editor/operating system you like to use. You can even configure emacs to save a file in two places, so you could save to your local machine for running tests and to the HPC file system simultaneously for big jobs.


geo spatial clustering based on another feature

There are many such approaches, for example spatial autocorrelation, Lisa, etc.

In the clustering domain, GDBSCAN is a generalization of DBSCAN where you could easily define neighbors as points being within a certain distance and having a similar price.

Nevertheless it is probably a good idea to look into actual geostatistics and beyond the limitations of sklearn.


Is there an oriented clustering algorithm?

If I remember correctly, non-negative matrix factorization (NMF) can be used as a clustering approach that can recover clusters that are along vectors, for example. It may work for your dataset. It factors a data matrix $D \in \mathbb{R}^{m * n}$ into two matrices $W \in \mathbb{R}^{m*k}$ and $H \in \mathbb{R}^{k * n}$. Effectively, $W$ contains the weights that are applied to each vector in $H$ to reconstruct the original data; one way of using this method is to interpret the $n$-dimensional vectors in $H$ as clusters (these vectors would be the 'directions' that your data is along) and the $k$-dimensional vectors in $W$ as the data-example-wise affinities for the different clusters. One method to cluster with this process is to simply place each of the $m$ data examples into the cluster with the index of the highest value in the $W$ vector.

There are implementations in several standard libraries, including sklearn, so it should be relatively easy to try it out.  Good luck, and welcome to the site!


How do scientists come up with the correct Hidden Markov Model parameters and topology to use?

I'm familiar with three main approaches:

A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes, and so have 44 states for the hidden phoneme layer in a voice recognition model.
Estimation. The number of states can often be estimated beforehand, perhaps by simple clustering on the observed features of the HMM. If the HMM transition matrix is triangular (which is often the case in failure prediction), the number of states determines the shape of the distribution of total time from the start state to the end state.
Optimization. Like you suggest, either many models are created and fit and the best model selected. One could also adapt the methodology that learns the HMM to allow the model to add or discard states as needed.


What is the name of this visualization with a circle and internal arcs?

This is a chord diagram. It is usuallly generated from a square matrix like the one you have. This can be done with specific libraries at least in R (circlize package) and Python (plotly). It's also possible to do it with d3.js but apparently not so easily.
General advice: I find the "X Graph Gallery" websites quite convenient for exploring different types of graphs: R graph gallery, Python graph gallery and D3 graph gallery.


Having a reward structure which gives high positive rewards compared to the negative rewards

Is my understanding correct and is it recommended to have such a reward structure for my use case ?

Your understanding is not correct, and setting extremely high rewards for the goal state in this case can backfire.

Probably the most important way it could backfire in your case, is that your scaling of bad results becomes irrelevant. The difference between 0 and -50 is not significant compared to the +1000 result. In turn that means the agent will not really care by how much it fails when it does, except as a matter of fine tuning once it is already close to an optimal solution.

If the environment is stochastic, then the agent will prioritise a small chance of being at the target temperatures, over a large chance of ending up at an extreme bad temperature.

If you are using a discount factor, $\gamma$, then the agent will prioritise being at the target temperatures immediately, maybe overshooting and ending up with an unwanted temperature within a few timesteps.

Working in your favour, your environment is one where the goal is some managed stability, like the "cartpole" environment, with a negative feedback loop (the correction to the measured quantities is always to force in the opposite direction). Agents for these are often quite robust to changes in hyperparameters, so you may still find your agent learns successfully.

However, I would advise sticking with a simple and relatively small scale for the reward function. Experimenting with it, after you are certain that it expresses your goals for the agent, is unlikely to lead to better solutions. Instead you should focus your efforts on how the agent is performing, and what changes you can make to the learning algorithm.

What I would do (without knowing more about your environment):

Reward +1 per time step when temperature is in acceptable range
Reward between -0.1 * temperature difference per time step when temperature is outside acceptable range. It doesn't really matter if you measure that in Fahrenheit or Celsius.
No discounting (set discount factor $\gamma =1$ if you are using a formula that includes discounting)

The maximum total reward possible is then +36, and you probably don't expect a worse episode than around -100 or so. This will plot neatly on a graph and be easy to interpret (every unit below 36 is roughly equivalent to performance of an agent spending 15 mins per day just outside acceptable temperatures). More importantly, these lower numbers should not cause massive error values whilst the agent is learning, which will help when training a neural network to predict future reward.

As an aside (as you didn't ask), if you are using a value-based method, like DQN, then you will need to include the current timestep (or timesteps remaining) in the state features. That is because the total remaining reward - as represented by action value Q - depends on the remaining time that the agent has to act. It also doesn't matter to the agent what happens after the last time step, so it is OK for it to choose actions just before then that would make the system go outside acceptable temperatures at that point.


First of all, your confusion matrix looks normal to me. As shown below, the majority is positive class, and the model prefers to predict most of the negative cases as positive, which is common in imbalanced classes. Your model predicted 8887 negative cases as positive.
True   0 [ 3890 | 8887 ]
Labels 1 [  844 | 20044]
            0       1
         predicted labels

Your classes are imbalance, so the model does the fit/prediction in favor of majority class.
To overcome this issue,  you must perform over/under-sampling to eliminate possible issues caused by imbalance classes. You can use imbalanced-learn library in this regard.
So to answer your questions:
A1: You read the confusion matrix incorrectly. The model is skewing to the majority class.
A2: You must balance your data. Don't forget to perform over/under-sampling on your training set only. In addition, you may need to use precision or recall but not accuracy only, based on your need; for example if one class is important and having less false positive or less false negative is your goal.


Python - Predicting data based on multidimensional array with Keras

Change first layer with

regressor.add(LSTM(units=num_units, input_shape=(x_train.shape[1],x_train.shape[2])))

You need to have 2 numbers as input shape.

Hope will help you.


How to get periodicity from timeseries data?

Looking at your data - the easiest way is to create a Last-N Days hourly average of the binary indicator - and then use a threshold (based on experimentation) to binarize it.
e.g. if your Last 10 Day hourly average looks like this:
0, 0, 0.6, 0.8, 0.9, 1, 0.9, 0.7, 0, 1, 1, 1, 0
Then, a threshold of 0.8 to binarize would result in the following:
0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0
and - then you could use the time slices associated with the 1's for the recommendation.
This might be the simplest thing you could do - There could however be more sophisticated algorithms out there.


can I use z-score normalization even if it doesn't make sense for my data to be negative?

I think it doesn't really matter that there can be negative values after as long as you rescale your data correctly at the end.

Think about what positive/negative values mean for z-score. This has nothing to do with whether your use case (for example speed) can realistically have negative values or not. With z-score positive values simply mean that the value is above the group mean while negative values can be interpreted as the opposite.

As long as you are able to rescale after your model (to get your data back to the original interpretation) using z-score should be fine.


How do I read the cord_19_embeddings_2020-07-16.csv from the COVID-19 Open Research Dataset Challenge (CORD-19) on Kaggle?

So, after a lot of digging, I found something in the comment section.
They are document embeddings.

There is a github repo that specifies an API.

Paper on arxiv

Example usage of a similar approach

Relevant Comments from the Kaggle Comment section on the Data Update Log for the CORD19 Dataset:

Comment 1

Comment 2

Examples how to visualize the embeddings in a Jupyter Notebook:
import pandas as pd
from whatlies import Embedding, EmbeddingSet
#Docs: https://rasahq.github.io/whatlies/api/embeddingset/

#transponse dataframe
sample_df = pd.read_csv('data/cord_embeddings_sample.csv', header=None, delimiter=',', index_col=0).T

def to_ems(df):
    ems_dict = {}
    for columnName, columnData in df.iteritems():
        ems_dict.update({str(columnName): Embedding(columnName, columnData)})
    return EmbeddingSet(ems_dict)

ems = to_ems(sample_df.head(10))

ems.plot_correlation()

ems.plot_interactive("ug7v899j","02tnwd4m")

You can even do NLP with the jsons from the dataset and link them to the embeddings via the UUID and SHA from metadata.csv.
Example:
Find words that relate to smoking and color the respective papers:
I created 2 EmbeddingSets where I filtered the embeddings for papers that have smoking-related words in their text body and subtracted their UUIDs from the list.
Both EmbeddingSets can be displayed in the plot.
from whatlies.transformers import Umap

# add 2 embedding sets

emb1 = non_smoking_ems.add_property('set', lambda d: 'non smoking papers')
emb2 = smoking_ems.add_property('set', lambda d: 'smoking papers')

both = emb1.merge(emb2)

#add a clustering transformer that reduces dimensionality (like umap) and visualise them
both.transform(Umap(2)).plot_interactive('umap_0', 'umap_1',color='set', annot=False)


How do I get one overall prediction, where each data point has many pictures?

You are describing a problem of supervised learning with multiple inputs. That is not an uncommon task and you can find many tutorials about multiple inputs for neural networks out there. Using Tensorflow, I personally recommend Keras Functional API for this task, since it gives you more control on the layers while keeping the high-level simplicity.


Is there a process flow to follow for text analytics?

This is a great place to start! While not catalogued in a "process flow", Daniel Jurafsky's book, "Speech and Language Processing" talks through the various calculations and steps related to analyzing text that you will find useful.

The reason I say that a process flow is not provided is because Jurafsky - in great detail - explains the pros and cons of particular methods applied throughout a pipeline, and how this could change results. As an example, when calculating perplexity (an inverse metric that quantifies how well a language model can predict the next word in a statement), you should capture beginnings, ends, and stop words of statements - as opposed to other methods that require the removal of stop words.


You could partition your dataset between train and test sets. Then arrive at the pattern parameters from the train set. After that apply your learnt model on the test set to evaluate its performance.
If you figure out that your model is performing poorly then you know you have overfit.
Also since this is a trading strategy and possibly involves timeseries, you want to ensure there is no data leakage and the test set is of a time frame later than that of the test set.


Yes, you are right. The soft-max layer outputs a probability distribution, i.e. the values of the output sum to 1. The sigmoid function outputs marginal probabilities and therefore can be used for multiple-class classification, when the classes are not mutually exclusive. Additionally the soft-max layer is soft version of the max-output layer so it is differentiable and also resilient to outliers. A problem with sigmoids is that as you reach saturation (values get close to 1 or 0), the gradients vanish. This is detrimental to optimization speed and soft-max doesn't have this problem. Another interpretation is soft-max as a generalization of sigmoid, actually when there are two classes they are the same.

Wrapping up you should use soft-max when the classes are mutually exclusive and sigmoid when the classes are independent. This can be summarized in the following table:

Soft-Max                            |                 Sigmoid
-------------------------------------------------------------------------
Used for multi-classification       |  Used for binary classification in
in logistic regression model.       |  logistic regression model.
-------------------------------------------------------------------------
The probabilities sum will be 1     | The probabilities sum need not be 1
-------------------------------------------------------------------------
Used in the different layers of     | Used as activation function while
neural networks.                    | building neural networks
-------------------------------------------------------------------------
The high value will have the higher | The high value will have the high
probability than other values.      | probability but not the higher

For more information on soft-max look at the following links: 1, 2 and 3. For a step-by-step guide, including usage in Python see this reference 4.

For more on soft-max vs sigmoid check this: 5, 6 and this 7.

If you want a more reliable source on why to use soft-max regression for mutually exclusive classes you can look here. This page is part of Unsupervised Feature Learning and Deep Learning Tutorial of Stanford University, it contains material contributed by Andrew Ng and others.


How does graph classification work with graph neural networks

The equation you pointed out in the paper has posed the graph learning problem based only on the nodes of the graph. Therefore, to perform graph level tasks like graph classification, one would need a 'special node' which introduces a node that represents the entire graph. This is all just to make the equation hold for graph level tasks which are not dependent on just one node.

The paper which you are reading is quite old and the new GNN formulations are based on the kipf-GCN (https://arxiv.org/abs/1609.02907). It is much simpler to understand, implement and has become the standard GCN. If you want to keep with the current hot topic of GNN, I would recommend you having a look at Kipf's blog (https://tkipf.github.io/graph-convolutional-networks/) for a simple intuition for the modern-day GCNs.


I guess there is no package for cnn but you can write your own convolutional layer. mxnet or h2o will be useful for it.

check this out:

http://dmlc.ml/rstats/2015/11/03/training-deep-net-with-R.html


How best to embed large and noisy documents

Doc2vec and similar algorithms are useful methods to create document embeddings.
You should try to include as much data and metadata as possible.
The size of the documents does not matter much because they will be project into a fixed dimensional embedding space.
As far as noise, the effect will be task specific. If you are doing similarity analysis and clustering, you can use approximate methods which are robust to noise.


Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar.
So technically it can indeed be done with a TF-IDF representation of documents as follows:

Collect the global vocabulary across all the documents and calculate the IDF for every word.
Represent every document as a TF-IDF vector the usual way: for every word, obtain the term frequency in the document (TF) then multiply by the global IDF for this word (IDF). Note that every vector must represent the document over the global vocabulary.
Use any clustering method over the vector representations of the documents: K-means, hierarchical clustering, etc.

Note that this method is unlikely to be as good as state of the art methods for topic modelling.


I have not found any unified package in Python like NbClust.  Check out the Python package rpy2 for using R modules in Python.


Confused about false positive and false negative in confusion matrix?

Think about the order of your test and prediction sets when constructing the confusion matrix. Here is a piece from one of my codes.

cm = confusion_matrix(y_test, y_pred)
print(cm)
Output:
[[TN FP]
[ FN TP]]

However, if I used:

cm = confusion_matrix(y_pred, y_test)
print(cm)
Output:
[[TN FN]
[ FP TP]]

That occurs since the predictions of the model are presented in the rows instead of columns now. Also confusion matrices can be NxN, or the classes may not be labeled as either 0 or 1. You can also change the places of TN and TP, think what should happen if you had classes named as 9 and 10. In other words, decision of Negative/Positive signs are our decision; we say what they are (hopefully in a reasonable way).
Hope I could help, please do not hesitate to ask more.
Regards.


Predictive modeling on time series: How far should I look back?

Hold out the most recent block of data (maybe something like a month) to use as your validation data.

Try multiple models and different ways of training those models. For example, training only on the most recent couple of months vs the whole time period.

Take a small number of your top performers and test their performance on the validation set and see what does better.

There’s nothing special about this answer. Anytime you want to know is X better than Y, testing performance on a held out validation set is almost always the way to go.


Error Metric for Prediction of sales quantity (by using Attributes of item)

This is a fairly common task, especially in fields like economics, but the choice of using a Neural Network makes your approach different from most.

Having said that, I think you can still use common validation metrics:

Mean Error
Mean Absolute Error
Absolute Error ("delta")

You probably don't need anything more fancy than that, especially if those are the error metrics that your business clients will be using to measure your success (which I imagine would be the case).

Having said that, if you want a fancier ML metric to use whilst developing your algorithm you might try Root Mean Squared Error (RMSE). Here's an article on the topic.

If you wanted to, you could also construct a pseudo-R-squared metric, but those are less used with ANN's and I wouldn't choose it myself for this particular case.


What you are trying to do here is forecast the future values of a time series. This is a predictive problem and the future values will depend on a number of latent factors. I will assume all we have access to is historical data from the series as your question indicates.

If you want to predict a future value for the time series, you should not only use the current value as an input, but rather you should use a chunk of the historical data. Since you have 18,000,000 instances, this is a lot, you can make your network quite complex in order to capture some latent trends hidden inside your data which can help predict the future value. To predict a value at time $t$ we will use the $k$ previous values. This hyper-parameter needs to be effectively tuned.

Restructure the data

We will structure the data such that the features $X$ are the $k$ previous time measurements, and the output target $Y$ is the current time measurement. The one that is being estimated by the model.

k = 3
X, Y = [], []
for i in range(len(col1) - k):
    X.append(col2[i:i+k])
    Y.append(col2[i+k])

X = np.asarray(X)
Y = np.asarray(Y)

Split your data

from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)

Using the data in a Keras model

This is a simple Keras model which should work as a first iteration step. However, due to the small amount of data you provided us I cannot get any meaningful results after training.

import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv1D, MaxPooling1D, Reshape
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

x_train = x_train.reshape(len(x_train), k, )
x_test = x_test.reshape(len(x_test), k, )

input_shape  = (k,)

model = Sequential()
model.add(Dense(32, activation='tanh',
                 input_shape=input_shape))
model.add(Dense(32, activation='tanh'))
model.add(Dense(1, activation='linear'))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.summary()

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))


Orange3 summarizing data, grouping data values

Unfortunately, not that I know of.
You could try Feature Constructor and define custom ranges or simply create a column with mean, then plot it. But that is not so different from Python Script.


How does the inertia cython implementation in scikit-learn for kmeans work?

& is the "address-of" operator in c, and that appears to be how it's being used here.  See these two SO posts.
Note the signature of _euclidean_dense_dense:
cdef floating _euclidean_dense_dense(
        floating* a, # IN
        floating* b, # IN
        int n_features,
        bint squared) nogil:

The first two inputs are pointers. So you need to pass the address, not a copy of the data.
Notice too that only the address of the element in the first column gets passed. If you look at the definition for _euclidean_dense_dense that becomes clearer: that function actually loops over the addresses of the rest of the columns in its computation.


Hello, anyone able to direct me to a "cheat sheet" of Neural Network equations with legends?

Look further for the "zoo neural network posters"


@Ram , you have correctly said. You might not need any algorithm to detect these known abnormality.

1) But even if u apply K-means, It will give you same result and will do it in automatically way.

2) Seems like you have just 1 variable -total ectivities, things get dirtier when you have multiple variables then you might have to look into LOF/ABOD other algorithm after identifying what exactly is abnormality for you.  Have a look here-

https://machinelearningstories.blogspot.com/2018/07/anomaly-detection-anomaly-detection-by.html

3) In ML , many a times you even do not know what exactly is an abnormlaity. Here you assume high activities is abnormality. depending on problem statement we need to have some unsupervised algo to identify possible abnormalities.


To me this problem looks similar to language modeling: a model is trained on a large amount of sequences, and then it can predict the probability of any input sequence. In your case a low probability would indicate an abnormal sequence.

My background is in NLP that's why I think language modeling, but I guess the same techniques are used for other problems as well. The fact that you have transitions and states suggests Markov Models, for which there are known methods for inference and estimation. So maybe you could design a more specific kind of model for your case and use something like the Baum–Welch algorithm.


In parametric models such as linear regression, logistic regression and multi-layers perceptrons, weights are updated with regards to the "difference" between the output of your model and the real label.

More precisely, weights are updated using the gradient descent / backpropagation procedure. It is composed of two parts : the forward pass and the backward pass.

For a given observation (or a set of observations), the forward pass is about feeding the model with observations and output a result a. This output "a" is then compared with the real value, the label y. Using some cost function metrics (such as Absolute Error or Square Error for regression purposes,  cross-entropy for classification purposes...), we can then compute j(y,a) which is the error between the output and the real value.

We can now run the backward pass which is about computing the derivative of the cost function with regards to any weight / bias coefficient in the logistic regression / neural network. We can the update coefficients such as :

Where alpha is the learning rate. So to answer your question, weights are not updated until they reach the expected value. We are just trying to reach the minimum in cost function surface by running gradient descent procedure.


I have worked on building a fraud detection solution using text mining, so I understand the scenario that has led to this question. I'll talk about the approach/techniques that is to be followed to build the fraud detection solution.I'll divide it into 4 sections

Build line of business fraud dictionary
Fraud detection & scoring
Tools
Additional notes

Building Fraud Dictionary:

As a first step you'll have to identify fraud concepts with help from subject matter expert for the line of business in concern. Fraud concept is a person, characteristic, entity, or event that represents a suspicious scenario and similar concepts can be grouped together to form a concept category. Each concept is further represented by words, phrases, entities etc. The presence of these words/phrases in the document implies occurrence of fraud concept in that transaction. The end result of this exercise will be a fraud dictionary that is a repository of concepts and suspicious key words.

For example considering the data set mentioned in the question, the numeric patterns become the concept category, and fictitious invoice numbers and fictitiously-generated transaction amounts are 2 different concepts. There will be a keywords/phrases that is associated to this concept which should be captured onto the fraud dictionary.

You can use NLP technique to build the dictionary.

Steps:

1) Convert the sentence to lowercase

2) Remove stopwords, these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words

3) Extract n-gram i.e., a contiguous sequence of n items from a given sequence of text. simply increasing n, model can be used to store more context

4) Assign a syntactic label (noun, verb etc.)

5) Knowledge extraction from text through semantic/syntactic analysis approach i.e., try to retain words that hold higher weight in a sentence like Noun/Verb

Fraud detection & scoring:

Phrases/Keywords to be semantically matched with Phrases/Keywords from Fraud Dictionary and Fraud concepts in transactions can be identified. At this point, care needs to be taken to maintain the context (negation, positive sense) in which a keyword/phrase has been used to avoid false positives.

For each identified occurrence of Fraud concept in the transaction, a weight of 1 can be assigned. This is done for calculating a suspicious score for the transaction. Higher the score high suspecious.

Note that using a combination of larger data set with high textual content, and extensive fraud dictionary would result in higher number potentially identified.

Tools:

You can consider using OpenNLP / StanfordNLP for Part of Speech tagging. Most of the programming language have supporting library for OpenNLP/StanfordNLP. You can choose the language based on your comfort.

Additional notes:

You can refer here to get an idea of extracting concepts from sentences
Fruther read my blog on Text mining 101 to learn more about

TM process overview
Calculate term weight (TF-IDF)
Similarity distance measure (Cosine)
Overview of key text mining techniques

Hope this helps.


Hashing trick with random forest in scala

Let's look at the error message:

found : Array[org.apache.spark.mllib.linalg.Vector] required: org.apache.spark.mllib.linalg.Vector

"found" is the type of object that it came across, "required" is the type of the object that the function accepts. The types look mostly the same ( org.apache.spark.mllib.linalg.Vector ) but the first is Array[X] and the second is X.

Why? Because trainingData.collect() gives you the RDD as an array of objects in the array.

What you probably want to do instead is map, to turn each vector into a LabeledPoint, instead of trying to turn the whole RDD into a single LabeledPoint.