Is there any way to analyze the format of text strings?

A machine learning approach to this would be create character-level encoding for the data. Then, you can run K Means (use silhouette score for judging the fit) or Louvain community detection (doesn't need the number of clusters as input). You can finally look at cluster heads for the formats.


What is the best way to treat datetime in the preprocessing step of machine learning

You already have a good beginning. Transforming data into 4 columns. Year, month, day and hour. Now These 4 are all categoricals, you can than just apply one hot Encoding. Than no Domination will happen.


Each tree starts as a single leaf and all of the residuals go to that leaf. Then we calculate the similarity score and try to split amongst some feature characteristic. For example say we only had one feature - height. We could split the residual if its respective x-value was say height > 180 or height < 180. And then proceed to calculate the similarity scores and the gain for the new leaves and continue down our tree

Here it seems like you are describing just a decision tree. Watch this tutorial to understand it a bit more

My question is now say we had two or more features such as height , age , weight , education etc. How does the algorithm know where to break the initial residuals? Does it compute it over all possible features and finds the best gain? I can imagine for large datasets to go through all features , then break it at some threshold and compare must take a long time.

Again you are talking about the decision tree and its  complexity, but yes, for large datasets is overly computationally expensive. To avoid this several strategies are used, as sampling, feature selection or quantization

Or does it start at some random feature and work its way down kind of like a random forest? I know there is a parameter within the algorithm that allows you to set the percentage of features per tree so adding on to my question once we specify this percentage how does it pick the features and more importantly once those features are picked how does it know which one to pick to break the initial leaf of residuals.

The random forest does random feature selection per split of the tree. The trees are grown greedy, so it tries all possible splits and chooses the best. If the original paper was too difficult try following this tutorial about random forest


Plotly is by far the best interactive visualization library/platform I have used, and it works very well with IPython/Jupyter too.

There are tutorials on Plotly's docs which help you integrate it with Jupyter. Another tutorial.


Best way to format data for supervised machine learning ranking predictions

This problem looks like a lot with the problem of ranking college football teams. I have never worked on this ranking problem, but I believe you can borrow some tools used there to build your model.

Here goes a couple of references:

Colley Matrix Rankings - This was one of computer rankings used by the BCS. It is also the only one that shared their methodology.

An example of the Colley Matrix Rankings - An easy example to follow.

The Perron-Frobenius Theorem and the Ranking of Football Teams - A well known reference that presents some ranking methods. This paper also shows how to assess the probability of winning a game based on the rankings of the teams.


LSTM: Taking previous output values as feature

LSTMs handle this for you.

You can search for some tutorial on the LSTMs that explains how they works. One that I like is this.


Does anyone could help me to understand what is the autoencoders?

Autoencoders are a neural network solution to the problem of dimensionality reduction.

The point of dimensionality reduction is to find a lower-dimensional representation of your data. For example, if your data includes people's height, weight, trouser leg measurement and shoe size, we'd expect there to be some underlying size dimension which would capture much of the variance of these variables. If you're familiar with Principal Component Analysis (PCA), this is another example of a dimensionality reduction technique.

Autoencoders attempt to capture a lower-dimensional representation of their data by having a hidden "bottleneck" layer which is much smaller than the dimensionality of the data. The idea is to train a neural network which throws away as much of its dimensionality as possible and can still reconstruct the original data.

Once you have an autoencoder which performs well, by observing the activations at the bottleneck layer, it's possible to see how an individual example scores in each of the reduced dimensions. This may allow us to begin to make sense of what each of the dimensions represents. One can then use these activations to score new examples on this set of reduced dimensions.


nlp - opinion mining vs sentiment analysis

Can someone point me in the direction of more sentiment analysis
  oriented resources?

There are tonnes of exceptionally good papers on sentiment analysis for your reference. Especially the ones on social network sites like Twitter.

However, I think the book by Manning and his course are a nice start as references to NLP and opinion mining.

  Can nlp, or something else, be used to derive unstated affect?

Yes, it can be. This would be my approach:

(Consider I already have a hand-classified training data) Using the bag of words approach, the most used words and phrases for a mood (stressful, jovial, etc) can be captured, and the subsequent training data can be ranked accordingly as (60% stressful, 40% jovial).

Additionally, you can also define thresholds for getting a single mood. Like:

if mood[score] > .5:
    current_mood = mood


In that link you posted, you can look at the python full solution here at the end and go through it to see what all is distributed.
In short, some parts are distributed, like reading data from the file, but the very important parts like the distance computation are not.

Running down, we see:

  sc = SparkContext("local[6]", "PythonKMeans")

This instantiates the context and creates a local cluster which the jobs will be submitted to

  lines = sc.textFile(..)

This is still setting up. No operations have taken place yet. You can verify this by putting timing statements in the code

  data = lines.map(lambda x: (x.split("#")[0], parseVector(x.split("#")[1])))

The lambda here will be applied to lines, so this operation will split the file in parallel. Note that the actual line also has a cache() at the end (see cache]). data is just a reference to the spark object in memory. (I may be wrong here, but I think the operation still doesn't happen yet)

  count = data.count()

This forces the parallel computation to start, and the count to be stored. At the end, the reference data is still valid, and we'll use it for further computations. I'll stop with detailed explanations here, but wherever data is being used is a possible parallel computation. The python code itself is single threaded, and interfaces with the Spark cluster.

An interesting line is:

  tempDist = sum(np.sum((centroids[x] - y) ** 2) for (x, y) in newCentroids.iteritems())

centroids is an object in python memory, as is newCentroids. So, at this point, all computations are being done in memory (and on the client, typically clients are slim, i.e. have limited capabilities, or the client is an SSH shell, so the computers resources are shared. You should ideally never do any computation here), so no parallelization is being used. You could optimize this method further by doing this computation in parallel. Ideally you want the python program to never directly handle individual points' $x$ and $y$ values.


Matching Data Text of Two Place with Exception

Rule of thumb: if a human with a lot of time wouldn't be able to do it, there's little chance a machine could. In this case if you ask somebody to classify whether a place is a park or not, they just can't do it correctly if the place name doesn't contain any indication that it is a park.
In a case like this you would need to use external resources in order to add indications about the place. For example you could search the place name in Wikipedia, and if Wikipedia says that it's a park then you add a boolean feature "true" for the place.
More generally I suspect that your approach is not very solid: you looked at a few results and noticed that your matching system makes mistakes about parks. If you look at more results there's a good chance that you will find more mistakes about various kinds of cases. Fuzzy matching works only when the strings are quite similar, so it's unavoidable that the system will make mistakes one way or the other. Even if you try to fix every kind of mistake one by one with some ad hoc exception, the system will never be completely correct. Importantly, this would make the system completely specific to the dataset, so it wouldn't generalize well to new instances.
This is an issue related to task specification and design: the automatic system can only achieve a certain level of quality, so maybe the task should be framed differently. For example, the system could be in charge of finding candidate pairs for matching, and a human expert would annotate the actual matches among the candidates. Or you could just decide to live with the errors, if they are not critical for the application.


You can check this whitepapers for time series prediction with KNIME.
https://www.knime.org/files/knime_bigdata_energy_timeseries_whitepaper.pdf
https://www.knime.org/files/Anomaly_Detection_Time_Series_final.pdf

I am not sure though if you are trying to predict how many events will happen say in the next hour or just count how many events have happened per hour in the past. In this last case, just use a GroupBy node.

Let me know if this helps

-- Rosaria


Why does my manual derivative of Layer Normalization imply no gradient flow?

If you're computing the deriviative of the layer norm for the purpose of using it in backprop, then you need to compute the derivative with respect to the parameters of the layer, not it's inputs.
i.e., fix $x$ and compute $\frac{\partial y }{\partial \mu }$ and $\frac{\partial y }{\partial \sigma }$
Regarding your derivation specifically
I did not look at it closely, partly because I suspect that its not actually the derivative you care about as I said above.  But it looks like you're over-complicating things in the sense that you should treat $\mu$ and $\sigma$ as fixed variables and not as functions of $x$.
If they're functions of $x$, it may be that you're running into something like this:
You define $y = \frac {x - f(x)}{g(x)}$ and you compute $\frac{\partial y }{\partial x }$
... quotient rule ... computation... re-arrange...whatever
then you plug $f(x) = x$ in to the result and correctly observe that $\frac{\partial y }{\partial x } = 0$. Which is kind of obvious if you plug in at the start, but has been obscured because of notation. Now, where you started is not as simple as what I've written, but this same thing could happen.


Yes, you are right in your understanding of the role of the decoder.
However, your use of "query" here, while somewhat technically correct, seems a bit strange. You are referring as "query" to the partially decoded sentence. While the partially decoded sentence is actually used as query in the first multihead attention block, people normally do not refer to it as "query" when describing stuff from the conceptual level of the decoder.
About needing the decoder in classification or regression tasks: the decoder is used when the output of the model is a sequence. If the output of the model is a single value, e.g. for a classification task or a single-value regression task, the encoder would suffice. If you want to predict multiple values of a time series, you should probably use a decoder that lets you condition not only on the input but also on the partially generated output values.


Is it reliable to use TensorFlow (ML in general) to classify baggage bag tags based on the presence of a green stripe?

The scientific answer would be, it depends.

In case you are using any kind of Deep net, then 40 images is far too little.
It might be helpful to describe your problem setting a little bit more in depth.
Are the bags always in the same place, or do they need to be localized first? These kind of details could help other users in their recommendations.

As a first approach, before you try a deep net or any kind of ML I would try a simple baseline first. Do you know what the exact pixel value of your green stripe is? You could then simply check whether this colour is present at all. This is rather coarse, but I would see how far this gets you and it is good to see whether your ML methods can beat this simple baseline.
Subsequently you could also think of trying to localize the bagtags (in whatever way you like) then cropping it and checking for the presence of this green stripe.


Perform PCA on columns of different length

According to my understanding PCA requires that you have the column of equal length, so you either need to

shorten the longer columns (basically just skip the incomplete observations) or

fill in the gaps in the shorter columns.

If you choose the second option, you'll need to learn about the concept of imputation (see the following link for reference Missing data imputation).
Please note, that you need to get rid of the dummy values representing missing response (666s) in your data before performig PCA, because these values are arbitrary (they also could be 9999 or -1) and would have significant impact on the results.
Furthermore, it's arguable whether PCA can be applied to binary data, see e.g. discussion under related SE question.


Question of pretraining text-generation task, it seems that pretraining is not work for a small model?

1, I pad some zeros in the input tokens for multi sentences. The output positions of output tokens should be exactly same to the input tokens, which means I should keep the padding zeros in the output tokens.
2, The pretraining time should be longer.


Given n ordered sets, each containing 6 numbers, generate the next set in the sequence

This should just be a sequence prediction problem, and a multivariate one. You can use LSTMs or GRUs in Keras to handle the sequences. In the simplest architecture, the inputs will be batches of (it seems) n-1 steps of 6 numbers, and the output will be the final 6 numbers in the sequence. This is pretty much how any RNN works. The loss will be mean_squared_error, I presume.

But given how much is unknown here about the distribution of the sequences, it's unclear whether this simple approach would work well, or whether even a more complex network can learn whatever patterns these follow.


Find shared properties of a cluster samples

Obviously you can check the variance of each attribute.

But unless the data is badly scaled, there will likely need the combination of attributes to explain the differences of clusters.


Tensorflow Calculate error for a single neuron

Easy, you have already done it.

mse = tf.reduce_mean(tf.squared_difference(out, Y))

This command means you are taking the mean of all the neurons over all the examples.

error = tf.squared_difference(out, Y)

This command will give you the error over all neurons over all examples. But make sure you calculate against the correct axes.


Combine K-nearest neighbor with categorical embedding

If I understand well, you label encode categorical variables and fed them to a neural network. If this is the case, you can try the following:

add the numerical variables
create and train an autoencoder
use the encoder part to map input to a vector space and perform k-nearest neighbor to it.

You can read the second method described in https://towardsdatascience.com/detecting-credit-card-fraud-with-autoencoders-in-python-98391cace8a3 It uses a dataset with numerical variables only, but since you label encode categorical variables it applies in your case too.


Named Entity Recognition is traditionally evaluated using precision/recall and F1 score [https://towardsdatascience.com/entity-level-evaluation-for-ner-task-c21fb3a8edf] - the medium article gives a low down on how to achieve this
I recently happened to read this article on a new approach for the same. Please see the details in the attached medium link : [https://towardsdatascience.com/a-pathbreaking-evaluation-technique-for-named-entity-recognition-ner-93da4406930c]  but havent tried this out yet though


Is the graphic of deep residual networks wrong?

(Per the diagram), $F(x)$ here is simply the entire two-layer non-linear chain that is operating on the input $x$. Then, the final output is simply $F(x) + x = H(x)$. That's it!

The thing that may be confusing you is that $F(.)$. In this case, they do not mean for $F$ to simply encompass one operation. Instead, it encompasses any set of operations processing $x$, up until you add $x$ back. Hope that helps!

PS: It is also common to see this type of nomenclature in a lot of DNN literature, whereby one refers to an entire deep non-linear chain as $D(x)$. For example in Generative Adversarial Networks, (GAN)s, $D(x)$ refers to the entire deep net devoted to the discrimination process, while $G(x)$ refers to the entire net devoted to the noise shaping. In both cases, they are composed of entire functions/nets, and do not signify simply one operation.


approach for multi label text classification

So, you are asking about how to develop this system / model, which can classify text.
Yes, it is a great idea to instantiate a "baseline" or dummy model, which can be rule-based or randomly assigns a label to a certain piece of text.
From this dummy model, yes you can then use RNN/LSTMs that does multiple-inputs (e.g. words in text) to single output probability over classes as a more sophisticated model and yes you would then compare the validation and test accuracy, F1-score, etc. to see if that improvement to the model is warranted by the change in the model's functionality to classify the texts.


How to use historic data (granularity at day level) for ML modeling?

Welcome to DSSE, Abdul.
So, you trying to predict pipe corrosion based on water flow. I assume that the corrosion measurement is taken in different timesteps than the water flow measurements.
If you have a fixed timestep for your corrosion measure both sum and average would work just fine, as this would be two features scaled by $\dfrac{1}{n}$, where $n$ is just the number of timesteps taken by your water flow measurement system in one timestep of your corrosion data measurements.
If your corrosion measure has varying timesteps then you should use the sum of the water flow since averaging would cause inconsistency. For example, the average of flow in 100 days could be the same as the average in 10 days, and the corrosion in 100 days would be way bigger. Having two similar inputs that map to two completely different outputs would cause performance issues for a regression algorithm.


Extracting sections from document based on list of keywords - Python

The ideal way to get the related sentences would be to try to get a sentence vector for the sentences you want to categorise and then compare the vectors of your predefined keywords with the obtained sentence vectors .
You can get the sentence vectors by just averaging the word vectors of the words present in the sentences . Once the sentence vectors are obtained , you can use cosine similarity to compare the keyword vectors and the sentence vectors . The one with the max cosine similarity will give you the result .


Data preprocessing: Should we normalise images pixel-wise?

We first center our data by subtracting the mean of the batch.  We also divide by the standard deviation, so our formula becomes:

$ z = \frac{x - \mu}{\sigma} $

where:

$ x $ is the pixel value

$ \mu $ is the arithmetic mean of the channel distribution

$ \sigma $ is the standard deviation of the channel, calculated as such:

$ \sigma = \sqrt{\frac{\sum^N_{i=1} |x_i-\mu|^2}{N}} $

By standardizing the data in this fashion, we obtain a mean ($\mu$) = 0 and a standard deviation ($\sigma$) = 1.  This allows SGD to converge faster.

It is easier to see how feature scaling can benefit simpler models such as SVMs, but CNNs benefit similarly.  (Mini-)Batch normalization makes every pixel have a similar distribution, which helps SGD converge faster.  This is because networks generally have shared hyperparameters.  For instance, the learning rate is a multiplier for all weights in a network, rather than having a learning rate for every weight; therefore, we want a single learning rate to have proportional influence in all regions.  Otherwise, the learning rate could make some weights over-correct and other weights under-correct.

This article does a great job of explaining CNN preprocessing.  I'll try to highlight the key points (the following images are pulled from it)

Let's consider a face recognition challenge:

Taking the mean (left) and standard deviation (right) of the batch, we get the following:

Our centered data resembles a face, but more importantly our standard deviation image shows great variance along the borders and everywhere except the face.  From our z-score formula, we can predict that the standardized values near the border and irrelevant areas will be relatively squashed more than values around the face.  The resultant normalization appears as such:

Normalizing the input images is widely used and often considered an essential preprocessing step.  However, making an absolute statement such as, "normalization is always beneficial" or "normalization is never beneficial" would be misguided.

In the scenario you presented, normalizing the data would not benefit the CNN, but the network would still be able to learn the data well.  However, this is a very primitive situation, and most real-world problems have significantly more noise and variance for which to account.  In general, normalizing the input image will improve SGD convergence time.


Cost function for Ordinal Regression using neural networks

Another approach was suggested in this paper for face age estimation:
https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Niu_Ordinal_Regression_With_CVPR_2016_paper.pdf
The authors use a number of binary classifiers predicting whether a data point is larger than a threshold, and do this for multiple thresholds. I.e. in your case the network would have three binary outputs corresponding to

larger than 0
larger than 1
larger than 2.

For example, for 'large (2)' the ground-truth would be [1 1 0]. The final cost function is a weighted sum of the individual cross-entropy cost functions for each binary classifier.
This has the advantage of inherently weighting larger errors more because more of the individual cost-entropy terms will be violated. Simply doing categorical classification of the ordered outcomes doesn't inherently have this feature.


How are deep-learning NNs different now (2016) from the ones I studied just 4 years ago (2012)?

You are right in that the basic concept of a deep NN hasn't changed since 2012.  But there have been a variety of improvements to the ways in which deep NNs are trained that have made them qualitatively more powerful.  There are also a wider variety of architectures available today.  I've listed some developments since 2012, grouped by training improvements and architecture improvements:

Improvements to training deep NNs

Hardware: The most obvious change is just the inexorable progression of Moore's law.  There is more computing power available today.  Cloud computing also makes it easy for people to train large NNs without needing to buy a huge rig.
Software: The open source software for deep learning is really enormously improved from 2012.  Back in 2012 there was Theano, maybe Caffe as well.  I'm sure there are some others, too.  But today we also have TensorFlow, Torch, Paddle, and CNTK, all of which are supported by large tech companies.  This is closely related to the hardware bullet point since many of these platforms make it easy to train on GPUs, which drastically speeds up training time.
Activation functions: The use of ReLU activation functions is probably more widespread these days, which makes training very deep networks easier.  On the research side, there is a wider variety of activation functions being studied, including leaky ReLU, parametric ReLU, and maxout units.
Optimization algorithms: There are more optimization algorithms around today.  Adagrad and Adadelta just been introduced in 2011 and 2012, respectively.  But we now also have the Adam optimizer and it's become a very popular choice.
Dropout: In the past few years, dropout has become a standard tool for regularization when training neural networks.  Dropout is a computationally inexpensive form of ensembling for NNs.  In general, a set of models trained on random samples of the dataset will outperform a single model trained on the entire dataset.  This is difficult to do explicitly for NNs because they are so expensive to train.  But a similar effect can be approximated just by randomly "turning off" neurons on each step.  Different subgraphs in the NN end up getting trained on different data sets, and thereby learn different things.  Like ensembling, this tends to make the overall NN more robust to overfitting.  Dropout is a simple technique that seems to improve performance in almost every case, so it's now used de rigueur.
Batch normalization: It's been known for a while that NNs train best on data that is normalized --- i.e., there is zero mean and unit variance.  In a very deep network, as the data passes through each layer, the inputs will be transformed and will generally drift to a distribution that lacks this nice, normalized property.  This makes learning in these deeper layers more difficult because, from its perspective, its inputs do not have zero mean and unit variance.  The mean could be very large and the variance could be very small.  Batch normalization addresses this by transforming the inputs to a layer to have zero mean and unit variance.  This seems to be enormously effective in training very deep NNs.
Theory: Up until very recently, it was thought that the reason deep NNs are hard to train is that the optimization algorithms get stuck in local minima and have trouble getting out and finding global minima.  In the last four years there have been a number of studies that seem to indicate that this intuition was wrong (e.g., Goodfellow et al. 2014).  In the very high dimensional parameter space of a deep NN, local minima tend not to be that much worse than global minima.  The problem is actually that when training, the NN can find itself on a long, wide plateau.  Furthermore, these plateaus can end abruptly in a steep cliff.  If the NN takes small steps, it takes a very long time to learn.  But if the steps are too large, it meets a huge gradient when it runs into the cliff, which undoes all the earlier work.  (This can be avoided with gradient clipping, another post-2012 innovation.)

New architectures

Residual networks: Researchers have been able to train incredibly deep networks (more than 1000 layers!) using residual networks.  The idea here is that each layer receives not only the output from the previous layer, but also the original input as well.  If trained properly, this encourages each layer to learn something different from the previous layers, so that each additional layer adds information.
Wide and deep networks: Wide, shallow networks have a tendency to simply memorize the mapping between their inputs and their outputs.  Deep networks generalize much better.  Usually you want good generalization, but there are some situations, like recommendation systems, in which simple memorization without generalization is important, too.  In these cases you want to provide good, substantive solutions when a user makes a general query, but very precise solutions when the user makes a very specific query.  Wide and deep networks are able to fulfill this task nicely.
Neural turing machine: A shortcoming of traditional recurrent NNs (whether they be the standard RNN or something more sophisticated like an LSTM) is that their memory is somewhat "intuitive".  They manage to remember past inputs by saving the hidden layer activations they produce into the future.  However, sometimes it makes more sense to explicitly store some data.  (This might be the difference between writing a phone number down on a piece of paper vs. remembering that the number had around 7 digits and there were a couple of 3s in there and maybe a dash somewhere in the middle.)  The neural Turing machine is a way to try to address this issue.  The idea is that the network can learn to explicitly commit certain facts to a memory bank.  This is not straightforward to do because backprop algorithms require differentiable functions, but committing a datum to a memory address is an inherently discrete operation.  Consequently, neural Turing machines get around this by committing a little bit of data to a distribution of different memory addresses.  These architectures don't seem to work super well yet, but the idea is very important.  Some variant of these will probably become widespread in the future.
Generative adversarial networks: GANs are a very exciting idea that seems to be seeing a lot of practical use already.  The idea here is to train two NNs simultaneously: one that tries to generate samples from the underlying probability distribution (a generator), and one that tries to distinguish between real data points and the fake data points generated by the generator (a discriminator).  So, for example, if your dataset is a collection of pictures of bedrooms, the generator will try to make its own pictures of bedrooms, and the discriminator will try to figure out if it's looking at real pictures of bedrooms or fake pictures of bedrooms.  In the end, you have two very useful NNs: one that is really good at classifying images as being bedrooms or not bedrooms, and one that is really good at generating realistic images of bedrooms.


Is it possible to ensure that all classes are represented in the output of a scikit-learn decision tree?

It's expected behavior that the tree will not define rules for all classes if the models doesn't see a need. If you just want to compare a neural net's class predictions, you can use tree.predict_proba(...) to get all classification probabilities. Use the outputs of the neural net after softmax and before binarize.

For the fold that had little knowledge of class 2, the probability would be 0 or close to 0 given how the tree was built. If you want to force the tree to represent class 2 and assuming that it is present in all folds of your cross validation, you can change the learning params to default.

Another, but not great way, it to upsample your data to a large degree. This is bad because it would invalidate the purpose of CV, but you could at least see a small representation of all classes and your mean classification scores will remain similar.

If there are plenty of each class, you can stratify the classes across folds.

Also, if the problem is that you have insufficient samples of a minority class, you should consider up sampling the minority during training. https://towardsdatascience.com/dealing-with-imbalanced-classes-in-machine-learning-d43d6fa19d2


How to create a new dataframe using the another dataframe

Not sure what you mean by "unique", I guess if there're n critics, m items, what you need is a dataframe with shape n*m rows and 3 columns, right? If so, try the demo:

#1. the original dataframe
df = pd.DataFrame([['a',1,5],['b',2,3],['b',3,2],['c',8,1],['a',1,5]],columns=['critic','item','rating'])

#2. create the first two columns(critic, item) by their permutation
from itertools import product
first_two_column = pd.DataFrame(list(product(set(df.critic),set(df.item))),columns=['critic','item'])

#3. merge the first two column with ratings, using left join to add nan, drop duplicate first
first_two_column.merge(right=df.drop_duplicates(),on=['critic','item'],how='left')


Can I classify set of documents using classifying method using limited number of concepts ?

Both methods work. However, if you retain all words in documents you would essentially be working with high dimensional vectors (each term representing one dimension). Consequently, a classifier, e.g. SVM, would take more time to converge.

It is thus a standard practice to reduce the term-space dimensionality by pre-processing steps such as stop-word removal, stemming, Principal Component Analysis (PCA) etc.

One approach could be to analyze the document corpora by a topic modelling technique such as LDA and then retaining only those words which are representative of the topics, i.e. those which have high membership values in a single topic class.

Another approach (inspired by information retrieval) could be to retain the top K tf-idf terms from each document.


Deep network not able to learn imbalanced data beyond the dominant class

1) A five-layer neural network is one heck of a complex model for a data set with less than 1 million points. (I’m trying to find a good link for this, but the intuition is that your choice of model should be driven by the complexity of the available data, and not by what you think the real target function is like.) If this is for a real-world project, a tool like XGBoost might work better on this data set — out of the box, you’ll spend less time dealing with problems related to imbalanced classes, poorly scaled data, or outliers. Of course if this is specifically for learning about neural networks, that advice isn’t much help!

2) For a class distribution that’s as skewed as your data, you might get more mileage from re-sampling the training data rather than re-weighting the classes during training. Down-sample the majority class first (just throw away majority samples at random); if that’s not satisfactory then try something more complicated like using SMOTE to up-sample the minority classes. Try taking this to the extreme; build a (collection of) new training sets by randomly sampling only 1,000 points from each class.

The intuition here is that, for neural networks, as far as I know, re-weighting classes basically means re-scaling the gradient steps during training based on the class weights. If the classes are skewed 10:1, that makes sense: we take a step that’s 10 times as far for a minority sample. If the classes are skewed 1000:1, as in your case, it makes less sense — we’ll take 1,000 small steps as we optimize on the majority class, and then a single gigantic step in an essentially random direction when we happen to see a minority sample, followed by 1,000 tiny steps trying to un-do this work, etc. We don’t see enough minority samples to allow information about their class to average out.


SGDClassifier - Why do I need to use argmax instead of argmin to find the lowest threshold satisfying given precision?

Okay, I solved this myself.
precisions >= 0.90 doesn't create an array with precision scores only above 90%, but it transforms this array to the array of Booleans, where precisions below 90% are turned to False and the others are True.
argmax, if there are multiple identical, maximum values (and True is max here) returns the first index of this occurrence.
I sometimes hate this book, why he just doesn't use method like "array.first_equals(True)"?


Advice on how to get started for someone a bit later in life?

I for myself wouldn't advise you to not take that route. I do agree with most of the points that were mentioned before but my conclusion is a slightly different one:

Trying to get into data science without any preceding field knowledge deserves respect and it is not impossible. Also, there is still a lack of qualified data scientist and I would predict that the demand is growing.

But you should consider the following points:

Data scientist is probably the job in the statistics/IT area that requires the highest level of education. Ask yourself if this is really what your aiming at.
If you choose to go on that route be aware that it will be extremely hard. Data science is so multifaceted and interdisciplinary that there is a lot to learn. Even if you'd have a couple of hours to study each day I'm afraid it would take years. Being able to work as a data scientist needs far more than an academical math knowledge.
You will have to compete with PhDs in their mid-twenties who already have years of experience in publishing and conducting machine-learning/math/programming projects. Employers will probably favor them over you.

If you don't care that it will be painful and want to start learning, I'd do the following:

Get basic knowledge of statistics, programming, and IT-infrastructure. You should be familiar with t-tests, ANOVAs, regressions and cluster-/classification techniques. You should start learning a statistical programming language. For starters I'd recommend R or Python. Basic knowledge of another higher programming language is useful. Knowing basic programming principles such as object-oriented programming doesn't hurt either. I'd recommend to take a look at Java since a lot of big data tools such as Hive, Hadoop or Spark are Java-based frameworks. Look at these frameworks, too. Understand what they are used for. At last you should have at least basic knowledge of databases and know the difference between a relational and a non-relational database.
Don't spend money on courses. The internet is full of free sources. Just take into account who offers them. There are a lot of lecturers and professors from very good universities who provide online courses or share their lectures.
Read books. Again: Don't spend money. Search the internet for free high-quality materials (see here for example).
Check out community sites such as Kaggle and join competitions. This is probably as close as you will get to real-job tasks.
Continue doing that for a couple of month and ask yourself again if this is really where you want to go. If yes, continue.

But realistically, here is what I would try to do:

Don't aim at becoming a data scientist right away.
Consider to start in a statistics-related job (e.g. controller, business analyst, marketing manager, game designer etc.) and work your way up. I strongly believe that such a strategy is more likely to be successful.

Because no matter what you do before starting in the industry, you will have to learn on the job.


will k-means clustering converge to the same results given the same data set?

They won't necessarily be the same. Consider observations equally distributed over a circle (radius = 1). Depending on the initial centroids, the algorithm will converge on different solutions. For instance, consider the case where two centroids are initially located on each side of one of the circle's diameters. Those can be any pair of points, and the algorithm will already be converged with different solutions.

However, there exist cases where the algorithm will necessarily converge to the same solution. For instance, consider points equally distributed over a segment, in a 2 cluster problem. It is quite clear (although a bit harder to explain) that any initialization will eventually converge to the same solution (this actually need sadditional assumptions, such as no point being on the edge of both clusters, at least).

In your example case, with a more complex structure, the problem is more difficult to analyze. There are some problems which will likely give the same results every time, other that will yield different results. But anyway you can't be sure, in the general case, that it will fall back to a single solution.


As you probably know, the Gaussian distribution is fully specified by its mean and covariance. theta0, thetaL, and thetaU are, respectively, initial values, lower bounds, and upper bounds on the covariance. You can set them based on your knowledge of the particulars of the problem. To learn more about Gaussian processes, I suggest visiting this great Web site. It even offers a free ebook.


Feature usage for machine learning algorithm

I would build a logistic regression with multiple independent variables. I don't think this is the only possibility, but logistic regression makes sense you are trying to model for the probability of purchase.

Obviously, your dependent variable will be whether a product is purchased, so it should be binary. Your independent variables will also be binary. You can also merge the binary variables into smaller number of categorical variables.

You should do do at least pairwise correlation (e.g. Phi coefficient). You can use it to merge highly correlated variables, such as, Microsoft Windows Word and Microsoft Windows Excel into Microsoft Office.


How to find the class name of a new image from the pre-trained model

In the original training session, there will have been the mapping between class name and numerical class ID (e.g. Cat = 0, Dog = 1, ...). The model you use will output just the IDs and you need to find the mapping that was used.

If your new data has been added in addition to the old classes, they should each have a new unique ID. If you used VGG architecture to train on your new classes from scratch, then the model will not be able to make predictions for the original VGG classes.


Logistics regression with polynomial features vs neural networks for classification

I expect what he's referring to is the combinatorial explosion in the number of terms (features)
as the degree of the polynomial increases.

Let's say you have $N$ measurements/variables you're using to predict some other variable.  A $k$th degree polynomial of those $N$ variables has $ N+k \choose k$ terms (see here).  This increases very quickly with $k$.

Example:

Say we have N = 100 variables and we choose a third degree polynomial, we'll have
 $ 103\choose3$ = 176,851 features. For a five-degree polyomial it goes to:
$105 \choose 5$ = ~96 million features.  You'll then need to learn as many parameters as you have features.

Compare to a NN:

Compare this to using a fully connected NN
say we choose K fully connected hidden layers with $M$ units each.  That gives:
$ (NM) + (K-1)(MM) + M$ parameters.  This is linear in $K$ (though the $MM$ term attached to it might be big).

For N = 100 variables again, two hidden layers, and 350 features nodes per layer we get 157,580 parameters - less than we'd need for logistic regression with third degree polynomial features.

Representational power

(Added this section after seanv507's great comment)  * See caveat below

The argument above was just a numbers game - how big of a NN can you get while still having the same parameter count as logistic regression with polynomail features.  But you're getting at something when you say in your question:

  I mean, when we talk about neural networks we're usually looking at a
  model with a very large number of parameters so why would logistic
  regression be more computationally expensive?

Well said.

Efficiency

Neural nets are universal function approximators, and we know that polynomials can approximate functions too.  Which is better? What should one use? I'd bet that, for a given parameter "budget", a NN could better approximate "more" functions than a polynomial with the same number of parameters.  It wouldn't surprise me if there was theory to back it up, but I don't know it off hand.  seanv507's statement

  One possibility is to assume that the nonlinear activation function is quadratic...and identify what range of polynomials the NN could represent.

is an interesting idea.  Empirically, NN's have done better for many hard tasks than polynomial representations, and that's pretty strong evidence.

*Caveat

As seanv507 says - this is the hard part.  The above statements won't always be true - I'd argue they're probably mostly true.  If a low-degree polynomial basis nicely and reliably separates your classes, then it's probably worth using / trying polynomial features.


The SelectKBest class just scores the features using a function (in this case f_classif but could be others) and then "removes all but the k highest scoring features". http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html#sklearn.feature_selection.SelectKBest

So its kind of a wrapper, the important thing here is the function you use to score the features.

For other feature selection techniques in sklearn read: http://scikit-learn.org/stable/modules/feature_selection.html

And yes, f_classif and chi2 are independent of the predictive method you use.


Does cross_validate in scikit-learn automatically fits and train the model?

Yes, you do not need to fit or train explicitly when using cross_validate(). Also see this example from the SKLearn documentation:

from sklearn import datasets, linear_model
from sklearn.model_selection import cross_validate
from sklearn.metrics import make_scorer
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
diabetes = datasets.load_diabetes()
X = diabetes.data[:150]
y = diabetes.target[:150]
lasso = linear_model.Lasso()
cv_results = cross_validate(lasso, X, y, cv=3)

Just as in your case you can already get the results with no further steps, e.g.

cv_results['test_score']

gives this:

OUT: array([0.33150734, 0.08022311, 0.03531764])


Is there a good systematic approach to explore and analyze data (prior to modelling)?

Yes, there is this.

Always check the distribution of the target
. If classification,% of each class. This is important for many type of metrics and losses.
If regression, check for eventual high concentration in some fixed value. For example, it happens often that because of some mistakes in processing the target ends up being imputed and there are a lot of zeros.

Always check for drift. Typically these are id's. They have no meaning, but often they drive models to absurdly high performance because of some hidden bias.

Understand the distribution of features. often what you expect in production will not match what is in test. Think of some distribution of populations, that were sampled in particular way in training but will be more evenly matched in production. This happens often in churn/targetting models

If you are working with structured data, take time to understand what your features mean. If there is a data catalog available, read it. Example: Users tend to register on the website before purchase. Then if you are fitting a targeting model, probably due to use registration as a feature (BTW: This is a real life example that I have seen happen not only once, nor twice, but over three times at three different companies.)

If you are working with text. Take some time to read a couple of examples. Just don't throw it in the model. NLP is one of the few use-cases where you have the most intuition on what the model needs to understand.


With unbalanced class, do I have to use under sampling on my validation/testing datasets?

Great question...  Here are some specific answers to your numbered questions:

1) You should cross validate on B not B`.  Otherwise, you won't know how well your class balancing is working.  It couldn't hurt to cross validate on both B and B` and will be useful based on the answer to 4 below.

2) You should test on both C and C` based on 4 below.

3) I would stick with F1 and it could be useful to use ROC-AUC and this provides a good sanity check.  Both tend to be useful with unbalanced classes.

4) This gets really tricky.  The problem with this is that the best method requires that you reinterpret what the learning curves should look like or use both the re-sampled and original data sets.

The classic interpretation of learning curves is:

Overfit - Lines don't quite come together;
Underfit - Lines come together but at too low an F1 score;
Just Right - Lines come together with a reasonable F1 score.

Now, if you are training on A` and testing on C, the lines will never completely come together.  If you are training on A` and testing on C` the results won't be meaningful in the context of the original problem.  So what do you do?

The answer is to train on A` and test on B`, but also test on B.  Get the F1 score for B` where you want it to be, then check the F1 score for B.  Then do your testing and generate learning curves for C.  The curves won't ever come together, but you will have a sense of the acceptable bias... its the difference between F1(B) and F1(B`).

Now, the new interpretation of your learning curves is:

Overfit - Lines don't come together and are farther apart than F1(B`)-F1(B);
Underfit - Lines don't come together but the difference is less than F1(B`)-F1(B) and the F1(C) score is under F1(B);
Just right - Lines don't come together but the difference is less than F1(B`)-F1(B) with an F1(C) score similar to F1(B).

General: I strenuously suggest that for unbalanced classes you first try adjusting your class weights in your learning algorithm instead of over/under-sampling as it avoids all of the rigor moral that we've outlined above.  Its very easy in libraries like scikit-learn and pretty easy to hand code in anything that uses a sigmoid function or a majority vote.

Hope this helps!


How do I calculate distance of test data point from centroids in KMeans scikit-learn?

you can try below approach
centroids is a matrix  with all cluster centers
centroids=[[20,40,60,80],[60,120,180,240],[100,200,300,400]]

TestData_vector=[130,170,250,300] #you new test data as a vector

import numpy as np
from sklearn.metrics.pairwise import euclidean_distances
euc_res=euclidean_distances(np.array(centroids), np.array([TestData_vector]))

# normalize the result
normlaized_res= (1/euc_res)/((1/euc_res).sum())
#convert to list and sort it
normlaized_res_list=normlaized_res.tolist()
sorted_res=sorted(normlaized_res,reverse=True)
#get the nearset cluster
nearest_cluster=[]
for i in sorted_res[:10] :
    nearest_cluster.append(normlaized_res_list.index(i))


I was taught the best way is to find the error for each k then plot them and look for the "elbow" on the plot.


What is the "dying ReLU" problem in neural networks?

A "dead" ReLU always outputs the same value (zero as it happens, but that is not important) for any input. Probably this is arrived at by learning a large negative bias term for its weights.

In turn, that means that it takes no role in discriminating between inputs. For classification, you could visualise this as a decision plane outside of all possible input data.

Once a ReLU ends up in this state, it is unlikely to recover, because the function gradient at 0 is also 0, so gradient descent learning will not alter the weights. "Leaky" ReLUs with a small positive gradient for negative inputs (y=0.01x when x < 0 say) are one attempt to address this issue and give a chance to recover.

The sigmoid and tanh neurons can suffer from similar problems as their values saturate, but there is always at least a small gradient allowing them to recover in the long term.


Data quality improvement as a part of preprocessing: Imputation

Firstly, filling a column which has most values as null might not make sense. You'll have to look at the nature of the business problem you face. You can try dropping those columns. You can drop the rows which contain null values if the number of null values for columns in that row fall below a particular threshold (based on your data, of course).
Use: dataset.dropna(inplace=True) to achieve this

That being said, there are multiple ways you can fill (impute) null values:

You can try filling values based on the mean (or median or mode): dataset.fillna(dataset.mean(), inplace=True). However this approach has it's limitations if your column is non-numeric.

If you want to use an sklearn implementation, try using the Imputer method.

from sklearn.preprocessing import Imputer

impute = Imputer(missing_values='NaN', strategy='mean', axis=0)

impute.fit(dataset)

dataset = impute.transform(dataset)

For categorical columns, try using fillna with the most frequent value in a column:

dataset[column].fillna(value=df['column1'].value_counts().index[0], inplace =True)

more information on this method can be found here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html

Try using a KNN to fill in missing values. There's a package called fancyimpute which will allow you to do this:

from fancyimpute import KNN

dataset = pd.DataFrame(KNN(number_of_neighbors).complete(dataset))

You'll need to choose the number of neighbors appropriately. Also, fancyimpute needs a numpy array as input.

There are many other methods, but these should do the trick.


If you use sigmoid-like activation functions, like sigmoid and tanh, after some epochs of training, the linear part of each neuron will have values that are very big or very small. This means that the linear part will have a big output value regardless of its sign. Consequently, the input of sigmoid-like functions in each neuron which adds non-linearity will be far from the center of these functions.

In those locations, the gradient/derivative value is very small. Consequently, after numerous iterations, the weights get updated so slowly because the value of the gradient is very small. This is why we use the ReLU activation function for which its gradient doesn't have this problem. Saturating means that after some epochs that learning happens relatively fast, the value of the linear part will be far from the center of the sigmoid and it somehow saturates, and it takes too much time to update the weights because the value of gradient is small. You can take a look at here as a solution for this problem.
If I want to explain the math part, suppose that you are using sigmoid as the activation function. If $\sigma$ represents sigmoid, its gradient is $\sigma (1-\sigma$). Now suppose that your linear part, the input of sigmoid is a positive number which is too large, then sigmoid which is:

$$ \frac{1}{1+e^{-x}} $$

will have a value near to one but smaller than that. On the other hand $\sigma$ will be so close to zero, multiplying $\sigma$ and $1 - \sigma$ will result in a small value, which means the value of the gradient is very small. If the value of the linear part is very small, then $\sigma$ will be close to zero and $1- \sigma$ will be close to 1 but smaller than that. Again, multiplying these will lead to a small value as the gradient.


Avoid recycling when merging with R Data Table

What you want to do is join-like behaviour that is among others documented here:
joins and others

To make things readable here a version with dplyr

ex1 <- data.frame(id = 1:10, name = replicate(10, paste(sample(letters[1:5], size = 2), collapse = "")))

ex2 <- data.frame(otherinfo = (1:5)+50, othername = replicate(5, paste(sample(letters[1:5], size = 2), collapse = "")))

dplyr::left_join(ex1, ex2, by = c(name = "othername"))

   id name otherinfo
1   1   ac        NA
2   2   ac        NA
3   3   ab        NA
4   4   da        53
5   5   ab        NA
6   6   cb        NA
7   7   eb        NA
8   8   de        NA
9   9   ad        NA
10 10   ec        52


Group datetime64 values per week in dataframe

Is this what you want to do? I am not sure why you need 52 x 4 columns, since you have 3 categories in your Value field.

What I did is to create a new column with the number of Week in that Year, using the dt.week.

import pandas as pd
import numpy as np
import random

date_expected = np.arange('2006-01', '2006-06', dtype= 'datetime64[D]')
cat = ['True','False', 'Maybe']

value = [random.choice(cat) for i in range(len(date_expected))]

data = {'Date_expected': date_expected,  'Value': value }
df = pd.DataFrame(data)
df = pd.DataFrame(data)

df['week'] = df['Date_expected'].dt.week

df.groupby('Value').week.value_counts().unstack(0).plot.bar(figsize=(10,30))


Why are parameter updates downscaled by uncentered variance (instead of centered variance) in Adam optimizer?

If anything, you don't have access to the mean. When we talk about the variance, we talk about the variance of a distribution. The distribution in this case is the total collection of gradients at each step of the optimization process, which you don't know until you're done.


I'm trying to build a ResNet 18 model for Cifar 10 dataset, but I'm not able to fit the data dimension

The problem is that your ResNet-18 follows the architecture for ImageNet as outlined in the ResNet paper:

However, spatial input dimensions of ImageNet are different from CIFAR10 (32x32) so the architecture does not match your input. Instead you can follow the author's description of their CIFAR10 architecture in section 4.2 of the same paper:

The plain/residual architectures follow the form in Fig. 3
(middle/right). The network inputs are 32x32 images, with
the per-pixel mean subtracted. The first layer is 3x3 convolutions.
Then we use a stack of 6n layers with 3x3 convolutions
on the feature maps of sizes 32; 16; 8 respectively,
with 2n layers for each feature map size. The numbers of
filters are 16; 32; 64 respectively. The subsampling is performed
by convolutions with a stride of 2. The network ends
with a global average pooling, a 10-way fully-connected
layer, and softmax.


Tensor Decomposition in TensorFlow for multinomial time series dimensionality reduction

"Each time length can different, but there are the same features for each sample." You can make them all the same length, see Myth 1 of
http://www.cs.ucr.edu/~eamonn/DTW_myths.pdf

"Want to end up with is a set of samples with a single feature in time" This might work, if the multinomial time series are "Dependent" See fig 1 of
http://www.cs.ucr.edu/~eamonn/Multi-Dimensional_DTW_Journal.pdf
Otherwise, this is a very very bad idea.

"With that, I can do clustering using distance determined by dynamic time warping." Sure, but you can do clustering with multinomial time series under DTW. See [a].

--

[a] Nurjahan Begum, Liudmila Ulanova, Jun Wang, Eamonn Keogh (2015). Accelerating Dynamic Time Warping Clustering with a Novel Admissible Pruning Strategy  SIGKDD 2015


Using an unsupervised Isolation Forest, how does one identify the optimal number of outliers from the anomaly scores?

The way Isolation Forests seem to be used in most of the cases involve having some kind of prior "guess" about what proportion of outliers you expect (if you want to be on the safe side, you might be encline to increase it for instance). Based on the proportion you set (it is the contamination parameter in scikit-learn), the observations are labeled according to their anomaly score.

That being said, nothing prevents you from coming up with your own rules after you ran the Isolation Forest. You can plot the anomaly scores and then see for yourself if another threshold could make sense based on the distribution of the scores. This would be a more empirical rule.


Why does Ensemble Averaging actually improve results?

I think there is a missunderstanding in your question. In your question you imply that you take the average of the weights of the networks, and you should not do this. Instead, what you average are the predictions of different networks. For this reason, if you average two correct predictions the result will be a correct prediction, and the problem you were thinking about does not exist.


Python SkLearn Gradient Boost Classifier Sample_Weight Clarification

The sample_weight parameter is used both to weigh the splits as well as to weigh the errors.

Errors

For example, using the LeastSquaresError, the implementation uses the sample weights thusly:

def __call__(self, y, pred, sample_weight=None):
    if sample_weight is None:
        return np.mean((y - pred.ravel()) ** 2.0)
    else:
        return (1.0 / sample_weight.sum() *
                np.sum(sample_weight * ((y - pred.ravel()) ** 2.0)))

So if we provide large weights to certain samples, their individual error will have a bigger influence on the overall error for that sample set. The squared prediction error for each sample is scaled by by the corresponding weight, and the sum is normalised by the sum of the sample weights.

If no weights are provided, (leaving the default sample_weight=None), then they are initialised to a vector of ones, example here.

Splits

In the case that you set the parameter min_weight_fraction_leaf upon class instantiation, requiring each leaf to obtain a minimum fraction of the total sum of weight to indeed become a leaf, then sample_weight is also used to scale that value.  You can see it here:

    # Set min_weight_leaf from min_weight_fraction_leaf
    if self.min_weight_fraction_leaf != 0. and sample_weight is not None:
        min_weight_leaf = (self.min_weight_fraction_leaf *
                           np.sum(sample_weight))


CNN - How does backpropagation with weight-sharing work exactly?

I think you're misunderstanding what "weight sharing" means here. A convolutional layer is generally comprised of many "filters", which are usually 2x2 or 3x3. These filters are applied in a "sliding window" across the entire layer's input. The "weight sharing" is using fixed weights for this filter across the entire input. It does not mean that all of the filters are equivalent.

To be concrete, let's imagine a 2x2 filter $F$ striding a 3x3 input $X$ with padding, so the filter gets applied 4 times. Let's denote the unrolled filter $\beta$.

$$X = \begin{bmatrix}
    x_{11}       & x_{12} & x_{13} \\
    x_{21}       & x_{22} & x_{23} \\
    x_{31}       & x_{32} & x_{33}
\end{bmatrix}$$

$$F = \begin{bmatrix}
    w_{11}       & w_{12} \\
    w_{21}       & w_{22}
\end{bmatrix}$$

$$\beta= [w_{11}, w_{12}, w_{21}, w_{22} ]$$

$$F*X = \begin{bmatrix}
\beta \cdot [x_{11}, x_{12}, x_{21}, x_{22}] & \beta \cdot [x_{12}, x_{13}, x_{22}, x_{23}] \\
\beta \cdot [x_{21}, x_{22}, x_{31}, x_{32}] & \beta \cdot [x_{22}, x_{23}, x_{32}, x_{33}]
\end{bmatrix}
$$

"Weight sharing" means when we apply this 2x2 filter to our 3x3 input, we reuse the same four weights given by the filter across the entire input. The alternative would be each filter application having its own set of inputs (which would really be a separate filter for each region of the image), giving a total of 16 weights, or a dense layer with 4 nodes giving 36 weights.

Sharing weights in this way significantly reduces the number of weights we have to learn, making it easier to learn very deep architectures, and additionally allows us to learn features that are agnostic to what region of the input is being considered.

EDIT: To further motivate this, here's an animation of a 3x3 filter applied to a 5x5 input


Tensorflow: Unable to Save checkpoint after every 2 global steps during training the SSD model object detection

Use the saver class:

saver = tf.train.Saver()
with tf.Session() as sess:
    # Run your ops here with sess.run()
    saver.save(sess, 'my-model', global_step=2)

This will create checkpoint files with prefix my-model at every 2 time steps.

Reference: https://www.tensorflow.org/guide/saved_model


This error is common when the SPARK_HOME environment variable is not set.

In the shell type export SPARK_HOME=/path/to/your/spark


What does $\mathbf{w^Tx}+w_0$ graphically mean in the discriminant function?

Indeed, there is a lot to understand from the $g(\mathbf{x}) = \mathbf{w}^T \mathbf{x}+w_{0}$.
So let us break down what can be geometrically interpreted:
The set $\{\mathbf{x} \mid g(\mathbf{x}) = 0\}$ is a hyperplane, if $\mathbf{w} \neq \mathbf{0}$. Then, the dimension of the hyperplane is $n-1$, if $\mathbf{x}  \in \mathbb{R}^{n}$. So for $n=2$ a hyperplane is a line, for $n=3$, the hyperplane is a plane.
The vector $\mathbf{w}$ is the normal vector. It is orthogonal to the hyperplane.
So if you take vectors $\mathbf{x}',\mathbf{x}''$ of  the hyperplane, the vector $\mathbf{x}''-\mathbf{x}'$ will be orthogonal to $\mathbf{w}$.
Thus in your figure, $\mathbf{w}$ defines the orientation of the hyperplane.
Now $w_{0}$ translates the hyperplane by a corresponding vector $\mathbf{t}$ ( see here for more details).
In the following, lets assume $||\mathbf{w}|| = 1$, which does not change the hyperplane.
We can always find a basis $\mathbf{w},\mathbf{y}_{2},\ldots,\mathbf{y}_{n}$ of $\mathbb{R}^{n}$ such that $\mathbf{w} \bot \mathbf{y}_{i}$ for all $i$, that is $\mathbf{w}$ is orthogonal to all $\mathbf{y}_{i}$.
Now given $\mathbf{x}$, there are coefficients $\lambda_{1},\ldots,\lambda_{n} \in \mathbb{R}$ such that
$\mathbf{x} = \lambda_{1} \mathbf{w} + \lambda_{2} \mathbf{y_{2}}+ \ldots + \lambda_{n} \mathbf{y_{n}}$.
Therefore $ g(\mathbf{x})-w_{0} = \mathbf{x}^{T} \mathbf{w} = \langle \mathbf{x},\mathbf{w} \rangle = \langle \lambda_{1} w+ \sum_{i = 2}^{n}\lambda_{i} \mathbf{y}_{i},\mathbf{w} \rangle = \lambda_{1} \langle w,w\rangle = \lambda_{1}$
and thus $ g(\mathbf{x}) = \lambda_{1}+w_{0}$.
The geometric conclusion is therefore: $g$ depends on how you move $\mathbf{x}$ in direction of $\mathbf{w}$.
(1) Given a vector $\mathbf{x}$, and you consider $\mathbf{x}+\mathbf{u}$ for some $\mathbf{u} \bot \mathbf{w}$, then $g(\mathbf{x}) = g(\mathbf{x}+\mathbf{u})$.
So if you move $\mathbf{x}$ "along" the direction of the hyperplane, $g$ does not change its value. So given $x',x''$ of the hyperplane, if you move $\mathbf{x}$ by $\mathbf{x}'-\mathbf{x}''$, you will have $g(\mathbf{x}+(\mathbf{x}'-\mathbf{x}'')) = g(\mathbf{x})$.
(2) If you move $\mathbf{x}$ further away from the plane (along the line that contains $\mathbf{w}$), the value $g(\mathbf{x})$ will either increase or decrease, depending in which direction you move. So if you move $\mathbf{x}$ by $\mu \mathbf{w}$, you will have $\mu+\lambda_{1}+w_{0} = g(\mathbf{x}+\mu \mathbf{w}) \neq g(\mathbf{x}) = \lambda_{1}+w_{0}$, if $\mu \neq 0$.
(3) $g(\mathbf{x}) = 0$ means $\lambda_{1} = -w_{0}$.
In your image, you therefore have:
(1) A point $\mathbf{x}$ on the plane has value $g(\mathbf{x}) = 0$.
(2) Each point $\mathbf{x}$ above the plane has value $g(\mathbf{x}) > 0$.
(3) Each point $\mathbf{x}$ below the plane has value $g(\mathbf{x}) < 0$.


Can't train input variable with Keras+Tensorflow

Try explicitly setting the variable as trainable by setting Trainable=True.
Also after the network is created, check if the variable is in the list of
trainables https://www.tensorflow.org/api_docs/python/tf/trainable_variables


This article is great to generate time series data in python.
Hope this helps.

https://towardsdatascience.com/basic-time-series-manipulation-with-pandas-4432afee64ea


Optimising Kernel parameters using training data in GaussianProcessRegressor of Scikit-learn

Given scikit-learn's API, you create a separate instance for each optimizer and compare the results to see which optimizer makes better predictions. It would looks something like:

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF

gp1 = GaussianProcessRegressor(kernel=RBF, optimizer=optimizer1)
gp1.fit(X, y)
y_pred1 = gp1.predict(x)

gp2 = GaussianProcessRegressor(kernel=RBF, optimizer=optimizer2)
gp2.fit(X, y)
y_pred2 = gp2.predict(x)

This blog post gives a general overview of fitting Gaussian Processes, including scikit-learn's GaussianProcessRegressor.


How to extract Question/s from document with NLTK?

Check out chapter 6 section 2.2 of the NLTK book.

EDIT: since apparently the community wants me to copy/paste stuff, here ya go:

2.2   Identifying Dialogue Act Types

When processing dialogue, it can be useful to think of utterances as a type of action performed by the speaker. This interpretation is most straightforward for performative statements such as "I forgive you" or "I bet you can't climb that hill." But greetings, questions, answers, assertions, and clarifications can all be thought of as types of speech-based actions. Recognizing the dialogue acts underlying the utterances in a dialogue can be an important first step in understanding the conversation.

The NPS Chat Corpus, which was demonstrated in 1, consists of over 10,000 posts from instant messaging sessions. These posts have all been labeled with one of 15 dialogue act types, such as "Statement," "Emotion," "ynQuestion", and "Continuer." We can therefore use this data to build a classifier that can identify the dialogue act types for new instant messaging posts. The first step is to extract the basic messaging data. We will call  xml_posts() to get a data structure representing the XML annotation for each post:

>>> posts = nltk.corpus.nps_chat.xml_posts()[:10000]

Next, we'll define a simple feature extractor that checks what words the post contains:

>>> def dialogue_act_features(post):
...     features = {}
...     for word in nltk.word_tokenize(post):
...         features['contains({})'.format(word.lower())] = True
...     return features

Finally, we construct the training and testing data by applying the feature extractor to each post (using  post.get('class') to get a post's dialogue act type), and create a new classifier:

>>> featuresets = [(dialogue_act_features(post.text), post.get('class'))
...                for post in posts]
>>> size = int(len(featuresets) * 0.1)
>>> train_set, test_set = featuresets[size:], featuresets[:size]
>>> classifier = nltk.NaiveBayesClassifier.train(train_set)
>>> print(nltk.classify.accuracy(classifier, test_set))
0.67


Reasons that LIME and SHAP might not agree with intuition

As pointed by Sean Owen, this is probably linked to correlation in the input variables.

Concerning LIME : you can read here : https://christophm.github.io/interpretable-ml-book/lime.html (paragraph 5.7) why a correlation between inputs could pose problem, more specifically due to an implementation problem. (gaussian sampling ignore features correlation).

Shapley values are designed to take correlation into account by construction (more specifically trough the symetry condition, hinting that the more variables are similar the more they will have a similar effect). However, as stated in https://arxiv.org/abs/1705.07874 (paragraph 4.2), SHAP rely on the independence of input to approximate Shapley values.

So both techniques you used are subject to caution when used on data with important correlations.


How significant the variables like "id", "region code" etc are in the predictive modelling?

That depends on the problem and the data. Usually, at hackathons, there are at least several thousand id's and hundreds of region codes in your data and your task is to train on one dataset and predict on another dataset with new customers in existing regions.

In this case, you can exclude the id-column. Other categorical variables like region, age-group, race, etc, are usually relevant. The region code can be used as a categorical variable, but it depends on the number of regions. If there are hundreds of region codes in your data, map these codes to a larger region. For example, assume your data contains the region codes of all American counties. The easiest way to go is to map each county to its respective federal state and use the state instead of the county.


How to do feature selection for clustering and implement it in python?

Often people confuse between unsupervised feature selection (UFS) and dimensionality reduction (DR) algorithms as the same. For instance, a famous DR algorithm is Principal Component Analysis (PCA) which is often confused as a UFS method! Researchers have suggested that PCA is a feature extraction algorithm and not feature selection because it transforms the original feature set into a subset of interrelated transformed features, which are difficult to emulate (Abdi & Williams, 2010).
An UFS approach present in literature is Principal Feature Analysis PFA. The way it works is given as;
Steps:

Compute the sample covariance matrix or correlation matrix,
Compute the Principal components and eigenvalues of the Covariance or Correlation matrix A.
Choose the subspace dimension n, we get new matrix A_n, the vectors Vi are the rows of A_n.
Cluster the vectors |Vi|, using K-Means
For each cluster, find the corresponding vector Vi which is closest to the mean of the cluster.

A possible python implementation of PFA is given below;
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from collections import defaultdict
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.preprocessing import StandardScaler
import pandas as pd
# create some dummy data
df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
                   'num_wings': [2, 0, 0, 0],
                   'num_specimen_seen': [10, 2, 1, 8]},
                  index=['falcon', 'dog', 'spider', 'fish'])
print(df)
class PFA(object):
    def __init__(self, n_features, q=None):
        self.q = q
        self.n_features = n_features

    def fit(self, X):
        if not self.q:
            self.q = X.shape[1]

        sc = StandardScaler()
        X = sc.fit_transform(X)

        pca = PCA(n_components=self.q).fit(X) # calculation Cov matrix is embeded in PCA
        A_q = pca.components_.T

        kmeans = KMeans(n_clusters=self.n_features).fit(A_q)
        clusters = kmeans.predict(A_q)
        cluster_centers = kmeans.cluster_centers_

        dists = defaultdict(list)
        for i, c in enumerate(clusters):
            dist = euclidean_distances([A_q[i, :]], [cluster_centers[c, :]])[0][0]
            dists[c].append((i, dist))

        self.indices_ = [sorted(f, key=lambda x: x[1])[0][0] for f in dists.values()]
        self.features_ = X[:, self.indices_]

# Usage
pfa = PFA(n_features=3)
pfa.fit(df)
# To get the transformed matrix
x = pfa.features_
print(x)
# To get the column indices of the kept features
column_indices = pfa.indices_

Results
        num_legs  num_wings  num_specimen_seen
falcon         2          2                 10
dog            4          0                  2
spider         8          0                  1
fish           0          0                  8
[[-0.50709255  1.73205081  1.23942334]
 [ 0.16903085 -0.57735027 -0.84802649]
 [ 1.52127766 -0.57735027 -1.10895772]
 [-1.18321596 -0.57735027  0.71756088]]

Reference
Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4), 433-459.


Why my neural network does not predict decimal values in range [-1,1]? When it is able to predict the integer values

First of all, during the 10000 integer test, did you use all the integers from 0 to 9999 in training? If yes, then you have fully covered the whole input range. This means that while testing, you actually feed the network with data that are identical to the training data, therefore the accuracy is very high. What is the result of the network if you train it with 10000 randomly sampled integers in range of 10,000 and then test it with ALL integers in range of 10000? This test will reveal if your encoder generalizes well enough.

Also keep in mind that the smaller the number, the more difficult it is for the network to train due to vanishing gradient. Therefore, decimal numbers can saturate the learning process and lead to lower accuracy. Try changing the batch size (use smaller batches) and see if you decrease your output error.


K-fold cross validation when using fit_generator and flow_from_directory() in Keras

So, I haven't found any solution regarding this application of cross-validation in fit_generator(), I hope it comes in one update of the Keras package, since cross-validation is an important part of training models.

What I have done so far, basically I split the dataset first then I pass the data and labels to the fit_generator. At the end of each step I save the model, at the beggining of each step I load the preciuos model to continue the training over the same model, but with a different k-fold of the dataset. I'm pasting the part of my code where I use this approach, hope it helps.

## Training with K-fold cross validation
kf = KFold(n_splits=k_folds, random_state=None, shuffle=True)
kf.get_n_splits(images_file_paths)

X = np.array(images_file_paths)
y = np.array(class_labels)

i = 1
for train_index, test_index in kf.split(X):
    trainData = X[train_index]
    testData = X[test_index]
    trainLabels = y[train_index]
    testLabels = y[test_index]

    print("=========================================")
    print("====== K Fold Validation step => %d/%d =======" % (i,k_folds))
    print("=========================================")

    trainGenerator = Generator(trainData,trainLabels,batchSize=batchSize,imageSize=imageSize,augment=True,grayMode=grayMode)
    valGenerator = Generator(testData,testLabels,batchSize=batchSize,imageSize=imageSize,augment=False,grayMode=grayMode)

    try:
        model = load_model(weights_path, compile=True)

    except Exception as OSError:
        pass

    model.fit_generator(
            trainGenerator,
            steps_per_epoch=len(trainData),
            epochs=epochs,
            validation_data=valGenerator,
            validation_steps=len(testData))
    i+=1


local minima vs saddle points in deep learning

Let me give an explanation based on multivariate calculus. If you have taken a multivariate course, you will have heard that, given a critical point (point where the gradient is zero), the condition for this critical point to be a minimum is that the Hessian matrix is positive definite. As the Hessian is a symmetric matrix, we can diagonalize it. If we write the diagonal matrix corresponding to the Hessian as:
$$
  D =
  \begin{bmatrix}
    d_{1} & & \\
    & \ddots & \\
    & & d_{n}
  \end{bmatrix}
$$
the Hessian being positive definite is equivalent to $d_1 > 0, \dots, d_n>0$.

Now let's think about deep learning cost functions. Deep learning cost functions depend on lots of parameters in a very complicated manner, so the Hessian will have a complicated expression itself. For this reason, we can think that the values of $d_1,\dots,d_n$ are not biased towards negative or positive values. For this reason, given any critical point, the probability of every value $d_i$ to be positive can be assumed to be $1/2$. Moreover, it is reasonable to assume that the values of $d_i$ do not depend easily on the values of $d_j$, due to the high non-linearity of the Hessian matrix, so we will take the probabilities of them being positive as independent events.

For this reason, given a critical point, the probability of it being a minimum is:
$$
P(d_1 > 0, \dots, d_n > 0) = P(d_1 > 0)\cdot \cdots \cdot P(d_n > 0) = \frac{1}{2^n}
$$

The probability of any critical point being a minimum decreases exponentially with the dimension of the input space. In deep learning,
this space can range from 1000 to $10^8$, and in both cases $1/2^n$ is ridiculously small. Now we are convinced that, given any critical point that we come across, it is very unlikely that it is a minimum.

But what about maxima?

The maxima of a function are the minima of minus the function. For this reason, all the arguments used previously can be used to minus the cost function and we conclude that every critical point has probability of $1/2  ^n$ to be a maximum.

For this reason, given a critical point, the probability of it being a saddle point is
$$P(saddle) = 1 - P(maximum) - P(minimum) = 1 - \frac{1}{2^n} - \frac{1}{2^n} = 1 - \frac{1}{2^{n-1}}$$

Which is very close to 1 if $n$ is large enough (which typically is in deep learning).


What are some standard ways of computing the distance between individual search queries?

The cosine similarity metric does a good (if not perfect) job of controlling for the document length, so comparing the similarity of 2 documents or 2 queries using the cosine metric and tf idf weights for the words should work well in either case. I would also recommend doing LSA first on tf idf weights, and then computing the cosine distance\similarities.

If you are trying to build a search engine, I would recommend using a free open source search engine like solr or elastic search, or just the raw lucene libraries, as they do most of the work for you, and have good built in methods for handling the query to document similarity problem.


When you calculate RMSE error on multi-output models you will normally end up with 3 RMSE's. You can average those errors into one by specifying weights. They basically reflects of bad error on y1, y2 and y3 are for your application.

Even if the outputs are averaged equally, keep in mind that RMSE is "scale sensitive". If y1 value is way higher than y2 and y3 values, it will also produce bigger RMSE.

If you want to get rid of this problem I recommend you to use RMSE% wich is a relative error metric (scale invariant) as mentionned here : https://stats.stackexchange.com/questions/190868/rmse-is-scale-dependent-is-rmse


Logistic regression if 3 categories in outcome variable

Logistic regression can work on multi-class. Frankly it is not big different from binary classifier. The question is how does it find the coefficients for each class which you had displayed. By default to find the coefficients for a single class, it goes with one vs rest combination. So setosa vs remaining all classes. It gets coefficients iteratively by minimizing cross entropy. So ultimately you will end up with a matrix having a size of n_classes vs features.

Coming to the interpretation part of the question. To interpret the importance of a single feature, you can do it as - Given the coefficients of every other feature to be same (in comparing a class with another) including intercept, the class that has the feature with highest coefficient has greater chance of engulfing the new point. This being said the features having high petal length and width and low sepal length and width contribute good for a data point to belong to class virginica. But it does not make much sense to interpret each coefficient individually. Hope I made some sense.


Why is the softmax function often used as activation function of output layer in classification neural networks?

The softmax function is simply a generalisation of the logistic function, which simply squashes values into a given range.

At the final layer of a neural network, the model produces its final activations (a.k.a. logits), which we would like to be able to interpret as probabilities, as that would allow is to e.g. create a classification result.

the reason for using the softmax is to ensure these logits all sum up to 1, thereby fulfilling the constraints of a probability density. If we are try to predict whether a medical image contain cancer or not (simply yes or no), then the probability of a positive result (yes) and a negative result (no) must sum up to one. o the model produces a probability vector for each outcome, in pseudo-code: p = [yes, no].

If the final logits in this binary classification example were p = [0.03, 1.92], we can see that they sum to 1.95. This is not interpretable as a probability, although we can see the value is much higher for no. In other examples where there might be 1000s of categories, not just two), we can no-longer assess so easily, which is the largest logit. The softmax gives us some perspective and (quasi-) interpretable probabilities for the categories.

EDIT

As described by @Neil Slater in his comment below, using the softmax followed by a log loss function does indeed lead to a model that predicts values in the range of true probabilities, thus making them interpretable, as well as providing some nice statistical properties (see the Properties section of Maximum Likelihood Estimation).

Also note: minimising log-loss is equivalent to maximising the Maximium Liklihood Estimation. See a mathematical explanation here.

Implementations of the log-loss include:

the log_loss from Scikit Learn
binary_crossentropy in Keras (for binary problems)
categorical_crossentropy in Keras (for cases with > 2 classes)

These first two method names also highlight that log-loss is also the same as cross-entropy (in the general machine learning context of computing error rates between 0 and 1).


Welcome to the site! I think that you should be asking yourself "why" you can't do this in Data Studio. The likely answer is that Data Studio is a visualization tool while you are looking to do a transform of your existing data.

Most visualization tools are really only good at handling data in it's final form. It sounds like you still need to work on the data a little more to get to that stage. You should consider doing some SQL and/or scripting work (python, R, Julia, whatever) on your data to get it ready in the right format prior to importing it into Data Studio. Good luck!


What are true positive, false positive, true negative, false negative in the given example?

Assuming cat as a positive class,

Confusion Matrix:

  TN | FP
     FN | TP

True Positive(TP): Images which are cat and actually predicted cat
i.e. 25
True Negative(TN): Images which are not-cat and actually predicted not-cat
i.e. 50
False Positive(FP): Images which are not-cat and actually predicted as cat
i.e. 20
False Negative(FN): Images which are cat and actually predicted as not-cat
i.e. 5

  Precision:  TP/(TP+FP)
  Recall:  TP/(TP+FN)

Precision: 25/(25+20) = 0.55,
Recall: 25/(25+5) = 0.833


Navigating the jungle of choices for scalable ML deployment

If you're looking for distributed training, you're probably looking for Apache Spark. It's not itself a training library (although it comes with Spark MLlib which implements most common algorithms in a distributed way), but Spark core is something other implementations often build on.

Pyspark is just the Python API for Spark. Mesos is a cluster manager, like YARN, that Spark can use. Hadoop is for our purposes here a storage system (HDFS) and YARN, so might have the data you're reading. Spark builds on that too.

Kubeflow is, AFAIK, a toolkit for running many relevant data science things on the Kubernetes resource manager. If you're using K8S this will help you get some common training and notebook tools running. I don't know how much it specifically assists distributing training.

Dask is a framework for distributing Python code but I haven't used it; I don't think it's a training library per se. Sagemaker is a hosted notebook and serving tool, mostly. (Plug: if you're interested in Sagemaker, but also using Spark, you'd probably like our Databricks platform even more.)

Many libraries have distributed implementations now. TensorFlow has its own distributed mode. Horovod from Uber also helps distributed TensorFlow in a more efficient way. There's TensorFlowOnSpark too. (Plug again, because this is near and dear to my heart: HorovodEstimator = Horovod + Spark)

MXnet, xgboost also have distributed implementations on Spark. There's BigDL and deeplearning4j, also distributed deep learning for Spark.

And there's more. You're probably going to run into Spark no matter what as its even more commonly used for ETL and other processing related to data science. Then pick your preferred platform. Then pick some distributed training tools to try on your problem.


Improve classification accuracy on Iris data set - tf.contrib.learn Quickstart tutorial

An accuracy of 0.967 is a good score if you achieved it on the test set, so make sure that you are talking about the test set here. Secondly, increasing the number of iterations on the dataset won't do anything if your solution has already converged. Thirdly, adding more layers is not a very good idea, since you are already using a DNN classifier and could lead to overfitting. That being said, try to caculate a cross_validation score for your dataset, as then run it on a test set. In corss-validation you basically train your model on different chunks of your dataset, and validate on the chunk you leave out, then test it on a separate test set. This way you can actually see how your hyperparameters are behaving and you won't have to guess. Once you calculate the test score this way, it means you have good values of the parameters. If you still don't feel satisfied with the accuracy, its time to try a different algorithm, using more units in each layer and trying different activation functions.


Creating a training dataset from analytical solution

I think your question depends almost entirely on the actual problem that you're trying to represent, i.e. it depends on (your) expert knowledge.
Yes, data generated this way should be considered synthetic. However there's no general requirement to add noise: adding noise (as well as how much and in which way) is normally meant to make synthetic data more realistic, i.e. closer to real data.
In general with synthetic data it's important to show that the data is representative enough of the real phenomenon it represents. Again this depends on expert knowledge but most of the time it's essential to make the distribution of the data realistic as well. So yes, if you know the true probability distribution it's certainly a good idea to make the synthetic data follow it.


What came first? Backpropagation or Sigmoid?

I imagine the sigmoid function would have made its way into ANN's "since the beginning" since it is used in computational neuroscience to approximate the observed activation function of biological neurons. See for example this 1972 paper.


Interesting question! I have not encountered it before so here is a solution I just made up, inspired by the approach taken by the word2vec paper:

Define the pair-wise similarity based on the longest common substring (LCS), or the LCS normalized by the products of the string lengths. Cache this in a matrix for any pair of strings considered since it is expensive to calculate. Also consider approximations.
Find a Euclidean (hyperspherical, perhaps?) embedding that minimizes the error (Euclidean distance if using the ball, and the dot product if using the sphere). Assume random initialization, and use a gradient-based optimization method by taking the Jacobian of the error.
Now you have a Hilbert space embedding, so cluster using your algorithm of choice!

Response to deleted comment asking how to cluster multiple substrings: The bulk of the complexity lies in the first stage; the calculation of the LCS, so it depends on efficiently you do that. I've had luck with genetic algorithms. Anyway, what you'd do in this case is define a similarity vector rather than a scalar, whose elements are the k-longest pair-wise LCS; see this discussion for algorithms. Then I would define the error by the sum of the errors corresponding to each substring.

Something I did not address is how to choose the dimensionality of the embedding. The word2vec paper might provide some heuristics; see this discussion. I recall they used pretty big spaces, on the order of a 1000 dimensions, but they were optimizing something more complicated, so I suggest you start at R^2 and work your way up. Of course, you will want to use a higher dimensionality for the multiple LCS case.


Can we make an AI to fine tune other AI hyper parameters?

Yes, you can make an AI for that. By AI, you mean algorithm which finds hyperparameters efficiently. There are many ways to find hyperparameters which would then come under the AI like using bandits to find hyperparameters, Bayesian methods to find hyperparameters and many other methods exist. Search for hyperparameter optimization and you will find a ton of methods which improve upon grid search and random search.
AI in the purest sense of it, if I take, then, I will assume that you are talking about reinforcement learning since that deals with taking actions and changing its behaviour (in this case, hyperparameters). You can create an AI for that.


Build a dictionary of common words that frequently appear in these documents (e.g., MEDICAL, SEX, AGE, etc).  Then, for each word in the output from OCR, check whether it is similar to a word in your dictionary; if so, then replace it with the dictionary word.  "Similar" might be defined as "edit distance <= 1".

For example, your sample output has the word "sax".  If your dictionary contains the word "SEX", "sax" would be detected as a misspelling of "SEX" (edit distance = 1).

This post-processing will help you detect some of the common words and help you identify the fields.  What it won't do is clean up errors in the OCR of codes, as those can be anything.  There's probably no good way to handle that.

Also, do a Google search on how to use Tesseract.  There are some best practices that seem to improve its output (e.g., convert to greyscale TIFF format, deskew text, binarize, and more).

Finally, some commercial OCR software is significantly better than Tesseract or any other free OCR.  e.g., Abbyy seems to be well-regarded.


Done: MLE is a somewhat abstractly defined concept, but in
essence it is your best guess at a parameter. In this case we assume
that the observed frequency is your best guess.
You want to calculate the probabilty of your observation (HasOutdoorSeating=0 in this case), given
IsGoodRestaurant=1. That probability is not 0 (check the first sample for instance)


How to Use Shap Kernal Explainer with Pipeline models?

The reason is kernel shap sends data as numpy array which has no column names.
so we need to fix it as follows:

def model_predict(data_asarray):
    data_asframe =  pd.DataFrame(data_asarray, columns=feature_names)
    return estimator.predict(data_asframe)

Then,

shap_kernel_explainer = shap.KernelExplainer(model_predict, x_train, link='logit')
shap_values_single = shap_kernel_explainer.shap_values(x_test.iloc[0,:])
shap.force_plot(shap_kernel_explainer.expected_value[0],np.array(shap_values_single[0]), x_test.iloc[0,:],link='logit')


This might get you started. Phrase length is determined by the range() function. Basically this tokenizes and creates n-grams. Then it counts each token. Tokens with a high mean over all documents (occurs often across documents) is printed out in the last line.

from sklearn.feature_extraction.text import CountVectorizer
import numpy as np
import nltk

text = """DESCRIPTION PROVIDED BY AUTHOR: The goal of my a...
Author provided: The goal of my b...
The goal of my c... END OF TRANSCRIPT
The goal of my d... END SPONSORED BY COMPANY XYZ
The goal of my e... SPONSORED: COMPANY XYZ All rights reserved date: 10/21
"""

def todocuments(lines):
    for line in lines:
        words = line.lower().split(' ')
        doc = ""
        for n in range(3, 6):
            ts = nltk.ngrams(words, n)
            for t in ts: doc = doc + " " + str.join('_', t)
        yield doc

cv = CountVectorizer(min_df=.5)

fit = cv.fit_transform(todocuments(text.splitlines()))
vocab_idx = {b: a for a, b in cv.vocabulary_.items()}

means = fit.mean(axis=0)
arr = np.squeeze(np.asarray(means))
[vocab_idx[idx] for idx in np.where(arr > .95)[0]]
# ['goal_of_my', 'the_goal_of', 'the_goal_of_my']


In general when you are evaluating statistics there are two main approaches: either you use some distributional assumptions (a generating model for your data) or not. Whenever you can you would probably go with a statistical model. But often the distributional assumptions can be wrong or they are not available to you.

If you are doing inferences about the mean of the population, than you can rely on central limit theorem and build a statistic for that. The same for variance. However what about estimating something which does not rely on general assumptions? Bootstrap works for any kind of statistic, this is where it lies its power. It's simple, and does require only minimal assumptions.

And there is another problem which appears in practice. Estimating mean rely on central limit theorem. It is true that your set of assumptions required only independent and identical distributed data. But it's something more. CLT works on large samples, it works in the limit. For small samples it can go wrong more often. And what about if your distribution is skewed. While it's true that CLT relies only on i.i.d. the needed sample size for estimating a mean on a skewed distribution is in general greater that for a symmetric distribution.

I found bootstrapping very useful in two main situations: when the sample is fairly small (but not tiny) and when the distribution is not clean (suppose it's a mixture of two distributions). And is very useful because you can build not only estimates of the population parameters, but also estimations for confidence intervals of p-values. And because it's easy to apply I also found sometimes useful even when you are assuming some model. Doing in parallel can help you to see if there is something really spooky in your sample or assumptions if inferences produces too different results, it makes you think or re-evaluate your setup.


How can I detect blocks of text from scanned document images

There are two options :

Scan the images with a higher DPI. This should accentuate vertical separation between paragraphs.
Train a Deep learning model for Text Detection in scene. Examples : https://github.com/qjadud1994/CRNN-Keras and https://github.com/mvoelk/ssd_detectors


Meaning of subscript in min max value function

I'm supposing you are reading the paper on GAN where $[...]$ is the loss function. $\mathbb{E}_{x \sim p_{data}(h)} [...]$ now represents the expected loss where the data is taken from the distribution $p_{data}(h)$.

Edit:
$\mathbb{E}[ ]$ is an operator which gives the expected value of the function on which it is acted upon. When specifically acted on the loss function it is called risk or empirical loss. And L(z) that you are asking is the loss.

For more on its definition and explanation, see the below questions:

Risk function or expected loss function

why do we calculate risk when we already have loss functions?


Trying to train Stylegan, stopping with errors after first tick

After further trial and error, it seems that commenting out the lines using the "metrics" object in Stylegan's training_loop.py script on lines 207, 264, and 267 have resolved the crashing issue.


How do you compare term counts between two different periods, with different underlying corpus sizes, without bias?

The approach suggested by j.a.gartner is good if you want to analyze the quarter-to-quarter change in frequency for a given term, considering only that term in isolation.  What it doesn't do is evaluate the relative frequency compared to all other terms in the corpus.  For this you could compare frequency rank for all terms quarter-to-quarter. This is the analysis used in Zipf's Law.  Another advantage of this analysis is that you can test whether the generating process is or is not stationary (e.g. governed by same distribution quarter to quarter).

The advantage of comparing frequency rank is that it doesn't depend on the relative size of the corpus for each quarter (as long as they are all "large").  In simple language, the frequency rank of the word "Christmas" in the 4th quarter will probably be the same, regardless of whether the (US English news) corpus is 10,000 articles or 1,000,000 articles.