If I understand correctly, you noticed that a lot of correct predictions were due to a case which always has the same label and is very frequent in your data, right?
Let me say that I think you had a good reasoning: you analyzed the problem, designed a potential solution and then tried it. This kind of problem is very common, but in this case the solution is not really to delete the repeated cases in your training data.
What happens is that you have imbalanced data, that is you have a class which appears very frequently: when you deleted the very frequent instance you mentioned, your data size went from 50000 to 2000, so this instance represents 96% of your data. This instance has probably almost always the same label, so my guess is that this instance with this class as label represents 85% of your data. At first you thought your classifier was working well since you had 85% accuracy, but it's very easy for the classifier to reach 85% of correct predictions if the most frequent class represents 85% of the data: it just needs to always answer this class.
The first lesson here is that accuracy is not a good evaluation measure for imbalanced data: it's too simple, it can be very high even though the classifier doesn't do anything useful. That's why you should probably look at precision, recall and f1-score by class, and typically at macro f1 score as a global evaluation score.
Second point, I'm going to mention resampling because it's usually the standard answer to imbalanced data: the idea is simply to artificially "re-balance" the distribution of the classes in the data by either:

undersampling the frequent class, that is using only a random subset of this class. Note that your idea to delete the frequent instance is actually not so different.
oversampling the other classes, that is repeating the rare instances as many times as necessary to make the data balanced.

Importantly, the resampling must be done only on the training set, otherwise the evaluation on the test set is biased.
Resampling is quite easy, but the results are often disappointing. This is because quite often the problem is deeper: the classifier is just not able to distinguish the classes from the information provided by the features.
That brings me to my final point: the fact that you get 10% accuracy after removing the frequent case shows that the classifier doesn't have a clue what it's doing. Most of the time designing a good ML system needs some serious work with the features, especially with text data. The type of model (and parameters) can also play an important role. You didn't explain what kind of model and what kind of features you use, so there's not much I can say about this. Don't hesitate to post a new question with these details, and maybe we can help you improve this model.


Accuracy of word and sent tokenize versus custom tokenizers in nltk

Why would we want a custom tokenizer?

Segementation is a very large topic, and as thus there is no perfect Natural Language Tokenizer.  Any toolkit needs to be flexible, and the ability to change the tokenizer, both so that someone can experiment, and so that it can be replaced if requirements are different, or better ways are found for specific problems, is useful and important.

  How could I measure if it was better that NLTK's tokenizer?

Anytime you are trying to quantify performance (ie: better) you will need to first define what is meant by better.  Once this is done, then typically you would perform the using the various methods under measurement, and then compare the results against your definition of better.  A couple of links which discuss these topics:

Performance of Different NLP Toolkits in Formal and Social Text
Optimizing to Arbitrary NLP Metrics using Ensemble Selection


Multi-Step Forecast for Multivariate Time Series (LSTM) Keras

You should use Seq2Seq models. Seq2seq models represent, in the RNN family, the best for multistep predictions. More classical RNNs, on the other side, are not that good for predicting long sequences.

If you need to implement a seq2seq model in TensorFlow 2.0 / Keras, each model follows the following structure:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, RepeatVector, Dense, TimeDistributed
from tensorflow.keras.activations import elu

# these are just made up hyperparameters, change them as you wish
hidden_size = 50

seq2seq = Sequential([

    LSTM(hidden_size, input_shape = (input_sequence_length, no_vars)),

    RepeatVector(prediction_length),

    LSTM(hidden_size, return_sequences = True),

    Dense(hidden_size, activation = elu)

    TimeDistributed(Dense(1, activation = elu))

])

and then train it as usual with seq2seq.compile() and seq2seq.fit().

If you want to stack more LSTM() layers on top of each other, simply add them to the model I depicted above. Please keep in mind that LSTM models are very computationally expensice; without a good GPU even this "basic" model could be painfully long to train.

One modification I'd suggest, looking at your image, is to make the LSTM-encoder and -decoder parts of equal size and depth.

Alternatively, you can implement a more classical "Autoencoder-like" architecture, with LSTM() layers for encoding and decoding, and Dense() layers in the middle. However, seq2seq models are the most powerful at the moment.

To my knowledge, the only models more state-of-the-art than this are attention models. The problem is that they are so much state-of-the-art that TensorFlow/Keras doesn't have built-in layers for them, and you'd have to create your own custom layers (it's a pain). The only extensive implementation of attention models I found is from this blog post, but things are going to be very very complicated here. I didn't try to implement one yet.


In a convolutional neural network (CNN), when convolving the image, is the operation used the dot product or the sum of element-wise multiplication?

The convolution can be written as

$I(x,y) = \sum_{i=0}^n \sum_{j=0}^m k(i,j) I(x+i, y+j)$

where the kernel $k$ is of size $n$ by $m$. The image is size $N$ by $M$. Ok so in your example we are evaluating $I(0,0)$, our kernel is $n=3$ and $m=3$. Then our image is $N=5$ and $M=5$. If you go through the equation then you will see that you get exactly the result you showed.

$(1∗1)+(1∗0)+(1∗1)+(0∗0)+(1∗1)+(1∗0)+(0∗1)+(0∗0)+(1∗1)=1+0+1+0+1+0+0+0+1=4$

This is a dot product as defined as being a sum of products. We multiply element-wise and then we add them all together. The result is the answer you got $4$.

The entire convolution operation across the image is a matrix where each entry is this dot product.


How do you Show a Difference between Two Groups (Clustering)

Use MANOVA:

https://en.wikipedia.org/wiki/Multivariate_analysis_of_variance

  In statistics, multivariate analysis of variance (MANOVA) is a
  procedure for comparing multivariate sample means. As a multivariate
  procedure, it is used when there are two or more dependent
  variables, and is typically followed by significance tests
  involving individual dependent variables separately. It helps to
  answer:

  Do changes in the independent variable(s) have significant effects on the dependent variables?
  What are the relationships among the dependent variables?
  What are the relationships among the independent variables?


Is it possible to detect drift with real time predictions?

You may or may not use ground truth to detect drift.
According to google:

What is data drift? Data drift is one of the top reasons model accuracy degrades over time. For machine learning models, data drift is the change in model input data that leads to model performance degradation. Monitoring data drift helps detect these model performance issues.

In predictive analytics and machine learning, the concept drift means that the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.

Covariate shift is the change in the distribution of the covariates specifically, that is, the independent variables.

So I see data drift and covariate shift very similar if not equivalent. According to these definitions:

You need the ground truth to measure concept drift.
You don't need the ground truth to measure data drift.

In order to measure data drift:

You may or may not need all the training data. If you model the predictors (let's say I fit a gaussian variable to my feature $x_i$, and it has mean $\mu_i$ and standard deviation $\sigma_i$) and save the parameters of their distribution it might be enough to summarize the distribution and you don't need the full training data.
I don't think you need to have the same sample size for the serving data at all.
Data drift needs to be done in a batch way, so you can do it with your API as long as you store the results and analyze them after a period of serving time. It doesn't make sense to say that a single observation has drift unless in very extreme cases.


How to use spectral clustering to predict?

If the paper didn't elaborate on this, it must mean that they do 1). They look for the 10 closest neighbors of a new point and use majority voting to assign a cluster. On a side note, strange that they use an even number of neighbors, since it may introduce ties.

"What are the other ways to assign a(n) (existing) cluster to a new point?"

There are plenty of ways but it doesn't mean that they are all good! You could randomly assign points to clusters and use the size of the clusters as weight, but that isn't such a great method.

Generally, a nearest-neighbor approach is what is used. Now, it can be done in many ways:

KNN as I explained before (with any number of K)
Calculate the mean point of each cluster, and assign to the closest cluster
For each cluster, calculate the most distant point to the new point, then assign the cluster that has the "least far" distant point
And so on.


Machine learning - features engineering from date/time data

I would start by graphing the time variable vs other variables and looking for trends.

For example

In this case there is a periodic weekly trend and a long term upwards trend.  So you would want to encode two time variables:

day_of_week
absolute_time

In general

There are several common time frames that trends occur over:

absolute_time
day_of_year
day_of_week
month_of_year
hour_of_day
minute_of_hour

Look for trends in all of these.

Weird trends

Look for weird trends too.  For example you may see rare but persistent time based trends:

is_easter
is_superbowl
is_national_emergency
etc.

These often require that you cross reference your data against some external source that maps events to time.

Why graph?

There are two reasons that I think graphing is so important.

Weird trends
While the general trends can be automated pretty easily (just add them
every time), weird trends will often require a human eye and knowledge
of the world to find.  This is one reason that graphing is so
important.
Data errors
All too often data has serious errors in it.  For example, you may find that the dates were encoded in two formats and only one of them has been correctly loaded into your program.  There are a myriad of such problems and they are surprisingly common.  This is the other reason I think graphing is important, not just for time series, but for any data.


Does Tensorflow support a Decision Tree Classifier?

Basically I guess TensorFlow does not support decision trees. I quote from here,

  This is a big oversimplification, but there are essentially two types of machine learning libraries available today, Deep learning
  (CNN,RNN, fully connected nets, linear models) and Everything else
  (SVM, GBMs, Random Forests, Naive Bayes, K-NN, etc). The reason for
  this is that deep learning is much more computationally intensive than
  other more traditional training methods, and therefore requires
  intense specialization of the library (e.g., using a GPU and
  distributed capabilities). If you're using Python and are looking for
  a package with the greatest breadth of algorithms, try scikit-learn.
  In reality, if you want to use deep learning and more traditional
  methods you'll need to use more than one library. There is no
  "complete" package.

You can see from here that there are other learning algorithms implemented in TensorFlow which are not deep models.

You can take a look at here for tracking algorithms implemented in TensorFlow.


When creating a classification model, should predictors with little correlation to the response variable be included in the model?

In general, a Kruskal-Wallis test (or any other univariate test) cannot guarantee that there are no useful interactions with other features that a decision tree might still pick up. That means you won't know for sure until you try it.

You can test the performance with and without these "weak" features and compare results. Make sure you control for overfitting when including all features. Alternatively, after fitting the decision tree, you can look at the target response when varying the feature values ("partial dependence").


recommendations for self-study in ML/Deep Learning/the underlying math

If you're looking for an introduction to mathematics for data science, take a look at the Coursera course Data Science Math Skills.


Gaussian process regression: sudden increase of the prediction's variance

Hmm, I've seen this before. It seemed to be some kind of overfitting on the time component where only very nearby samples were taken into account, before defaulting to the prior (which has zero mean). That would explain why increasing the length scale would help.

Do you see large changes in the hyper parameters of the kernel between iterations (gp.kernel_.get_params())? You could consider using the hyper parameters from a previous iteration as initial guess for the next iteration.


How to implement a convolutional autoencoder?

Transposed Convolutions is what you are looking for, for more details take a look here:
https://medium.com/towards-data-science/types-of-convolutions-in-deep-learning-717013397f4d


Should use sklearn or tensorflow for neural networks?

Sklearn doesn't have much support for Deep Neural Networks. Among the two, since you are interested in deep learning, pick tensorflow.

However, I would suggest going with keras, which uses tensorflow as a backend, but offers an easier interface.


Extracting information with corresponding fields

you’re definitely on the right track with NER. As for determining the ‘class’ of what you’ve extracted, I think you have two major options:

Train a new entity type for each.
Use a set of rules that examine the term in context to determine the class.

I think you might have more luck with 2, if the language surrounding the terms is pretty static. Also, I haven’t watched it yet, but I thought this might help as well - seems to be about exactly your problem domain. https://www.youtube.com/watch?v=KrXJmaSHBJU.


Because the layers in the CNNs must be already able to extract features from images.

Normally this procedure takes thousands of iterations to be completed, so if they were not trained we could't have the concept of one-shot learning.


How to determine sample rate of a time series dataset?

General Explanation
Generally speaking, the frequency of the data is the difference between consecutive time-stamps.
If all is well, that difference will be constant across your time-series; in this case, this difference is the frequency of your data.
In other cases, it might be a bit more complicated - for example, you can have missing data, or data where some timestamps are shifted a bit forward or backward.
In which case, you might want to 'smooth out' those issues, for example by taking the median of such differences (if some samples come a bit too early and some come a bit too late), or the mode (if some samples are missing, meaning that most will come at the right time and some will be too late by an exact multiple of the real frequency).
Python (pandas) Example
Here's an example in Python, using the pandas library:
let's make up a time series with missing values. Its frequency is 5 minutes, but about 5% of the samples are missing (its data is random integers between 1 and 20, and the 1's have been dropped)
import pandas as pd
import random

index = pd.DatetimeIndex(start='2020-01-01 00:00:00', end='2020-04-01 00:00:00', freq='5T')
s = pd.Series(index=index, data=None)
s = s.apply(lambda x: random.randint(1,20))
s=s[s.ne(1)]

Now let's look at the time-difference between existing (not dropped) samples:
s.index.to_series().diff().value_counts()

yields
00:05:00    23629
00:10:00     1191
00:15:00       63
00:20:00        2

i.e. the vast majority of samples are really 5 minutes apart, but not all of them.
We'll take the actual frequency by taking the median of those differences:
s.index.to_series().diff().median()

yields
Timedelta('0 days 00:05:00')


activity_regularizer are used to control the output of a neural network. They tend to make the output smaller. Suppose the loss function is give as :

loss function = DataLoss + regularizationLoss

Then for weight_regularizer, regularizationLoss = f(Weights in a network).
But for activity_regularizer, regularizationLoss = f(Predicted outputs from a network). Activity regularizers are generally used when you are quite aware of the distribution of the test dataset.

For your second argument, I would say you are pretty wrong. ANN as well as CNN both can suffer from overfitting. In order to prevent the model to overfit, we generally use a lot of regularization techniques among which Dropout is quite popular.


How do I use matrix math in irregular neural networks such as those generated from neuroevolution (NEAT)?

It looks like I created a new way to use matrices with irregular neural networks in the process of answering my own question. Everyone must now refer to this method as the Capobianco Irregular Neural Network method. The CINN method basically treats the entire irregular neural network as a two layer fully connected network, albeit you treat most of the weights as being 0 (because they don't exist). This results in two sparse matrices. To be clear, you treat the input layer as being connected to EVERY hidden neuron, even if they are several layers away. Similarly, you treat the output layer as being connected to EVERY hidden neuron and input neuron.

The following example referencing the above picture uses initial h0 and h1 values of 0, which can be thought of as the values of the hidden states at the beginning of t0. There's two tricks I discovered.

First, just concatenate the [i0 i1 i2] input vector with the hidden state vector which gives us:

[ i0 i1 i2 h0 h1 ]

Next concatenate the weight matrix for the entire neural network except for the output weights:

     ( i0 i1 i2 h0 h1 )
(i0) [[ 1  0  0 w0  0 ]
(i1)  [ 0  1  0 w1 w2 ]
(i2)  [ 0  0  1  0 w3 ]
(h0)  [ 0  0  0 w9  0 ]
(h1)  [ 0  0  0  0  0 ]]

Every row shows the outgoing weight the value in parentheses has to the corresponding column value, with the exception of the area where the input vector values would otherwise "interact". We need this identity matrix to retain the original input values for the last step.

Before that it might be helpful to multiply out what we have using real values. To be clear, you want to find the dot product of:

                    [[ 1  0  0 w0  0 ]
                     [ 0  1  0 w1 w2 ]
[ i0 i1 i2 h0 h1 ] . [ 0  0  1  0 w3 ]
                     [ 0  0  0 w9  0 ]
                     [ 0  0  0  0  0 ]]

http://matrixmultiplication.xyz/ isn't perfectly accurate, but I really like how it shows you visually how each pair of terms are combined. This will result in a new vector [i0 i1 i2 h0* h1*]. h0* and h1* represent the final values of the hidden states at the end of the original time-step t0 (note that these will be the new h0 and h1 values at the beginning of t1), while the i0, i1, and i2 remain unchanged because we utilized the identity matrix.

Finally all we have to do is multiply this new vector by the matrix containing all the weights connecting to the output layer:

      ( o0 o1 o2)
(i0 ) [[ 0  0  0]
(i1 )  [ 0  0 w8]
(i2 )  [ 0  0  0]
(h0*)  [w4 w5 w6]
(h1*)  [ 0 w7  0]]

We can also do this all at once:

                    [[ 1  0  0 w0  0 ]  [[ 0  0  0 ]
                     [ 0  1  0 w1 w2 ]   [ 0  0 w8 ]
[ i0 i1 i2 h0 h1 ] . [ 0  0  1  0 w3 ] . [ 0  0  0 ] = [ o0 o1 o2 ]
                     [ 0  0  0 w9  0 ]   [w4 w5 w6 ]
                     [ 0  0  0  0  0 ]]  [ 0 w7  0 ]]

Note that is just for output vector at the end of t0. If you're not dealing with time data or using an RNN you don't need to worry about this, but if you're trying to determine outputs at a later time-step, you'll have to do something like this:

                  [[ 1  0  0 w0  0 ]                       [[ 1  0  0 w0  0 ]  [[ 0  0  0 ]
(input @ t0)       [ 0  1  0 w1 w2 ]   (input @ t1)         [ 0  1  0 w1 w2 ]   [ 0  0 w8 ]   (output @ t1)
[i0 i1 i2 h0 h1] . [ 0  0  1  0 w3 ] . [i0 i1 i2 h0* h1*] . [ 0  0  1  0 w3 ] . [ 0  0  0 ] = [o0 o1 o2]
                   [ 0  0  0 w9  0 ]                        [ 0  0  0 w9  0 ]   [w4 w5 w6 ]
                   [ 0  0  0  0  0 ]]                       [ 0  0  0  0  0 ]]  [ 0 w7  0 ]]

Note that this example is for an INN created using a genetic algorithm, so back-propagation is unnecessary in this example. If you wanted to utilize back-propagation you'd begin that process after calculating your output vector.


Why did Tufte call this a "superbly produced duck"?

If you assume duck to mean "irrelevant decorative elements" there are a few things that strike me as likely: (1) the "squares" only roughly indicate location of the target area; (2) squares represent volume, which is incongruent when overlaid on 2d geography; (3) shading of mountains/geographic features doesn't add detail.

Also problematic: comparison of h20 volume applied to crop types is difficult to discern using grids. Quick, which crop color uses more water??

If you want to compare volume to crop type then you should use a plain column chart, where it is easy to distinguish relative heights. This is analogous to why bar/column charts are preferable to pie charts - humans are better at comparing heights/lengths vs area/volume. If, on the other hand, the point of the graph is to compare crop type irrigation by area, you would want to display a column graph with regions next to each other, but grouped by crop type.

As far as a superbly produced duck, my guess is that the Applied Irrigation Water  is a duck, but a superb duck, at least when compared to the monstrous duck on the preceding page of the Visual Display of Quantitative Information 2nd Ed.


You already made it log scale which would have been my first idea. How about use lattice to make it three plots with different Y start and end? Each plot should have a center point that covers one of the three groupings you have and you can stack them 1x3 (vertically).


How to generate data if algo itself is involved in the process with a feedback loop?

This does sounds like a bad idea, since you are selecting your data beforehand and hence likely to cause sample bias. Have you looked at anomaly detection approaches?


inputs = Input(shape=[1])
emb = Embedding(input_dim=cap_shape_dummy.shape[1], output_dim=3)(inputs)
x = Flatten()(emb)
output = Dense(units=1,activation='sigmoid')(x)
model = Model(inputs=inputs,outputs=output)


How to prepare/augment images for neural network?

The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features.

The winners of the Data Science Bowl 2015 have a great write-up regarding their approach, so most of this answer's content was taken from:
Classifying plankton with deep neural networks. I suggest you read it, specially the part about Pre-processing and data augmentation.

- Resize Images

As for different sizes, resolutions or distances you can do the following. You can simply rescale the largest side of each image to a fixed length.

Another option is to use openCV or scipy.
and this will resize the image to have 100 cols (width) and 50 rows (height):

resized_image = cv2.resize(image, (100, 50))

Yet another option is to use scipy module, by using:

small = scipy.misc.imresize(image, 0.5)

- Data Augmentation

Data Augmentation always improves performance though the amount depends on the dataset. If you want to augmented the data to artificially increase the size of the dataset you can do the following if the case applies (it wouldn't apply if for example were images of houses or people where if you rotate them 180degrees they would lose all information but not if you flip them like a mirror does):

rotation: random with angle between 0° and 360° (uniform)
translation: random with shift between -10 and 10 pixels (uniform)
rescaling: random with scale factor between 1/1.6 and 1.6
(log-uniform)
flipping: yes or no (bernoulli)
shearing: random with angle between -20° and 20° (uniform)
stretching: random with stretch factor between 1/1.3 and 1.3
(log-uniform)

You can see the results on the Data Science bowl images.

Pre-processed images

augmented versions of the same images

-Other techniques

These will deal with other image properties like lighting and are already related to the main algorithm more like a simple pre-processing step. Check the full list on: UFLDL Tutorial


Which input to use when generating a new sequence

As you have seen, normally you need a "special token" to be given to the decoder as the first element in its input to start the autoregressive generation.

However, given that your output are real (floating point) numbers, it is a bit trickier, as you are not dealing with a discrete token vocabulary where you could simply reserve a token for that.

I would suggest using a specific value, like $0.0$. Your model should be able to figure out the no influence of the $0.0$ in the first position.

Another option would be to learn the value to be used as first token. You would have an extra trainable parameter that you use as value for the first position.


General Machine Learning Workflow Question

I agree with most of the answer. However, I think you are missing some points including the cross-validation step. I try below to provide an overview of a common machine learning project. I assume a common project is a supervised machine learning problem (like iris dataset).

1. You defining the 'scope' or aim of the project :

You have to define the purpose of the learning.
When working with business in a company, it is a good idea to correctly express the need, the value of the project and its goal.
You have to define the evaluation metrics (accuracy, recall, F1 score, AUC...). You can also define a minimum result you want to reach (say 80% accuracy for example).
You can also ask yourself about the level of interpretability you need (do you care about model explanation? If no, maybe you could try more blackbox algorithms such as boosting, neural networks...).

2. Explore your data :

Using statistics, visualization and intuition, try to learn your dataset and understand your features and labels.
You can also search for missing data and outliers. Correct these observations refers as data cleaning process.
Understanding your input variables will greatly help you to create and select relevant features.

3. Generating features/attributes from your own code/algorithm :

This phase refers as features engineering. It is about creating features relevant to the learning problem.
In this phase, you can clean your missing data and outliers in order to help the learning.
You can derive new features from input variables relevant to your learning problem (handle categorical variables, rescale your features, apply transformations on input variables).

4. cross-validation :

Cross-validation refers to your algorithm evaluation. In supervised machine learning, it is common, at least, to split dataset into 3 datasets (train, validation and test).
Train dataset (about 60% of data) aims to train the algorithm.
Validation dataset (20% of data) helps to find the best hyperparameters of your model (max depth for a tree, regularization for a linear/logistic regression...).
Finally, test set (20% of data) gives you the true result you get on unseen data. It is the final evaluation.

5. Machine learning, feed the matrix into an algorithm :

In this part, you train machine learning algorithms with regards to the cross-validation process (part 4).
You can test different models. Some yield different performance results. Interpretability is not the same neither.
To help the learning, you can diagnose your algorithm performs on both train and validation sets. This diagnostic is also called learning curves. It can tell you how to improve your learning. The purpose of learning curve is to help handle the underfitting/overfitting tradeoff. Underfitting is when you have a large bias error meaning your algorithm is not complex enough while overfitting means your algorithm is too complex and learns perfectly but is not able anymore to generalize learning on new unseen observations.
You can also look at residuals (errors between predictions and real values) to improve your algorithm.
Make features selection may also improve your algorithm learning.

6. Restitution

Interpret the model and the performance you get.
Create restitutions to business? Run into production?

Improve your machine learning model and performance is mostly about improving the above introduced points. By making new exploration, create new features, try a more powerful algorithm and so on, you can reach best results. Machine learning is a whole pipeline you have to optimize. I also think machine learning projects managements are really suitable with agile approaches.


Is gradient descent slower for finite differences?

(In response to the comments)

Consider logistic regression with labels +1/-1, where the function is $f(x) = \sum log(1 + \exp(-y * (x \theta)))$ . Taking finite differences requires computing $\sum log(1 + \exp(-y * (x (\theta + \epsilon_n))))$ (where $\epsilon_n$ is a small value for only one variable) for each variable (in each evaluation, you require plugging in the current values of all other variables too). The analytical solution is $\sum residual \theta$ - you only require plugging in the current values twice. You can keep a sum of $\theta x$ to save calculations (still more calculations with the differences approach), but as your function gets more non-linear, there’s less that you can precompute between variables – e.g. if you have 3 hidden units, changing the first unit will not let you reuse computations for the others.

In terms of big-oh notation, if you are looking at number of evaluations and you consider the gradient evaluation as $O(n)$ (keep in mind vectorization makes a difference though), I guess you won’t see much difference, but if you look at others aspects you’ll see that the analytical solution requires fewer calculations.

In addition, optimization techniques such as L-BFGS will not work if you don’t use precise gradients.


Measuring impact of missing feature on model performance

I would just remove each feature individually and see how that effects performance on my test set.  No need to get fancy.


The more accurate term would be "uncertain", but you are correct in your line of thought.
Your classifier (based on learned or engineered features distributions) can't determine with certainty which class to assign to this merginal cases.
It isn't necessary a case where they have a completely different distribution, more likelely that they are part of the same distribution but are located on its tail close to the tail of the distribution of the Second class (think 2 gaussians that at some area share similar density values).

Depending on the amount of such cases, you should examine them and attempt to trace the source of confusion. Maybe some clever preprocessing could help or different features (and maybe these are just outliers).

In the multi class case, you would probably get similar behavior, but each time on a subset of all classes. I.E. the model is certain that the sample is one of 2-3 classes (out of 10 total classes for example) but it can't be certain from which specific class. So you will get probabilities of similar values for these classes and very small probabilities for the rest.


How to load a saved model in TensorFlow?

The TensorFlow documentation for tf.saved_model.load might help:

  SavedModels from tf.estimator.Estimator or 1.x SavedModel APIs have a
  flat graph instead of tf.function objects. These SavedModels will have
  functions corresponding to their signatures in the .signatures
  attribute, but also have a .prune method which allows you to extract
  functions for new subgraphs. This is equivalent to importing the
  SavedModel and naming feeds and fetches in a Session from TensorFlow
  1.x.

You might have to use deprecated v1 api call
https://www.tensorflow.org/api_docs/python/tf/compat/v1/saved_model/load


What are the features to distinguish two short term signals (time series)

The examples that you provided look similar to one another in duration and amplitude, so normalization might not be a problem.  But consider whether you will always be comparing the same duration and whether you want to compare the relative change in the time series or the absolute change.  The relative change can be measured by normalizing the series so that the first value (starting point) is set to 0.  Depending on your problem, this might be worth considering.

Second, think about some sort of smoothing algorithm to get rid of some of the jitter in your data.  Examples include an Exponentially Weighted Moving Average or a second order Holt-Winters EWMA.

The suggestions in the comments are good ones... both Fourier Transform via FFT or moving to a wavelet formulation are interesting, but they are still a one-to-one in terms of the amount of data in (in temporal space) being equal to the amount of data out (in frequency space).

I would suggest first including the raw data or a down-sampling of the raw data with some additional features like area under the curve, max, min.  I would stay away from procedures which amplify the jitter of your data (like derivatives) in favor of those which decrease the jitter (like integrals).

A nice option, which implicitly smooths would be to take a running integration of your data and then plot the running integral rather than the values themselves.  This wouldn't require apriori smoothing and is still deterministic.

Following this, I suggest doing a whole bunch of feature extraction/feature engineering and then trying a LASSO regression to pick out the most useful features.

Some feature engineering ideas include:

Fourier Transform via FFT
Possibly only keeping the n (~10) largest Fourier Modes from above rather than the full set.
Integrals of your data.
Min.
Max.
Number of zero crossings.

Suggestions to look at the number of changes in the second derivative are tough with time series data due to its stochastic nature.  The slopes (derivatives) are very wild and depend largely on how much smoothing has been applied and how the data is being sampled.


The input to an LSTM must be a batch of sequences of vectors of real numbers, i.e. 3D tensor). Textual inputs are discrete tokens, so they are a batch of sequences of integers (indexes to the vocabulary table), i.e. a 2D tensor.
Before any reshaping, you must transform each integer value into a vector. For that, the usual approaches are one-hot encoding and embedding tables:

With one-hot encoding, you encode each integer index as a vector with 359 elements, where all are 0s except the position at the integer index, which is a 1.

With an embedding table, you keep a table with 359 vectors of an arbitrary dimensionality (the embedding size).

In either case, you get the "extra dimension" that you were lacking with integer indices.


One easy approach is to use a modified VAE loss with a stochastic encoder. In other words, let $z=E(x)\in\mathbb{R}^n$ and $y = D(z)$ be your pseudo-decoder (not a decoder, since you said your output is not meant to perform reconstruction; in fact it's not strictly necessary at all). Then, as in VAEs, define the probabilistic encoder $E(x)$ via
$$
\mu(x) = f_{\theta_1}(x), \;\; \Sigma(x) = g_{\theta_2}(x), \;\; z\sim \mathcal{N}(\mu,\Sigma)
$$
where $\Sigma$ is a diagonal covariance matrix. Use the reparametrization trick for backprop. Commonly this is written $z\sim q_\theta(z|x)$.

Now, let your current loss (not necessarily reconstruction) be written $\mathcal{L}(x,z,y)$. Our "pseudo-VAE" regularized loss for one data point is then just:
$$
\mathfrak{L}(x)=\mathcal{L}(x,z,y) + \beta\, \mathcal{L}_\text{KL}(x,z)
$$
where $\beta\in\mathbb{R}^+$ and the KL loss is written as
$$
\mathcal{L}_\text{KL}(x,z) = \frac{1}{2}\left[
-\log|\Sigma(x)| - n + \text{tr}(\Sigma) + \mu^T\mu
\right]
$$
assuming you want to be $\mathcal{N}(0,1)$ distributed.

Alternatively, you can treat each minibatch as approximating the marginal distribution, and then do moment matching. Concretely, let $X=(x_1,\ldots,x_n)$ be a minibatch of inputs, and $E(X)=Z=(z_1,\ldots,z_n)$ be the corresponding batch of encodings (embeddings). Suppose you want to make it "look like" $\mathcal{N}(\mu,\Sigma)$. Then let
$$ \widehat{\mu}(Z)=\frac{1}{n}\sum_i z_i \;\;\;\;\;\&\;\;\;\;\; \widehat{\Sigma}(Z)  =\frac{1}{n-1}\sum_i (z_i - \widehat{\mu}(Z))(z_i - \widehat{\mu}(Z))^T $$
so that our moment matching penalty (which you can add to your current loss function) is just
$$
\mathcal{L}_\text{M}(X) = \alpha |\mu - \widehat{\mu}|^2 + \gamma || \Sigma - \widehat{\Sigma}(Z) ||_F^2
$$

Another, slightly more complex approach, is to use an adversarial method, as in adversarial auto-encoders, adversarially learned inference (ALI), etc...
Basically, you train another network $C(z)$, which you train like the discriminator of a GAN (e.g., LS-GAN, WGAN-GP), but your generator is just $E(x)$. You train your critic $C$ to differentiate $ z\sim\mathcal{N}(0,I_n) $ and $z=E(x)$, and maximize its loss when training the "generator". This is equivalent to a conditional GAN, where the noise input is fixed (a Dirac Delta) or non-existent, the output is just the embedding, and the true samples are simply normally distributed random vectors.


How to find the probabilities of certain events occurring in a defined sequence?

My answer is inspired by this question in Code Review SE

import random
import string
from collections import Counter

def test_data(n, m, choices):
    return [[random.choice(choices) for _ in range(m)] for _ in range(n)]

data = test_data(50000, 9, ["A", "B", "C", "D", "E", "F"])

def subsequence_counts(sequences):
    return Counter([seq for seq in map(''.join, sequences)])

counter = subsequence_counts(data)

print(counter['BCEEECFAF'])

What the above code does:

1) The test_data function, creates a n lists with m items using elements of the choices parameters.

In specific, we create 50000 lists with 9 items length using the capital letters A, B, C, D, E, F

This is just for testing the code

2) The function subsequence_counts uses a list comprehension and a Counter to keep track of every single combination of the sublists we have.

3) Finally, you can access the Counter and get the frequency of any event you want.


Storing large data sets for python machine learning algorithm consumption

There are a lot of technologies out there to handle. These are the most popular ones I know of:

For moderate data to operate in chunks- Pandas -  pd.read_csv('train.csv', chunksize=chunksize)

For larger data - dask, Hadoop with R

Refer this for various other suggestions.


How do I cluster data that is a mix of text & categorical data?

First and foremost, there is no best way to do this task in general. It will require some creativity and problem-solving

Here is one potential idea. Clustering involves some kind of similarity metric. You can compute a similarity between text documents (after some term weighting like TF-IDF you could use the cosine similarity for example). You can also compute a similarity between objects with categorical data. Matching coefficients are good option for this. For example, http://en.wikipedia.org/wiki/Simple_matching_coefficient  is about as basic as it gets, but there are many other measures of similarity for categorical data that you can find with a simple google search.

So if I have one similarity matrix A that gives the similarity based on the text, and one similarity matrix B that gives the similarity based on the categorical attributes, then I could take some weighted combination of A and B and use it as an overall similarity matrix for clustering. Perhaps a convex combination would work well and give you some flexibility to choose which measure of similarity you want to weight more heavily (since it sounds like the categorical information might not be as good):

choose some parameter $0\leq \alpha\leq 1$ and compute an overall similarity matrix S as

$$\textbf{S} = \alpha \textbf{A} + (1-\alpha)\textbf{B}$$

Then you could use any number of clustering algorithms using the similarity information. For example, $k$-means should work just fine on a similarity matrix as input. Any type of graph/spectral clustering algorithm is also well suited for this type of input.


What is the name of this statistical interaction?

You might want to look at conditional entropy, H(A|B) and H(B|A).


Quantifying the Reproducibility of LDA Models

As you already noticed, Topic Models aren't reproducible due to the probabilistic algorithm. I did some topic models on a large corpus (10k documents, 30M words), and generally, between two different topic models about 10% of the topics were incomparable. My corpus is mainly English, but contains some Latin texts, and for a given number of topics I had between 1 and 3 topics consisting of Latin language words predominantly.

What to do now depends on what you want to learn from your Topic Model. When you want to "interpret" the Topic Model (using methods from the humanities), the best thing is to commit to one topic model (i.e., fixing the random seed).

When you want to study some derived measures (like topic diversity), make sure that the derived measures taken from different topic models are "stable", i.e., centred around one value with small statistical fluctuations.

You can also try to create an ensemble of topic models and try some techniques to get an "average topic model" out of it. I have no experience yet in this business and cannot suggest techniques here.


How to handle differences between training and deploying of an RL agent

If I train an agent for taking actions for 15 mins during the training process, is it ok I make my agent take actions at every 5 min interval during deployment?

It is impossible to say in general. Depending on the nature of the environment and controller, this may work just fine, or may completely destroy the ability of the agent to function at all.

I would suspect that you could get away with this change, if your controller is adjusting things that manage rates of change - e.g. the power levels of heating or cooling. That is because the amount of change forced per time step (by both controller and environment) is likely to scale with the size of time step. The agent is also likely to choose to continue the same action at $t' = 1, 2, 3$ that the original agent chose at $t = 1$, because the state will not have changed as much.

Much better though would be to alter your training simulation to be more accurate in this regard. There is a trade-off here. The more faithful your simulation is to real world use, the more likely it is that you will train an agent which works well in the real world. However, creating precise and accurate simulations can be hard work. I cannot tell how much work going from 15 minute time steps to 5 minute ones would be for you - just saying the change, it doesn't seem like much effort, but perhaps it is not under your control.

  If I train an agent on a particular ambient temperature profile, will the agent by heart / remember the profile used for training and expect the same temperature profile during deployment for it to work well ?

Over-fitting and failure to generalise are issues with neural networks wherever they are used. Yes, this is possible.

You can control for one or both of these changes between training and production systems - time step differences and temperature profile. Check to see whether your controller has generalised enough to deploy it, by testing your agent in a simulation with characteristics closer to production. For instance, you can have the simulation work with newly generated, and previously unseen (to the agent) external temperature profiles.

You can also, at least in theory, decrease the time step size in the simulation environment, although I recommend that you make that change during training instead, unless there is some strong reason why it is not possible. It is better to fix parameters that you can, rather than expect the agent to generalise over them - especially in your case where you only have one target value.

If you are not able to adjust your simulation to use it for testing, then you have to test your trained controller in a real-world environment. At that point you will want to monitor it closely to see how well it performs. That is good practice regardless of an additional test phase in simulation. Having a test phase in simulation is simply a good idea to mitigate risk and reduce costs.


Correct LSTM model to predict shuffled data

The input for an RNN like LSTM should be in the shape of (batch, timestep, features).
Base on my experience, if your current X_train.shape is in 2D you would like to reshape it into 3D array by X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1)) Then It will be something like (total number of samples, 7,1); 7 is because you have 7 different features.
By the way, you probably would like to start with a single layer of Dense and LSTM.
Further, using multiple layers of LSTM would require you to set return_sequences to True for all LSTM layers before the last one
model.add(LSTM(128,return_sequences = True))
model.add(LSTM(128))


Classification with feature not available at time of model creation

It seems the simplest way to go would be to build a line at each time step after 'time created'. With 'status_n-1', 'status_n'. That will allow to deal  with the notion of time in rows. You also might want to :

Deal with time relatively : instead of considering status at a given hour, you probably want to work with status since the creation of the task.
You will need to deal with task ponderation (longer tasks will get more rows) one way or another. You may add some ponderation in your model, based on 1/(expected length).
Add some features : to me it is unclear what feature you use for prediction. As is, you seems to be trying to predict time of completion based on start time / expected length. You won't learn much except for which tasks takes more than expected / some daily effects. I think this can be achieved more efficiently / more clearly with simple statistics.


Why is convnet transfer learning taking so long?

It might have to do with ImageDataGenerator. Data augmentation can be computational expensive. Remove that code and see if the model trains faster.


From what I understand - your problem is "sample selection bias" problem. Any kind of pattern to select a subset out of large data may lead to bias. This raises two question.

How to choose? Random/stratified random (If you have multiple classes) under sampling to obtain a smaller subset.
How big to choose? we can set percentage of undersampling.

Reducing training time : We can perform preprocessing : dimensionality reduction techniques to remove correlated data by aplying techniques like- PCA. Or Sparse reconstruction methods - Transform data to sparse data and then process.


Algorithms inherently supporting multilabel classification

If the algorithm inherently supports multi-label classification, then it's usually an implicit feature of the algorithm rather than an implementation detail.

For example, MLPs inherently support multi-label classification because the output layer has a perceptron for each class, and each of these perceptrons output a probability for that class.  The vector of outputs will predict an example's membership among all labels.  Similarly, the the leaves of a Random Forest (or any other tree-based algorithm) can contain arbitrary-length vectors that describe the probability of the example belonging to each label.

The one-vs-rest strategy is used generalize binary classifiers (e.g. logistic regression) to multinomial problems.  Multinomial problems are distinct from multi-label problems.

Edit:

Clarified my answer w.r.t. the differences between multinomial and multi-label problems.


Yes you can use LSTM or RNN,but from my point of view can be other possibilities.
If you want to use the recurrent neural network you should use a supervised training mode:

(many - to - many)

The input x1 is the move made by the rat at time1 and y1 is the move of the rat at time 2, x2 is the move of the rat at time 2 and y2 is the move of the rat at time3 and so on...

But for your kind of problem a recurrent netowork could not be the best choice. What i recommend you is to use a decision tree, or a random forest algorithm, since the possible outcomes of the rat are just 4: right, left, up, down, and you have a finite state of positions, "the maze". The best you can do is build a random forest algorithm where the features can be, for instance, the walls around the rat, the distances of these walls from it,the distance from the exit and the number of times the rat has passed through that state, and the labels are: right, left, up, down.

Remember that a machine learning model works better then a deep learning model if the data are not consistent and a random forest is a very powerful machine learning tool


How to approach this multi label classification problem and what will be its accuracy metric?

As with any data science problem: Approach the problem by first exploring the data and then trying the simplest solution that you think might just work, to some extent.

To explore the data, look at the distributions of all the features. Are they at all helpful to identifying the segments? How many trades do you have in the segments, are the classes balanced (more or less similar counts in all classes)? If not, you need to be careful with a metric like accuracy and you may be better off using F1 score, for example.

For an algorithm, how about starting with a decision tree and taking it from there? You can use it as a baseline model to compare to the results of other experiments. For example more complex algorithms using trees like boosted trees or random forest, or neural networks.

The performance metric is often averaged across all classes, using a technique called “1 versus all”. For that you start with the first segment, like “equity”. You can use the model to identify “equity” or “not equity”, and calculate the F1 score for this task. Repeat for all other classes and average.


Extract canonical string from a list of noisy strings

As a naive solution I would suggest to first select the strings which contain the most frequent tokens inside the list. In this way you can get rid of irrelevant string.

In the second phrase I would do a majority voting. Assuming the 3 sentences:

Star Wars: Episode IV A New Hope | StarWars.com
Star Wars Episode IV - A New Hope (1977)
Star Wars: Episode IV - A New Hope - Rotten Tomatoes

I would go through the tokens one by one. We start by "Star". It wins as all the string start with it. "Wars" will also win. The next one is ":". It will also win.

All the tokens will ein in majority voting till "Hope". The next token after "Hope" will be either "|", or "(" or "-". None of the will win in majority voting so I will stop here!

Another solution would be probably to use Longest common subsequence.

As I said I have not though about it much. So there might be much more better solutions to your problem :-)


So, your task is to detect the passive voice from sentences. Currently, you have defined some rules to detecting the passive voice and you have noticed that there some exceptions to your defined rules.
Therefore, it would be a good idea to develop a model to predict the probability of a sentence being passive (or active).
You can do this by encoding the sentence as a sequence of words (converted into word embeddings) using a Recurrent Neural Network (RNN) or LSTM. This encode will encode the words into a "hidden representation". This hidden representation can then be decoded using a neural network with a final softmax output layer which then outputs the probability that the sentence is written in the passive voice and the active voice respectively.
As this would be a supervised learning problem, you would need to label examples of passive- and active-voice sentences.


BERT classifier with Ktrain API is unable to predict new data

try without list operator []:

predictor.predict('I am very happy to meet you!')


Similarity coloring of a self organizing map

I don't know exactly how the above was created, but I did something similar in the late '90s and reproduced it recently using somoclu in Python. While I've not applied the colours back to the SOM cells, this would be pretty trivial.

I've put the notebook on kyso, I hope it's useful.


Why do you define the last dimension of input_shape as $3$? Just put your desired input dimensions accordingly and it should be fine:

input_shape = (img_width, img_height)

Update with the full code:

The best way would be to use TimeseriesGenerator instead of ImageDataGenerator but there seems there not flow_from_directory method meeting your needs. So, I think the best solution is to squeeze the last dimension of the generator output. Also, you have a color_mode option that allows to generate a 1-channel only tensor for grayscale images.
Full code of concerned parts:

model = Sequential()
model.add(Lambda(lambda x: x[:,:,:,0], input_shape=(*input_shape, 1)))
model.add(LSTM(units=256, return_sequences=True))
model.add(LSTM(units=128, return_sequences=True))
model.add(LSTM(units=64))
model.add(Dense(128))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    color_mode='grayscale')


Ranking ATM based on Utilization and Economic Data (Scoring/Rank Model)

If you want to use a prediction model, then you need a well-defined target. In your case, the Utilization of an ATM is a vague term. You cannot measure it as it is right now. If you solve this problem, then what you are looking at is algorithms called Ranking Methods or Learning-to-Rank

Abstract from Wikipedia

  Learning to rank or machine-learned ranking (MLR) is the
  application of machine learning, typically supervised, semi-supervised
  or reinforcement learning, in the construction of ranking models for
  information retrieval systems. Training data consists of lists of
  items with some partial order specified between items in each list.
  This order is typically induced by giving a numerical or ordinal score
  or a binary judgment (e.g. "relevant" or "not relevant") for each
  item. The ranking model's purpose is to rank, i.e. produce a
  permutation of items in new, unseen lists in a way which is "similar"
  to rankings in the training data in some sense.

Let's move to another working domain to make the example easier to understand. The example was taken from TowardsDataScience tutorial page.

You have an e-commerce shop and you want to rank your products with that way that will be sorted on a search page and maximize your revenue. The features you have are the attributes of the product and the target is if the visitor bought or not on that session.

You train then a Classifier (LogisticRegression for instance) and get the probability of prediction for that class as the ranking.


There seems to be no summary method for objects of class spikeslab, so the default summary method is used which prints the internal list elements.

Just print the object to get a nice summary - maybe that's what you want? Is this the output you expected?

> model1_ss
-------------------------------------------------------------------
Variable selection method     : AIC
Big p small n                 : FALSE
Screen variables              : FALSE
Fast processing               : TRUE
Sample size                   : 7
No. predictors                : 3
No. burn-in values            : 500
No. sampled values            : 500
Estimated mse                 : 7.5765
Model size                    : 2

---> Top variables:
                            bma  gnet bma.scale gnet.scale
Mentally.Unhealthy.Days   2.290 2.331    13.003     13.236
Physically.Unhealthy.Days 0.926 0.951     1.084      1.113
-------------------------------------------------------------------


Signal classification with convolution neural network

A CNN usually requires a lot of data. And of course even more data if you have  a very large feature set. And even more data if you have many output classes. The number of output classes makes the necessary amount of data samples to increase exponentially.

The most important part of any machine learning project is to extract the most telling features from your raw data. Ask yourself in your signal what are the telltale signs that should tell your model what class the signal was drawn from.

You should try to use some feature extraction techniques, then you should try to use a model better suited for the size of your dataset.

Data pre-processing and feature extraction

Data pre-processing goes from your raw data and remolds it to be better suited to machine learning algorithms. This means pulling out important statistics from your data or converting your data into other formats in order to it being more representative. For example if you are using a technique which is sensitive to range, then you should normalize all your features. n your case if you have a signal you might want to extract some information about the temporal and frequency domain transforms. You can try something like empirical mode decomposition (EMD).

Then, you want to use feature selection. From all the information you extracted from your data not all of it will be useful. There are machine learning algorithms such as: PCA, LDA, cross-correlation, etc. Which will select the features that are the most representative and ignore the rest.

In your case

As a general rule of thumb, the amount of data that I suggest to have for shallow machine learning models is $10 \times \#features$. For deep models I usually suggest $100 \times \#features \times \#outputs$. So you are limited by the size of your dataset. In your case a CNN is not recommended. Only recently a group published a quite controversial result of training a CNN with 100,000 examples. The community generally believes that this is way too little and it is not feasible to expect good results when training a CNN with very little data.

The Model

I suggest you use a general machine learning technique such as SVM. This will likely result in better result. Try these techniques instead and see what results you get: k-NN, kernel SVM, k-means clustering.


Daily access pattern of a real cloud storage

Try these links from UCI Machine Learning Repository:

Anonymous Microsoft Web Data
MSNBC.com Anonymous Web Data


Wrong result when solving: "chance that two random cards differ in color and value?"

There's an error in your code:
cards = np.array([c+str(n) for c in ("A", "B", "C", "D", "E") for n in range(1, 11)])

will produce "A10" and "A1" among other values, and
if a[0] != b[0] and a[1] != b[1]:

will return true when a=A10 and b=A1, for example. This is why you're probably consistently underestimating the number of differences.
An easy fix would be to use:
cards = np.array([c+str(n) for c in ("A", "B", "C", "D", "E") for n in range(0, 10)])

instead, which is more readable anyway.  But if I were doing this, I might use itertools.product.
Even after this fix it's normal to not always get exactly the theoretical value, but it's bad if there's a bias (i.e. consistent under- or over- estimation).


clustering multivariate time-series datasets

For most clustering approaches, first you need to choose a similarity measure. Some common default ones for raw time series are Euclidean distance and Dynamic Time Warping (DTW).

When you have computed the similarity measure for every pair of time series, then you can apply hierarchical clustering, k-medoids or any other clustering algorithm that is appropriate for time series (not k-means!, see this).

Update: if the number of time series (along with their size) makes it computationally not acceptable to compute pairwise distances, then one option can be to extract features from each time series, and then use such features as proxies for the time series in the clustering process. Some examples of such features are maximum value, number of peaks, mean value. There are libraries like tsfresh in Python that are meant to easily extract such kind of features from time series. With these features, then any clustering approach like k-means can be applied.


Notation for features (general notation for continuous and discrete random variables)

As far as I am concerned, there is no distinction between a continuous and a discrete variable when it comes to notation. So $P(X_k=x_k)$ is perfectly fine for either.


Poor performance of regression model for imbalanced data

Neural Networks can in general be interpreted as a regression problem and as such, you could apply well known ways of dealing with this. This paper gives you a good introduction to different approaches. For instance you can upsample the minority class, or you could do loss weight balancing during training.

For instance, consider the data point $x_i$ that can belong to one of two classes $a$ and $b$. For instance, class $a$ is here the minority. Then you would, during training, multiply the calculated loss with the weight. $$L_i=\begin{cases}l_a\cdot x_i,&\text{if datapoint $x_i$ is in the minority class}\\l_b\cdot x_i,&\text{if datapoint $x_i$ is in the majority class}\end{cases}$$ with $l_a$ > $l_b$ which would be natural for this example.

You could also look into continous performance measures that could work nicely with imbalanced dataset. For instance the generalized dice (F1) score.


What is the difference between a regular Linear Regression model and xgboost with objective set to "reg:linear"?

Linear regression is a parametric model: it assumes the target variable can be expressed as a linear combination of the independent variables (plus error). Gradient boosted trees are nonparametric: they will approximate any* function.

Xgboost deprecated the objective reg:linear precisely because of this confusion. It has been replaced by reg:squarederror, and has always meant minimizing the squared error, just as in linear regression.

So xgboost will generally fit training data much better than linear regression, but that also means it is prone to overfitting, and it is less easily interpreted. Either one may end up being better, depending on your data and your needs.


Specifying neural network output layout for object detection

I'd say how you order your sequences or normalize data will have much less importance than where you get features for those classifier/detection vectors (where you attach those to your base CNN network).

Your idea falls into approach called "Single Shot Detector" (as opposed to region proposal/sliding windows) and I'd recommend looking at SSD (Liu et al) and YOLO (Redmon, Farhadi) models – there is much more to it to get it performing well.


How to give names/labels to topics in LDA

I suppose it is late to answer, but it may turn out to be helpful for others.

LDA presents all topics as a mixture of keywords with their weights
For example, in gensim implementation it looks like this:

topic #0: 0.009*river + 0.008*lake + 0.006*island + 0.005*mountain + 0.004*area + 0.004*park + 0.004*antarctic + 0.004*south + 0.004*mountains + 0.004*dam

So you may name a topic using the keywords with the largest weights (like River+Lake).
Or you may use something like WordNet to find the most common hypernym for them.
For example, in python you can do it like this.

from nltk.corpus import wordnet as wn
wn.synset('river.n.01').lowest_common_hypernyms(wn.synset('lake.n.01'))


Why are the weights of my first layer and last layer in the CNN change while the middle layers don't?

Its ofcourse dataspecific, but this behavior is mostly likely not harmful. What could indicate dangerous behaviours is wild behaviour of cv-metric, learning rates  etc...

You can think of every cnn layer task to learn specific features/pecularities of a photo. In the first layers they will catch the most general patterns, edges etc... Middle layers are more specific and they look at some finer details but not all too specific. And final layers are there for the small details that can really discriminate the classes and make the right prediction.

So in your case weights are not updated significantly because information in middle layers is not that significant.


Conv bias or not with Instance Normalization?

Bias should not be used before InstanceNorm exactly as before BatchNorm. Whether you set affine to True or False, these layers first normalize the input, so any added bias is discarded.


Is there a flexible way to get the original data indices from each cell of a confusion matrix?

You can absolutely get this information, but not from the confusion matrices. You want to be comparing the prediction vectors themselves, not the confusion matrices, because as you've rightly identified, the confusion matrix dumps all the false postives / negatives into the same buckets (so that we can see how full each bucket is, also useful information).
You can do all sorts of comparisons of the prediction vectors to get the info you want (assuming you're using numpy arrays):
# Which examples did classifier A get wrong?
A_mistakes = np.invert(pred_A == ground_truths)

# Which examples did classifier B get wrong?
B_mistakes = np.invert(pred_B == ground_truths)

# Where did the classifiers make the same mistakes?
common_mistakes = A_mistakes == B_mistakes

# Where did the classifiers make a wrong prediction that the other didn't?
unique_mistakes = np.logical_xor(A_mistakes, B_mistakes)

You can then compare unique mistakes with A_mistakes or B_mistakes to trace them back to a classifier (or anything else you're interested in). You can also just np.sum the binary vectors to count the number of unique mistakes, common mistakes, etc.


Small number of estimators in gradient boosting

First what is n_estimators:

  n_estimatorsinteger, optional (default=10)
   The number of trees in the forest.

Gradient Boosting and Random Forest are decision trees ensembles, meaning that they fit several trees and then they average(ensemble) them.

If you have n_estimators=1, means that you just have one tree, if you have n_estimators=3 means that you have 3 trees and that it predicts the results of each tree and then it "averages" the result to get you the best.

It is not a good idea to force hyperparameters, cross-validation is the way to go in hyperparameter selection.

If you make your search a bit bigger you might be able to find different results.  Per example move wider in the subsample, in the penalties, in the max depth...

More parameters to search will mean more computational time.


Wich GPU-card to make deep learning with a MacBook Pro?

I also have a Macbook and I do data science on it. I looked at this same issue about  1 year ago and it's very possible that the landscape has changed since then.

But, at that time, I came to the conclusion that my best and most cost-effective options were to either (1) get a notebook with a GPU (of which there are many now due to the popularity of e-sports) or (2) get some space on AWS and their GPU servers. I really couldn't find an "enhanced Mac" option that would be as quick or work with as many tools as the above 2 options can offer you.

FYI - on the surface, the AWS option may seem expensive. But if you write your code locally and test on smaller samples and then only move that code to AWS when you need to do the deep learning phases (and only keep the AWS server running for those minutes) it can be a surprisingly affordable way to go.


'Training' data is really just splitting data you have already collected into test or training sets. For example, if you want to build a classifier for handwritten numbers, you collect thousands of samples of handwritten numbers like the MNIST database. When you think you have enough data to build a model, you then split it into train and test sets (usually by randomly assigning individual samples to one group or another at a specific ratio).

I think where your confusion lies is in the idea of collecting a 'training' set first as if it's truly independent from the test set. When collecting handwritten numbers, the researchers did not say, well, we have 10,000 samples, let's build a model with 10,000 samples and then have it running for our future data sets that we have not collected yet - in fact, that strategy is particularly bad and can lead to overfitting.

What you would do is take those 10,000 samples and split them - say 7,000 for training to build a model, 3,000 for testing said model - and maybe you would randomly build up many 7,000/3,000 models and take the average of the parameters for those models that you built. Then you can say our model predicts our test set with an accuracy of 97%, we think it will work well on data we have not yet collected.

How you collect that initial data set is specific to the process you are trying to understand. Maybe it's clicks on a website, images from a satellite, or electrical recordings from an ensemble of neurons. Sometimes you pay money to collect data - like a census or survey or maybe even buying another company that collected a bunch of user data that you want - typically, data collection is an inherent process to what you are doing and you are using statistical methods to make models and inferences about your population of interest.


Which metrics for evaluating a recommender system with implicit data?

There are two major types of evaluation - online and offline.
Online evaluation means showing the model's predictions to actual users. Since the goal of a recommender system to sell more products, the best overall best metric for a recommender system is increasing sales to actual users. This is best done by putting the model in production and A/B test if the model increase sales. This approach is not always possible given the limited resources (time or access to a production system).
Offline evaluation means simulating online evaluation by holding out existing data to evaluate the model.
If possible, split the data based on time. Train the model on the earlier data. Test the model on the later data. For a given pairing of product and user, the model will predict buy or not buy (binary classifier). The model can be evaluated as any binary classifier. For given domain, precision or recall may be more important.
However, this is not often times possible because time data many not be tracked or product-user pairs maybe sparse. If time is not tracked, the data may be split randomly to simulate time. If product-user pairs are sparse, product-user pairs are clustered along latent factors.


Naive Bayes Should generate prediction given missing features (scikit learn)

Your question is sensible. The way in which posterior probability is calculated in the classical Naive Bayes classifier (in sklearn) is like  summation of the conditional probabilities of the all the features in the dataset. Even though the features are treated as conditionally independent, to learn the classification probability all the features are always used in this setup. Once the model has been learned you still all those features to calculate the posterior for a new observation. The conditional independence is just an assumption that is taken to make the statistics and math obey the rules and work.
But slightly modifying the way in which the posterior is calculated you can use Bayesian approach to make predictions even with the absence of certain features. Using Bayesian approach to make predictions in the absence of certain features is still an ongoing work. You may want to have a look at this paper in which Bayesian approach is applied to astronomy to do classification with missing values.
Naive Bayes can be easily implemented in python as it is a simple calculation. The sklearn Naive Bayes does not support predictions with missing values and can be tricky to implement to what you are thinking. Maybe writing your own piece of code would be better :)


AttributeError: type object 'Lambda' has no attribute 'shape'

The error happens on this line:

lm = Lambda(decoder_concat_input)

Keras is not able to infer the input shape to the layer so you need to specify the input_shape parameter explicitly. But it is also not clear why you are using a Lambda layer in the first place. The layer is supposed to wrap a function but decoder_concat_input is itself a Layer. That is probably an error.


How to find patterns in a series of timestamps

The question can be super difficult if you only have the number of clicks per time-stamp. The reason is that you might find many different recurrent patterns!
For instance certain time periods in year, month or day (according to the functionality of website) may attract a certain temporal pattern of clicks.
But if you have more relevant features the story is different.

After all I will point out some directions and hope it helps:

Autocorrelation may initially give you a good insight about what is happening there.

Then go for Time-Warping methods to define similarity function and seek it along time-series.

Embedding in high-dimensional spaces disclose the temporal recurrent pattern of a time-series. It's mostly from physics literature and is called State-Space Recunstruction. This method transfers your time-series into a m-dimensional space using this and there you can see the time-series as a dynamic system which may help in detecting recurrent dynamics.


Why training a Restricted Boltzmann Machine corresponds to having a good reconstruction of training data?

1) I believe this comes from general property of Maximum Likelihood estimation, namely, that it is equivalent to minimization of KL-divergence between the model distribution and the (empirical) data distribution.

Proof: Suppose we have data $X=\{\mathbf{x}_1 \ldots \mathbf{x}_N\}$ drawn independently from unknown data generating process. To approximate its distribution we consider family of distributions $p_{\text{model}}(\;\cdot\;;\theta)$ parameterized by parameters $\theta$. Then
$$
\theta_{MLE}
=\text{arg}\max_{\theta}p_{\text{model}}(X;\theta)
=\text{arg}\max_{\theta}\prod_{n=1}^N p_{\text{model}}(\mathbf{x}_n;\theta)
=\\
=\text{arg}\max_{\theta}\sum_{n=1}^N \log p_{\text{model}}(\mathbf{x}_n;\theta)
=\text{arg}\max_{\theta}\underbrace{\left(\frac{1}{N}\sum_{n=1}^N \log p_{\text{model}}(\mathbf{x}_n;\theta)\right)}_{  \mathbb{E}_{\mathbf{x}\sim p_{\text{data}}(\mathbf{x})}\left[\log p_{\text{model}}(\mathbf{x};\theta) \right]},
$$
where $p_{\text{data}}(\mathbf{x})=\frac{1}{N}\sum_{n=1}^N \delta_{\mathbf{x}-\mathbf{x}_n}$ is (empirical) data distribution. Now by definition
$$
D_{KL}(p_{\text{data}} \| p_{\text{model}}(\;\cdot\;;\theta))=\mathbb{E}_{\mathbf{x}\sim p_{\text{data}}(\mathbf{x})}\left[\log p_{\text{data}}(\mathbf{x})-\log p_{\text{model}}(\mathbf{x};\theta) \right]
$$
Observe that the first term is independent of $\theta$ thus minimizing KL-divergence is equivalent to minimizing only the second term, which is a quantity with minus sign of what Maximum Likelihood maximizes. $\blacksquare$

Now if training data has high probability according to the RBM $\Leftrightarrow$ model distribution of RBM is close to empirical data distribution, then taking into account that RBM is trained using MCMC-based algorithm (CD-k/PCD-k), if we assume that state of Markov Chain particles are close to their stationary distribution, then feeding training example into RBM shouldn't change it too much after reconstruction. Therefore the low reconstruction error is expected in this case, and typically this happens in practice.

2) But as you correctly pointed out, even though it is convenient to measure reconstruction error during the training process, it is a poor measure of what actually is going on inside RBM during learning, since this is not the function RBM is aim to optimize (RBM maximizes probability of visible units $p(\mathbf{v})$ over all training data, which is equivalent to minimizing resp. KL-divergence).

Unfortunately, measuring (log-)likelihood directly is intractable in this model, but there exist better proxies to the true likelihood, such as pseudo-likelihood (more here or in the Hinton's practical guide you mentioned).


Choice of f1 score for highly imbalanced dataset?

To summarise this answer,

Macro calculates and F1 score for each class then averages them.

Micro calculates the recall/precision for each class, averages them then calculates the F1 score.

Micro average is preferable in a multi class situation where one class heavily outweighs the others.


Is it worth to upgrade CUDA and cuDNN while having older GPUs?

There is no right answer to this question, because there are many factors to consider, for instance:

Indirect dependency on specific/minimum CUDA version: with deep learning frameworks like pytorch and tensorflow, a specific version of the framework depends on a specific version of CUDA/cuDNN, so the decision of which (minimum) CUDA version to have is directly dependent on the framework version you need.

Bug fixes: new versions or old versions may introduce unexpected bugs that make upgrading/not upgrading not an option.

Performance regressions: some new CUDA versions may introduced performance regressions for older cards (e.g. this, this).

About support for new features, these are normally tied to new hardware (i.e. tensorcores), so it is not frequent to gain new features just by upgrading. But it is not impossible either ¯\(ツ)/¯
So, the answer is: it depends on the case.


Algorithm to calculate nerual network training time?

The best way to know is to train two epochs. The first epoch often takes longest, because data loading takes place with some caching.

The second epoch will give you an accurate time for each epoch.

Things to help the guesses that you might include in your question:

What are the specifications of p2.xlarge?
Where are your images stored? on the instance EBS storage? s3 storage? A separate file-system?
In which format is your data stored?
Perhaps the number of parameters in the model, but that can also be misleading!


is it acceptable if the reward of test of DQN is lower than reward of training of DQN in minimization problem?

There is a lot of information that might be useful for a more precise answer in your case, but is hard to extract from your question description.

I think I can understand the following:

You are training to minimise a cost instead of maximise a reward (this is unusual in RL, but not a problem).
Random agents score variably, anything from 120 to 400 average cost.
After many training steps, your agent is scoring 6.1 average cost, measured over the most recent 100 time steps.
In testing, the agent scores better, an average cost of 5.2.

The usual way of testing a DQN agent (or any off-policy agent, of which Q Learning or DQN are one type) is to stop exploring actions. I am assuming you do this too. During training, your agent was not always taking the actions that it determined were best. In tests, you are interested to see what the agent thinks is "best" and how good that is. So it should be expected that test scores better than recent training. If you were training using $\epsilon$-greedy action selection, then your scores would be consistent with $\epsilon \approx 0.1$

However, your measurements have limited accuracy in any case. It might be worth calculating your standard error on your test data, to get a sense of how accurate your measure really is. The easiest way is to run your test multiple times, create an array of scores and calculate the standard deviation, mean and standard error as usual using the test results as samples of a random variable.

  Could we say our DQN is OK?

That depends entirely on the problem. It seems that it does far better than acting randomly, which shows at least it has learned something.

In order to get a sense of how well it has done, you need to compare it with something meaningful related to the problem:

For simple problems, such as toy problems used to prove DQN works, it is possible to calculate an optimal answer and see how close the DQN can get.
For problems that humans can attempt too, you can compare the performance of an agent to a human. Ideally a human who is expert at the task.
For problems for which there are some publications, you can compare with state-of-the-art. This is true for many Atari 2600 games for instance.
For problems which have a practical purpose, you can compare with a goal, such as would the agent be good enough to make money or reduce loss compared to current practice.
For some problems you could compare to a simple rules-based agent, coded to perform the specific task.

  The more time steps should be more stable?? What is the effect of time steps?

Up to a limit, the more time and samples that a DQN trains on, the better its performance will be. The best thing is to plot this - number of timesteps vs average cost. You may see sudden increases or even "catastrophic forgetting" (where the agent does no better than random, or even worse) at some times. But if the agent has trained successfully there should be a general improvement over time that reaches a limit as the DQN approaches the maximum capability of the agent.


Does using unimportant features hurt accuracy?

I used the gradient boosting classifier in a project a while ago. We had about 130 features and since the system had to be implemented on a microcontroller I tried to reduce the number of features as far as possible.

Here is a plot showing the performance (F1-score) of GBC models trained with the top n most important features:

From that image it looks like "good-enough performance" is already reached with only 30 out of 130 features, while the other 100 (unimportant) features don't seem to have much positive nor negative influence.

Only when you zoom in a bit on the Y-axis you see this:

This shows that the performance reaches its peak (0.93) at 90 out of 130 features. So 40 features are unimportant if you want the best model.

To answer your question "does this have an impact other than computation time?": it does have an impact on performance, since the performance decreases again with more than 90 features, but only slightly.

To answer your second question "If it does hurt accuracy, how should I select which features to get rid of?":

I did the following:

Select the top n features
Use a cross-validated grid search to train a GBC model on the top n features
Repeat with the top n+1,... features
Select the model / features with the best score

How to select the top n features in step 1):

I actually didn't use GBC for feature selection, but random forests. I don't know why, but my experiments have shown that selecting features with RF and then training a GBC model on them works better than selecting features with GBC and then training a GBC model on them.


Ratio of positive to negative sample in data set for best classification

It is recommended to have around equal number of instances in each class. If not, then you should normalize the data. 4 or 5 times more positive class size could produce highly biased classifier. However, if your negative class is not well defined then there are ways to train your classes with only one class as mentioned in the paper, Learning from positive examples when the negative class is undetermined- microRNA gene identification.


Pandas merge column duplicate and sum value

You may use

df2 = df.groupby(['address']).sum()

or

df2 = df.groupby(['address']).agg('sum')

If there are columns other than balances that you want to peak only the first or max value,  or do mean instead of sum, you can go as follows:

d = {'address': ["A", "A", "B"], 'balances': [30, 40, 50], 'sessions': [2, 3, 4]}
df = pd.DataFrame(data=d)
df2 = df.groupby(['address']).agg({'balances': 'sum', 'sessions': 'mean'})

That outputs

         balances   sessions
address
A              70       2.5
B              50       4.0

You may add as_index = False to groupby arguments to have:

  address  balances  sessions
0       A        70       2.5
1       B        50       4.0


Overfitting for minority class after SMOTE w/ random forests

smote algorithm depends on the data set you have.
If you have severe data imbalance, like the one you have in your case smote algorithm may not be able to help if the variations within the minority class is very high and the similarities  between the two classes is very high.
But How to know if this is the case.
Try to duplicate samples from the minority class train a non linear svm and check the results if the classification accuracy is very low then this the case.

Smote use knn to create new samples but if the variation within the minority class is very high then using smote will use samples that are not real neighbors even.
To be honest with you , there is no clear solution for this problem but i can suggest the followings:
1. Try borderline smote : it is a modified version of smote algorithm
2. Try smote boosting : it is a modified version of adaboost where adaboost algorithm is augmented with smote
3. If you can modify the smote boost to consider  borderline smote instead of smote


How can I combine two datasets and assign factors to each?

Hierarchical indices

Have a look at here (scroll a bit underneath for the code)


I found an answer in a related post here. Another user was getting a similar error about ordering height, but theirs came with the suggestion to apply as.hclust() first. I converted the agnes object to an hclust and passed that directly into cutree. That seemed to solve the problem.


Get specific object dimensional position from image

Assuming you already have a logo sample to detect. In that case you can use OpenCV template detection. You need to pass source logo and target image.
Link - Template Matching using OpenCV


You've followed the right process well. (Well it's possible there's an error, like not randomly sampling the test set.)

I think the issue is simply that you have nevertheless overfit. The Kaggle held-out test set may not be, due to chance, that like the provided training data.  There's not a lot you can do except favor low-variance models more than low-bias models in your selection process.


The AlphaGo Zero paper tests a few different architectures:

Seperate 12-convolutional-block policy and value networks (as in AlphaGo)
A single 12-convolutional-block network with dual policy and value outputs
Separate 20-residual-block networks
A single 20-residual-block network with dual policy and value outputs
A single 40-residual-block network with dual policy and value outputs

In the AlphaZero paper AlphaZero Go wins 60 games in a 100-game match against the 20-block single resnet AlphaGo Zero, but it is not explicitly stated which architecture AlphaZero is using. It is either the 40-block single resnet since this was the best, or it is the 20-block single resnet in order to make a fair comparison with AlphaGo Zero.


Why won't my SVM learn a sequence of repeated elements

What you are facing is a small but crucial definition difference:
novelty detection:
The training data is not polluted by outliers, and we are interested in detecting anomalies in new observations.
outlier detection:
The training data contains outliers, and we need to fit the central mode of the training data, ignoring the deviant observations.
OneClassSVM is an Unsupervised Outlier Detection. Therefor your data needs to have outliers in order for the algortihm to detect them. My best guess, why its prediction every input as an outlier is, that if there are no real outliers, everything must be an outlier.
Let me demonstrate this quickly. I adjustet the kernel to linear
import pandas as pd
from sklearn import svm
import numpy as np

nu = 0.5
gamma = 1.0
ones= pd.DataFrame(np.ones(100))

clf = svm.OneClassSVM(nu=nu, kernel="linear", gamma=gamma)
clf.fit(ones)
clf.predict(-1) # -1
clf.predict(1) # -1
clf.predict(1.00001) # 1
clf.predict(2) # 1
clf.predict(10) # 1


As for stateful LSTM and its understanding, refer to here. Quoting an answer from there:

"I’m given a big sequence (e.g. Time Series) and I split it into smaller sequences to construct my input matrix X. Is it possible that the LSTM may find dependencies between the sequences?

No it’s not possible unless you go for the stateful LSTM. Most of the problems can be solved with stateless LSTM so if you go for the stateful mode, make sure you really need it. In stateless mode, long term memory does not mean that the LSTM will remember the content of the previous batches."

Therefore, stateful is useful if you wish to save the state of the neurons for the next training session instead of resetting it.

  "My goal here is to generate sequences that looks like something that
  the model have seen already."

Then maybe this sequence2sequence (seq2seq) LSTM encoder-decoder is exactly what you need.


Correct me if I'm wrong here:

You've already written the code that test/trains the model and you're looking for a way to put that model on the web somewhere so users can interact with it?

There's a couple of way's you can go about that, but it depends on what you need out of the final product. Is this something clients will be using? Friends? Just you? Flask allows you to prototype quick web apps and still maintain a good amount of control over what the finished product looks like. Spyre is an amazing way to make a web app at near light speed, although it may not be suited well to your use case, depending on what you're doing. It's a little harder to make unique-looking things with Spyre, but I always suggest it to people.

At any rate, the standard thing to do here is to Pickle your model after it's trained and then load it to memory in a web app. After that you can call the model.predict(X) on an X that is sent to your program by a user.


The effect of the image type and the image conversion on deep learning CNN model

I haven't seen this discussed anywhere really, so an interesting thought.

I think the answer will be that it doesn't matter. I think you should stick to one format (not important which, but don't mix them). Once the images are read into memory for processing and training a model, they are going to just be numbers in an array, regardless of the format they were loaded from.

Perhaps the numbers will differ slightly, but we usually normalise the input to something like the range [0, 1], so that wouldn't matter.

If your file formats change the image in a way, such that some features of the image are different (like the hue), then perhaps results will differ slightly, but the contents and spatial positioning of objects remain the same relative to themselves.

EDIT:

I should mention that converting between "image types" is nothing more than using a different compression algorithm to store the data. So when a photo is taken, the camera software compresses the raw sensor data into the target format (jpg, png, ...) and that format will have characteristic artefacts. Perhaps we don't recognise them, but they are there, and there is research done trying to reverse the information loss due to such compression algorithms. Here just a random example.


First, when we as humans interpret "Fyonair is from Fuabalada land" we use our knowledge of "is from" and "X land" to infer that Fyonair is probably a person and Fuabalada land probably a location. Therefore our process is not (at least not completely) unsupervised: we have seen this sentence structure before ("training") and we use our "model" to "predict" the two entities.

A good (supervised) NER is trained to recognize this kind of patterns as well. The example that you mention might be a bit too hard, but with something like "Dr Fyonair is the new CEO of Fubalada" a decent NER should be able to recognize that "Dr Fyonair" is a person (thanks to the "Dr") and that "Fubalada" is a company (thanks to the "CEO of") even though it has never seen these particular proper names.

So a standard (supervised) NER is supposed to recognize entities it has never seen before, provided it has clues in the sentence about them. Technically it's not really a state of the art NER if it recognizes only the entities that it has seen during training, it's just a string matching program. It's true that NERs make a lot of errors with entities which haven't been seen previously, but that's simply because they are harder to catch in general.


implementing an algorithm that mixes data clustering and linear regression

A lot of the values from your data seem to be of the same size, or very small difference, also it seems it is important to deal with noise. The same problem is true for image compression. I think a good approach would be to use Haar-Wavelet Transform which compresses the information by a very huge amount and reduces the noisy data, like it does for image compression. After the transofrm, you need to apply quantization on the resulted data, here you can simply set all values to zero which have the smallest absolute value up until some bound, the remaining values you can also quntize into groups, if their difference is less than some bound. By doing this, you get rid of all noise. From here you would have to specify more precisely, what you want to achieve next. You want to predict z?
For this task you can simply use Decision Tree learning methods, using the maximum information gain to decide on which attribute to make the first splits. Random Forests, which use a lot of different Decision Trees, where each one can be seen as an expert on some concrete task, should give very good results on this data, a good quantization method is very important for trees. You can also implement them all by yourself, without using any black box libraries, you learn much more from implementier it yourself. "McGrawHill_-_Machine_Learning_-Tom_Mitchell" Has very well explanations on those algorithms, explaining the mechanics behind all those black box like approaches.


Anyway to know all details of trees grown using RandomForestClassifier in scikit-learn?

I think Terence Parr's answer is now partly outdated. You can get the same (and more) with:

print("Tree depths: ", [t.get_depth() for t in model.estimators_])
print("Tree number of leaves: ", [t.get_n_leaves() for t in model.estimators_])


Detecting redundancy with Pearson correlation in continuous features

If the correlation between two features $x_1$ and $x_2$ is 1 that means that you can write $x_1 = c\cdot x_2 + a$. The only knew knowledge there is are those two constants, the individual values can be retrieved knowing this. I highly doubt there is anything a machine learning algorithm can learn from this and it is a fact that for some having this kind of correlation between features can hurt your performance quite a bit, so I would test it a bit but I would say it's very likely you can remove one of the two, and which one is not going to matter.


It reminds me of this question, the training loss is decreasing faster than the validation loss. I understand there is some overfitting, as the model is learning some patterns that are only in the training set, but the model is still learning some patterns that are more general, as the validation loss is decreasing as well. To me it would be more of an issue if the validation loss increased, but it is not the case.
Edit
Usually neural networks are trained with all the data, training by using mini-batch gradient descent already does what you mention in your approach without the need of storing the model in memory. So, I would train with as much data as possible, to have a model with the least possible variance. If you are not feeding the data using generators and the whole dataset doesn't fit into memory, I recommend to use them, or train with a model which is as big as possible given your memory limitations.


What is the structure of a MLP that allows for arbitrary decision boundaries?

I understand that the fact that MLP allows for arbitrary decision boundaries derives from the fact that an MLP is a universal approximator.
The theoretical area addressing the capability of neural networks to be able to approximate any function are the different flavors of the universal approximation theorem, for which there are two directions: the arbitrary width case (which you refer to when you assume that the ability comes from the three layers), and the arbitrary depth case.
The wikipedia page contains the mathematical formulation of both directions, as well as reference to their respective proof for different assumptions.