As per business statement it is a exceptional scenario when minimum subsidy will be provided. So for displaying common behavior of data you can drop these exceptional values from table. You can use box-plot to visualize spread of data and then remove else, if you are aware with max range in normal scenario you can drop rows having value more than that.


How can I know how to interpret the output coefficients (`coefs_`) from the model sklearn.svm.LinearSVC()?

Here's one (admittedly hard) way.

If you really want to understand the low-level details, you can always work through the source code. For example, we can see that the LinearSVC fit method calls _fit_liblinear. That calls train_wrap in liblinear, which gets everything ready to call into the C++ function train.

So train in linear.cpp is where the heavy lifting begins. Note that the w member of the model struct in the train function gets mapped back to coef_ in Python.

Once you understand exactly what the underlying train function does, it should be clear exactly what coef_ means and why we draw the lines that way.

While this can be a little laborious, once you get used to doing things this way, you will really understand how everything works from top to bottom.


Anybody know what this type of visualisation is called?

This type of visualization is called Arrow and Range plot.
You can refer here


Pandas Query Optimization On Multiple Columns

Short answer
There are far faster ways vectorized ways to do this than brute-force loops. One particular option while remaining Pandas-level would be
(tra_df.groupby(tra_df.columns.tolist())
       .size()
       .reindex(tst_df.values.T.tolist(), fill_value=0)

This should offer you an enormous performance boost, which could be further improved with a NumPy vectorized solution, depending on what you're satisfied with.

Long answer
Suppose that your training and test data looked something like this.
import pandas as pd; import numpy as np; from numpy.random import RandomState

rnd_st = RandomState(8675309)

tra_df = pd.DataFrame(dict(A=rnd_st.randint(0, 10, 10**3),
                           B=rnd_st.randint(0, 10, 10**3)))
tst_df = pd.DataFrame(dict(A=rnd_st.randint(0, 10, 10**3),
                           B=rnd_st.randint(0, 10, 10**3)))

First of all, you aren't really using Pandas correctly with
def f(A, B, x, y):
    data = df.loc[df[A].isin([x]) & df.B.isin([y])]
    return len(data)

isin isn't intended to match against single values like this, but rather a list or set of values when you don't have another option. In your case, you can achieve the same boolean index directly with
(df.A == x) & (df.B == y)

Furthermore, there's no point indexing your DataFrame again with df.loc only to take the length of it... you might as well just use sum directly on the boolean vector to count the matches. Putting this together and using apply over the test DataFrame, you can achieve your same output with
tst_df.apply(lambda x: ((tra_df.A == x.A) & (tra_df.B == x.B)).sum(), axis=1)

But this is still very slow - wrapping the above approach in a function and comparing it to your original approach on the sample data
%timeit your_way(tra_df, test_df)
1 loops, best of 3: 819 ms per loop
%timeit direct_er_way(tra_df, test_df)
1 loops, best of 3: 673 ms per loop

Much faster would be to use groupby and then reindex, as instead of brute-force looping this offers a vectorized solution where we are effectively hashing the counts.
(tra_df.groupby(tra_df.columns.tolist())
       .size()
       .reindex(tst_df.values.T.tolist(), fill_value=0)

Benchmarking,
 %timeit groupby_way()
100 loops, best of 3: 3.04 ms per loop
I'm not that well-versed in NumPy, but I can safely assume that were this function still not fast enough to meet your needs then a NumPy vectorized solution avoiding some of the overhead would be the next step.


There are multiple ways to tackle this. I'll suggest two here.

[1]
The one that probably requires the least amount of preprocessing work is to throw the event data into a recurrent neural net that can handle variable sequence lengths. Map the event categories to a small embedding space and run it through the RNN. You can then concatenate the remaining static features and put a feed-forward classifier on top.
This would take care of 1) and 2). In order to account for 3) there are several tricks.

You can append the time to event to the embedding vectors
You can multiply the embedding vectors with a "time mask" (i.e. feed the sequence of event lagging times into a RNN with sigmoid activation und multiply element-wise. The idea is that the sigmoid activation will put a weight on the event based on the time passed, e.g. an event many months ago will be unimportant and therefore receive an activation close to 0). See section 3.1. in this paper
That paper also suggests another solution where you can learn a joint embedding of time and event (section 3.2).
Note that both methods report only minor (but consistent) gains over standard RNNs.

[2] Given that your number of event categories is low you could also just one-hot encode the event or use a tree-based method like gradient boosting and feed the events, the time between events and your remaining features as separate inputs into the model. Unless your dataset is very huge this will perform equally well if not better because you don't have to learn embeddings and have a lot less model parameters.

The only difficulty is that you will now have to fix the sequence length. But you can pad shorter sequences with N/A or 0 values and I would not expect any performance decrease because of it!


Predicting equipment failure with time series alarm data

I think what you are describing would be called anomaly detection. I suggest trying a different approach. There are several standard solutions to deal with this topic, here are a few.
Main problems to address are:

Setting a good threshold to balance false alarms with missed
events. Selecting the model will influence this setting (see below for 2 typical models)

Correct Labels/responses: If you have enough examples of alarms that are labelled correctly, you might look at clustering methods to determine the correct labels for un-labelled or mis-labelled examples. This will give you a larger dataset to use for training.

Selecting the model:
Model 1: Manual - Multivariate using the Covariance Matrix
Alarm settings to decrease false alarms:
You can use Euclidean (for spherical distributions), or better yet, Mahalanobis (for ellipsoidal distributions) distance and a n Sigma threshold from the centroid to determine what is the normal reading for the machine and then setting a cutoff for what is normal vs what is an anomaly.
Model 2: Neural Network - Autoencoder
Train on normal data (filter out alarm data) and test on new data looking at the probability distribution in the re-construction error in the model output (high MAE), and selecting a threshold based on the output

Online reference (may need account to access):
https://towardsdatascience.com/how-to-use-machine-learning-for-anomaly-detection-and-condition-monitoring-6742f82900d7


Propensity Modeling for Retail Marketing: Model Adjustments Based on Consumer Life Changes.

If you have the purchase data for all these customers, this could be one way to approach the problem:

You could either cluster the customers into natural groups based on their most recent purchases or find association rules (the likelihood of the customer purchasing in a particular category based on their most recent purchases).

I'm not sure how well this approach is going to work for you, but it worked for me in my scenario of propensity scoring. So, if it makes sense, try to relate your data to mine.

I had a list of customers and their purchase behaviors. From those purchase behaviors I deduced association rules to determine which customer is likely to purchase in what product category (based on their previous purchases- associated with the previous purchases of the group).

You can also include recency and frequency of the product purchases into the model to decide whether or not to recommend a particular product/offer to the customer for a specific time.

Based on the confidence, support and lift metrics of those rules for each customer, I mapped them to highly likely to purchase in this category with confidence being their propensity score.

Let me know if this works with your data.


How to apply ensemble clustering method?

You might find the following libraries helpful:

Python: Cluster Ensembles
R: diceR


Is hyperparameter tuning more affected by the input data, or by the task?

In other words, is hyperparameter tuning more affected by the task (which is constant) or by the input data?

It's correct that the task is constant, but hyper-parameters are usually considered specific to a particular learning algorithm, or to a method in general. In a broad sense the method may include what type of algorithm, its hyper-parameters, which features are used (in your case which embeddings), etc.

The performance depends on both the data and the method (in a broad sense), and since hyper-parameters are parts of the method, there's no guarantee that the optimal parameters stay the same when any part of the method is changed even if the data doesn't change.

So for optimal results it's better to tune the hyper-parameters for every possible pair of ML model and words embeddings. You can confirm this experimentally: it's very likely that the selected hyper-parameters will be different when you change any part of the method.


How to start analysing and modelling data for an academic project, when not a statistician or data scientist

Typically, quantitative analysis is planned and performed, based on research study's goals. Focusing on research goals and corresponding research questions, researcher would propose a model (or several models) and a set of hypotheses, associated with the model(s). Model(s) and its/their elements' types usually dictate (suggest) quantitative approaches that would make sense in a particular situation. For example, if your model includes latent variables, you would have to use appropriate methods to perform data analysis (i.e., structural equation modeling). Otherwise, you can apply a variety of other methods, such as time series analysis or, as you mentioned, multiple regression and machine learning. For more details on research workflow with latent variables, also see section #3 in my relevant answer.

One last note: whatever methods you use, pay enough attention to the following two very important aspects - performing full-scale exploratory data analysis (EDA) (see my relevant answer) and trying to design and perform your analysis in the reproducible research fashion (see my relevant answer).


Random Forest but keep only leaves with impurities below a threshold

If some leaves are pruned then it means that the model cannot predict the instances which would normally have fallen into these leaves. The logic of a decision tree is to represent every possible instance: like any supervised learning method, it must be able to predict for every possible instance as input. In some cases where all the leaves have low purity, this condition would even make the model unable to predict anything at all. This is why there is no such algorithm, at least not as a regular supervised learning algorithm.
However you can rely on the probability predicted for the instance instead, this gives you exactly the same information in a way consistent with standard ML.


Basically, it's called an error because you're making the wrong decision. Like for instance, let's say your null hypothesis is something like: The patient does not have cancer. And the Alternative: The patient does have cancer. A type 2 error in this regard would be like saying "Oh, this patient does not have cancer" while they actually do have cancer. We are FAILING to REJECT a FALSE null. So I guess the most simple way to explain this is, we aren't catching a mistake, and letting it slide by. Of course this isn't all there is to it, but it should help kind of show what a type 2 error is and why it's something you want to avoid.


Two parallel models for semantic segmentation in Keras

This is probably laye but incase someone else is stuck, check out this link.

Basically, your model expects two inputs as defined here:

model = Model([input1, input2], output_layer)

Therefore you will need to pass a list of two inputs with the same shape as you defined here:

input1 = Input(shape=(480,480,3))
input2 = Input(shape=(480,480,1))


R : Counting the number of observations per category

You can also do this with the dplyr package. The dplyr package has the functions group_by to group your data by one or more variables and summarise to do some aggregation function. The dplyr package also supports the 'pipe' notation %>%. This notation means the output of the previous function is the first argument of the next function. Here is what it might look like for one of your variables. The dplyr package is also nice in that its function do not require column names to be quoted or in character vectors.
library(dplyr)
my_summary_data <- mydata %>%
    group_by(Replicate) %>%
    summarise(Count = n())
# The last line creates a new column named Count with a value calculated by n(),
# which counts observations (rows) per group.

The output looks something like:
my_summary_data
Replicate  Count
1          8
2          7

The group_by function can group by multiple columns, so
my_summary_data <- mydata %>%
    group_by(Replicate, Node) %>%
    summarise(Count = n())

will produce:
Replicate  Node  Count
1          1     5
1          2     3
2          1     7


Gene innovation numbers in NEAT implementatoins

The innovation number is generated by a single, unique counter for all genomes. The purpose of the innovation number is to have an ID that indicates corresponding genes in two different genomes.

For example, a gene may originate in a genome due to mutation in generation 1. In future generations this gene may have been passed on to other genomes through crossover. So now we have multiple genomes with our gene in it. If two genomes with this gene are selected for crossover with each other, we want to know that this gene is identical in both genomes. We can tell that these genes are identical, because they have matching innovation numbers. If each genome had their own counter, then each genome would have its own unrelated gene for each innovation number. This tells us nothing about the nature of our network and negates the reason we added innovation number.

As an aside, here is a more thorough outline of NEAT straight from the author of the paper


How does layer normalization work exactly?

To understand how it works, we can have a look at the paper related to the question: "Layer Normalization", J. Lei, J. Kiros and G. Hinton.

Question: layer normalization normalizes across all the features for fully connected layers

Not exactly. What layer normalization does is to compute the normalization of the term $a_i^l$ of each neuron $i$ of the layer $l$ within the layer (and not across all the features or activations of the fully connected layers).
This term $a_i^l$  is given by the weighted sum of the activations of the previous layers:
$$ a_i^l = (w_i^l)^T\, h^l$$
Where $h^l$ is the vector of activations from the previous layer and $w_i^l$ is the vector of weights that connects the activations of the previous layer with the neuron $i$.
This normalization is computed within the layer because is done by using the mean ($\mu^l$) and standard deviation ($\sigma^l$) of the values of all the neurons in the same layer, $l$ for the current sample:
$$ \mu^l = \frac{1}{H}\sum_{i=1}^Ha_i^l\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\sigma^l=\sqrt{\frac{1}{H}\sum_{i=1}^H(a_i^l-\mu^l)^2}$$
Where $H$ is the number of neurons in the layer $l$. Hence, the normalization of the $a^l_i$ terms is given by:
$$\bar{a}_i^l = \frac{g_i^l}{\sigma^l}(a_i^l -\mu^l)$$
Where $g^i_l$ is defined by the authors as a scale (or gain) value for each neuron $i$ of the layer $l$ which has to be learnt by the NN.
Once we have obtained the normalized term $\bar{a}_i^l$, we compute the activation of each neuron, $h_i^{l+1}$ by:
$$ h_i^{l+1} = f(\bar{a}_i^l + b_i^l)$$
Where $b_i^l$ is the bias term of the neuron, and $f(\cdot)$ is the activation function.

Question: Does that mean that for each batch dimension we have to learn the normalization parameters?

Yes, that's true. For each sample we would have to recompute the terms $\mu^l$ and $\sigma^l$ in order to do the normalization of the $a^l_i$ terms.

Question: What do we do at test time if our batch dimension is different?

At test time we have to compute only the predictions (forward propagation). We aren't learning the weights, biases and the $g_i$'s terms, we are testing them. However, because the activations on each layer have to be normalized we have to compute again the terms $\mu^l$ and $\sigma^l$ for each layer and sample (irrespective of the mini-batch size).


How to calculate KL-divergence between matrices

I think you can. Just normalize both of the vectors to be sure they are distributions. Then you can apply the kl divergence .
Note the following:
- you need to use a very small value when calculating the kl-d to avoid division by zero. In other words , replace any zero value with ver small value
- kl-d is not a metric . Kl(AB) does not equal KL(BA) . If you are interested in it as a metric you have to use the symmetric kl = (Kl(AB) +KL(BA) )/2


The simple way here is to make a one-hot over letters. So you will have many columns with ones and zeros. This may now be an input layer for your nn.


Just some quick fixes.
Normalization takes care of scale of data across the columns. e.g.
If one data column is in the ranges of 1000s and other in 10s then normalization will work.

But if you want to take care of the skew-ness one way is to simply take log of the data. It pulls in the outliers. as log increases very slowly for positive values. Do it before the normalization (Z score in your case)

Other ways are taking nth root of the data.


Positively skewed target label in regression

Something that might work is normalizing/standardizing the output, e.g. with target over [0,1] (see min-max scaling). Changing the distribution isn't ideal, and I can't see how changing the mean would lead to increased performance. If it makes sense, you can also potentially experiment with changing from a strict continuous linear regression to a categorical interpretation (see ordinal regression). Classification is generally easier than regression, so if you're able to frame it in that way sometimes it helps performance, especially if the end goal in mind is, for example, only to determine a binary decision (buy/not buy).

For example, for income, regressing over strict amount will lead to imbalance, and the difference between 100 billion net worth and 500 billion net worth is negligible to the mean. If you can accept classifying might be useful, you can use logistic regression, SVM etc and experiment with generating a classification instead of a direct prediction.


Plots of data using DBSCAN algorithm not making sense

Dont try to visually confirm it.

You are plotting your clustering resutls in ONLY two dimensions and you expect that all of the information is in these two dimesnions. That is very unlikely. If you plot 3 dimensions you will see even  more seperability and it will make a bit more sense. In any case you need a metric for example Silhouette that tells you how well you clustered. Visualisation is just a sanity check if you know your features already.


Classification loss function: how to implement individual weights for each observation and class

What you are referring to is called a weighted loss function.
In Keras:

Define a dictionary with your labels and their associated weights or just a list of the weights (by class order):

loss_weights = {0: 0.5,
                 1: 0.2,
                 2: 1.5}...
Feed the it to the compile method:

model.compile(optimizer=opt,
               loss='categorical_crossentropy',
               metrics='acc',
               loss_weights=loss_weights)

The loss value that will be minimized by the model will then be the weighted sum of all individual losses, weighted by the loss_weights coefficients.


How to resolve "invalid number of 'breaks'"

It looks like your data isn't actually a 2-dimensional frame, just a 1-dimensional series of strings.  (Hence the error when you try to select out an income column.)  So casting to numeric doesn't make sense.

You'd want to split rather than just replacing periods, but also your data strings aren't period-separated; it's almost space-separated, but you also have data strings that have internal spaces ("Homeowner free and clear"), so you can't just split on spaces.  You'd have to be more clever in splitting up the strings into the columnar data you want.  Or, hopefully, your data is more cleanly set up elsewhere.


F1 score vs accuracy, which metric is more important?

Well it is highly dependent on your use cases and how your data is distributed.

Let me breakdown the pros and cons in practice :

F1-Score

Pros :

Takes into account how the data is distributed. Useful when you have data with imbalance classes.

Cons :

Less interpretable. Precision and recall is more interpretable than
f1-score, since it measures the type-1 error and type-2 error. However, f1-score measures the trade-off between this two.
When positive class is minority class, the score is quite sensitive when there is switching where the ground truth is positive.

Accuracy

Pros :

Easy to understand.

Cons :

It does not take into account how the data is distributed (example case below). This error could be crucial and might lead to incorrect conclusion.

Here is an example depicting cons of accuracy. Without loss of generality consider a binary classification task. Imagine having a data which of 100 samples with 90 negative sample and 10 positive sample. Suppose you have a classifier that predicts all negative. You will have an accuracy of 90%, but let's consider the f1 score, you will actually get 0 because your recall (which is a component of f1 score) is 0.

In practice, for multi-class classification model (which is your use-cases) accuracy is mostly favored. f1 is usually used for multi-label or binary label where the classes are highly unbalanced.


How to optimize hyperparameters in stacked model?

I believe the most common way involves some slight data leakage during the training step that is often ignored. The "correct" way involves giving up more training data but many have empirically realized that giving up more training data is often worse.

Split your data into training and testing.
Split the training set into k-folds.
Train your base models using all of the k folds. Save the predictions from the best (optimized in terms of hyperparameters) base models, for each of the k folds.

Example: suppose we use 3 fold cross validation and have two base models. For each base model, we find the best hyperparameters that on average optimize some loss function over these three folds. We then save the predictions made by this optimal base model over these same three folds (i.e. we save the exact same predictions that you must have made when scoring your model over all three folds).

In essence, you are transforming your original training set with your base models in pieces and reconstructing it back together.

Using your reconstructed training set of predictions from folds 1, 2 and 3, use k fold cross validation again to train your combiner (i.e. find optimal hyperparameters again but for the combiner). You can choose to use the same splits as you did in step three but it doesn't matter. This is where the problem lies with data leakage. No matter how you split your data here you will have variables in your test set that were directly created from observations in your training set. For example, suppose fold 1 and fold 2 in the previous example make up the training set, and therefore fold 3 is used for validation. Since both fold 1 and fold 2 were created using fold 3 observations there is a chance that you overfit here due to optimism in your validation scores. In Kaggle competitions this is often ignored. My guess is probably because if you are stacking you have a large dataset, and if you have a large dataset minor data leaks become less of an issue (in general).
Fit your base models using your entire training dataset, using the hyperparameters found in step 3. Predict the training set to generate the required meta features for your combiner, and also predict the test set created in step 1.
Fit your combiner using the training set generated in the previous step, using the hyperparameters found in step 4.
Predict the test set that you had created from your base models in step 5, using the combiner model from the prior step. Score your loss and report this score as your models performance.


Is it meaningful to use word2vec for non-string inputs like time series analysis?

The goal of Word2vec is to represent each element of a sequence into an "embedded space", i.e. a lower dimensional space where "similar" elements are located closer to each other.

I think this is a bit off topic for time series analysis. If you want to detect outliers, you might try looking at the normalized distance of each datapoint from the trend, and set a threshold.

Alternatively, if you really want to stick with Neural Networks, you can use Autoencoders. They can be applied to outlier detection, but it's not the simplest model to implement.


Why K-harmonic means is less sensitive to random initialization than K-means?

Harmonic mean is less sensitive to outliers.

Say that you have 3 numbers: 1 1 and 30001

Their arithmetic mean (average) is 10001 and their harmonic mean is 2 that is much closer to the majority of the points.

So in a way, using an harmonic mean is more forgiving.

A Bayesian look at the harmonic mean

Assuming that the closer a point is, the more likely it is to be in a cluster.

One can choose to model the distribution of distances from an origin point, as an Gamma-Exponential model with a priors $\alpha=m$ and $\beta=\frac{m}{\mu_0}$ where $\mu_0$ is the initial mean and $m$ is the strength of the prior.

And given observations of $\{d_1,d_2,\dots,d_n\}$, the updated model's mean is:
$$\mu'=\frac{m+n}{\frac{m}{\mu_0}+\frac{1}{d_1}+\frac{1}{d_2}+\dots+\frac{1}{d_n}}$$
We got our beloved harmonic mean.

You could argue that the harmonic mean in k-means context is an assumption on the distribution of distances.


How do I implement Dual-encoder model in Pytorch?

There isnt any as of yet, but scout this website and as soon as it comes it will be there.


Is it OK to use the testing sample to compare algorithms?

Basically, every time you use the results of a train/test split to make decisions about a model- whether that's tuning the hyperparameters of a single model, or choosing the most effective of a number of different models, you cannot infer anything about the performance of the model after making those decisions until you have "frozen" your model and evaluated it on a portion of data that has not been touched.

The general concept addressing this issue is called nested cross validation. If you use a train/test split to choose the best parameters for a model, that's fine. But if you want to estimate the performance of that, you need to then evaluate on a second held out set.

If you then repeat process for multiple models and choose the best performing one, again, that's fine, but by choosing the best result the value of your performance metric is inherently biased, and you need to validate the entire procedure on yet another held out set to get an unbiased estimate of how your model will perform on unseen data.


Big difference in randomTree accuracy with train and test sets

Overfitting is a common problem in random forest, you can use cross Validation methods (Cross validation is a technique to build models that are not prone to overfitting) example - K-fold cross validation, stratified k fold..

Try this using train package-
control <- trainControl(method = "cv", number = 5)
and add this as a control parameter to the model.


How to use Cohen's Kappa as the evaluation metric in GridSearchCV in Scikit Learn?

Cohen's kappa was introduced in scikit-learn 0.17.

You can wrap it in make_scorer for use in GridSearchCV.

from sklearn.metrics import cohen_kappa_score, make_scorer
from sklearn.grid_search import GridSearchCV
from sklearn.svm import LinearSVC

kappa_scorer = make_scorer(cohen_kappa_score)
grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=kappa_scorer)


Relating changes of a value in time to known events

You seem to be looking for a clustering mechanism for your event sequence data based on a dissimilarity/ distance measure.
I would recommend you to explore TraMineR package.


Why does my neural net always converge on a trivial solution?

The problem turned out to be that the input had not been properly scaled - making sure all epochs had standard deviation = 1 solved it.


I have few basic questions on R Stem and Leaf Plot

With the default scale, you see that numbers left of the bar go up by two - hence anything after the 4 is forty-something or fifty-something:

> stem(d,scale=1)

  The decimal point is 1 digit(s) to the right of the |

  0 | 577
  2 | 7
  4 | 19
  6 | 0325
  8 | 50378

Using scale=2, you'll see numbers left of the bar go up by one, so now you can get exact reconstruction of your input, since you input integers:

> stem(d,scale=2)

  The decimal point is 1 digit(s) to the right of the |

  0 | 57
  1 | 7
  2 |
  3 | 7
  4 | 1
  5 | 9
  6 | 03
  7 | 25
  8 | 5
  9 | 0378

Going further you can even split it by first and second five within each decade:

> stem(d,scale=4)

  The decimal point is 1 digit(s) to the right of the |

  0 | 57
  1 |
  1 | 7
  2 |
  2 |
  3 |
  3 | 7
  4 | 1
  4 |
  5 |
  5 | 9
  6 | 03
  6 |
  7 | 2
  7 | 5
  8 |
  8 | 5
  9 | 03
  9 | 78

Stem plots do not always guarantee you can reproduce the data by reversing the process, and that's not what they're for.


Imputing missing value based on filtering result of another column

You can use pandas groupby and transform methods. This maps the groupwise mode to the index of the original dataframe.
df['C2']=df.groupby('C1').transform(lambda x: x.fillna(x.value_counts().index[0]))['C2']
df

The output is:


You should probably do some reading in the field of "Natural Language Generation", since this seems to relate most directly to your question.

But the way you have described the process -- "text mining...text building" -- leads me to wonder if you are aiming for something much more ambitious.  It seems as though you aim to automate the process of 1) reading natural language texts, 2) understanding the meaning, and then 3) generate new texts based on that semantic knowledge.  I'm not aware of any general-purpose end-to-end systems that can do that, not even specialized systems by the likes of Palantir.  What you are aiming for would probably pass the Turing Test for fully capable Artificial Intelligence.


A correlation matrix would help discover any pair-wise correlation (between any 2 variables). I think that would be a great place to start, as it is just one line of code for initial analysis.

After that, if you want to investigate "an increase in parameter 1 and parameter 2 is correlated with a fall in parameter 10", you could write code to form interaction variables like sum / product / divide and build a correlation matrix between the interaction variables vs. the original variables.

Personally, I found the corrplot R package useful in visualizing the correlation matrix. Hope this helps.


Classification task - how to handle large data file?

With pandas:

import pandas as pd

dataset = pd.read_csv(path, chunksize=1000) #number of rows per chunk

for chunk in dataset:
    #process chunk etc


Can we use package "dplyr" on R base 3.0.2?

dplyr 0.3 requires R 3.1+. If you're stuck on R 3.0.x, you have to use dplyr 0.2.x.


Find points on a map close to given points

I can tell you how I would do it, but there is almost certainly a faster implementation.

Assuming you start with, for each point in $L$, the distances to each warehouse, $w \in W$. These distances should be calculated by the haversine formula.

You can find the distance to the $N$th closest point in $w$ by using the quickselect algorithm. This is very similar to the quicksort algorithm but only sorting the parts that you care about. The average case for quickselect is $O(N)$ but you'll need to repeat for each $l \in L$.

Note that, since the square is monotonic for positive distances, you only need to minimise

$$\sum_{l \in L} maximum_{w \in W}(distance(l, w)) $$

I found a handy implementation of the quickselect algorithm on KoderDojo


intuition behind the difference between likelihood function of discriminative and generative algorithms

Discriminative model learn to classify an x into class y: the conditional probability distribution p(y|x).

Generative models learn the joint probability distribution p(x,y).It can be transformed into p(y|x) by a Bayes rule.

You get different likelihood functions because the way those models classify data is different.


How can I generate reasonable dummy/artificial data from a pre-existing time-series data?

This problem is referred to as scenario generation, commonly used in stochastic optimization. The key problem is that you have to respect correlations among the different variables, which means that you need to model the joint multivariate predictive distribution. One way to bypass this is to use Copulas, which only require modelling the marginal probability densities. An approach widely used in engineering is the following:

Model the marginal (empirical) probability distribution of each variable separately. You can also model the conditional probability distribution using probabilistic forecasting if needed.
Estimate the empirical correlation matrix between the time series.
Use the empirical correlation matrix and the marginal conditional distributions to fit a multivariate Copula.
Obtain correlated scenarios by sampling from the multivariate Copula.

I am including a standard reference from the field of wind forecasting, which models the temporal correlation from the same time series. This can be applied to the case of different time series (spatial correlation).


Machine learning to predict pollution at a single location

Not enough karma to simply comment, so I have to log this as an answer.
Does it make sense? Sure! But keep in mind that if you train your model on a single location, it can only really be trusted to be used on future data from that same location. Could your model generalize to other locations perhaps? Maybe, but your model didn't see anything so you can't be sure. Same thing applies to having only 2 months worth of data. You can't be sure if this model will be accurate when it's scoring data that comes from a month it hasn't seen before.
If you really want to work on this type of problem and just don't have the data for it - this link has a similar dataset that might interest you.


Why does only Adam optimiser perform well in my case?

There could be a couple of possible reasons:

One reason could be the Adam optimizer is a combination of several other
optimization techniques (e.g., momentum and running average of
gradient squares). The combination of those techniques works well on
multi-label text classification.
Another reason could be that multi-label text classification is a sparse
problem. The Adam optimizer in TensorFlow has a sparse
implementation.


Calculating optimal path(s) to increase probability of a classification

This is a multivariate optimisation problem. You have a function f(X) that returns an output that you want to maximise - Maximising the probability of belonging to class. That function f(X) is just your model.

You also mention "constraints", and again this is standard constrained optimisation territory.

The problem itself may be hard to solve, but the framework seems to be that.

Have a look at scipy optimise here


Can overfitting occur in Advanced Optimization algorithms?

There is no technique that will eliminate the risk of overfitting entirely. The methods you've listed are all just different ways of fitting a linear model. A linear model will have a global minimum, and that minimum shouldn't change regardless of the flavor of gradient descent that you're using (unless you're using regularization), so all of the methods you've listed would overfit (or underfit) equally.

Moving from linear models to more complex models, like deep learning, you're even more at risk of seeing overfitting. I've had plenty of convoluted neural networks that badly overfit, even though convolution is supposed to reduce the chance of overfitting substantially by sharing weights. In summary, there is no silver bullet for overfitting, regardless of model family or optimization technique.


Predict how many days late or early someone will finish their work

If we assume that each task delivery is independent of eachother, and the process does not change a lot over time (stationary), we can treat this as a standard regression problem.

Since this is about deadlines, we expect that there might be variations over time, or patterns of delay across the seasons of the year or week. So time-based features might look something like:

|deadline_year|deadline_week_number|deadline_day_of_week|

We also expect that the size of a delay might depend on the size of the task. So if you have the start date, or an estimate on number of days, definitely include that. If people can have multiple tasks at the same time, include that also.

|workdays_between_start_and_deadline|workdays_estimated|concurrent_tasks|

And we expect that delays may depend on the person who performs the task, and who created the task.

|task_owner|task_creator|

Use Exploratory Data Analysis and your knowledge about the processes that created to find more of these possible relationships. Use scatterplots of each feature against the target days_delayed (negative=before time, 0=on time).

One can start with a strong non-linear model like RandomForest. This can give estimates which can be scored (by mean squared error for example), and indicate whether your features are predictive or not.
To get probability intervals, you can use a Bayesian model such as Bayesian Ridge Regression. This is a linear model, so may have to spend more time on feature engineering to make the relationships between feature and target (roughly) linear.


Does an RL agent learn during exploitation?

It depends on the game the agent is playing. If there are rewards all over the environment, the agent learns only when the coefficient of exploration is greater than zero. That is, if you are only allowing it to exploit, the agent may aswell just exploit the first reward it meets which ends a game.

In this case, it will find the first reward that ends the game, and will not change its algorithm (will not learn any other way of playing). On the other hand, if you allow it to explore, it may eventually find another better strategy (learn).

It is always good to conciliate the ratio of exploration and exploitation. It should always be capable of exploring, even if the coefficient is low. That is the whole advantage of Reinforcement Learning.


How to create user and item profile in an item to item collaborative filtering? (Non-rating case)

You can use your item and user profiles to generate a prediciton function (i.e. a function that will predict how relevant a coupon will be for a user, also known as a representation for your items and users). Typical functions used for this purpose are Dot Product and Cosine Similarity. This step ensures the intelligence in the recommendations will incorporate categories information.

Once your scores are predicted, you must compare them to your historical interactions by using a loss function (some examples are Root-mean-square-error and Kullback-Leibler divergence). This step will produce an error that you can use to inform your algorithm for learning (e.g. adjust the weights in your representation functions for items and users).

For more details I strongly suggest you to check out the great slides compiled by James Kirk, which provide a framework to unify the research that has been done on the topic.


Explaining instances of a machine learning regressor

Lime introduced support for regression problems in their latest version. A notebook with examples is available here.


Why you shouldn't upsample before cross validation

To see clearly why the procedure of upsampling before CV is mistaken and it leads to data leakage and other undesired consequences, it is useful to imagine first the simpler "baseline" case, where we simply upsample (i.e. create duplicate samples) without SMOTE.
The first reason why such a procedure is invalid is that, this way, some of the duplicates due to upsampling will end up both to the training and the validation splits (CV folds); the result being that the algorithm is validated with some samples that have already been seen during training, which invalidates the very fundamental requirement of a validation set (fold) and it is actually the very definition of data leakage. For more details, see own answer in the SO thread Process for oversampling data for imbalanced binary classification; quoting from there:

I once witnessed a case where the modeller was struggling to understand why he was getting a ~ 100% test accuracy, much higher than his training one; turned out his initial dataset was full of duplicates -no class imbalance here, but the idea is similar- and several of these duplicates naturally ended up in his test set after the split, without of course being new or unseen data...

But there is also a second reason: this procedure shows biased performance measures in our validation folds that are no longer representative of reality: remember, we want our validation folds to be representative of the real unseen data, which of course will be imbalanced. Performing CV after upsampling results also to artificially balancing our validation folds; doing so, and claiming that we get X% accuracy when a great part of this accuracy will be due to the artificially upsampled minority class makes no sense, and gives misleading impressions. For details, see own answer in the SO thread Balance classes in cross validation. Notice that the author of the post you have linked to says (rather cryptically, and only in a parenthesis):

(we are smart enough not to oversample the test data)

For more corroboration, here is Max Kuhn, creator of the caret R package and co-author of the (highly recommended) Applied Predictive Modelling textbook, in Chapter 11: Subsampling For Class Imbalances of the caret ebook:

You would never want to artificially balance the test set; its class frequencies should be in-line with what one would see “in the wild”.

Now, it's true that the above hold for the case of balancing through simple upsampling of the minority class; but SMOTE does not do that - it uses interpolation to create synthetic samples that are "close" enough to the real minority ones. How does this change the situation?
Not much.

The second reason stated above (biased performance measures in the validation folds) is still fully applicable - in fact, it holds independently of the exact nature of the upsampling (duplicate samples or synthetic ones).

Given that the synthetic samples generated by SMOTE are indeed highly correlated with the real ones, the problems due to the first reason mentioned above are still largely present, although somewhat ameliorated.

In contrast, the pipeline approach does not suffer from these issues, because it first splits into training and validation folds, and applies SMOTE subsequently only to the training ones.


First, you are not dumb. We are all learning here.

Second, I have looked at your code and I can see the problem closer now.

The code below works, but it doesn't give anything useful (nor does it lead to any speedup). I'll try to explain why.

library(iterators)
library(foreach)
library(doParallel)
library(mfe)

data = iris

# foreach
split = detectCores()
eachStart = 25

# set up iterators
iters = iter(rep(eachStart, split))

# set up combine function
cl = makeCluster(split)
registerDoParallel(cl)

result = foreach(nstart=iters, .packages = c("iterators", "mfe")) %dopar% {

  metafeatures(Species ~ ., iris, groups=c("general", "statistical", "model.based"))

}

stopCluster(cl)

This will return a list of length equal to the number of cores on your machine, but all of the elements in the list will be exactly the same. You are getting no speedup (in fact, your code will run slower if you use this) because all you are doing is running the exact same piece of code n = (number of cores on your machine) number of times, namely this:

metafeatures(Species ~ ., iris, groups=c("general", "statistical", "model.based"))

You aren't iterating over anything in your foreach loop to give any benefit to running in parallel. The overall idea of running stuff in parallel (in this context, anyway) is to run the exact same piece of code (i.e. a function) on each worker node, but with different inputs to the function's arguments in each call. In the context of a for loop, you can think of this as a single iteration of the for loop being sent to a single worker (splits, as you defined them in your code) on your computer. The benefit of course is that if you have many cores on your computer, different iterations of your for loop can be computed at the exact same time (vs. on a single core machine, you would have to work on a single iteration of the for loop one at a time). It is important to note that this all implies that each iteration in your for loop is independent of all other iterations and that the order in which you compute iterations does not matter (i.e., your algorithm is NOT sequential).

Here is one example that I can think of where you could consider running in parallel: good old k-fold cross validation. Each resulting training set receives the exact same pre processing but all you are changing is what folds make up the training and test sets.

This could be written like this:

library(foreach)
library(doParallel)
library(caret)
library(e1071)
library(tidyverse)

data = iris

cl = makeCluster(detectCores())

#Generate three folds from our dataset

folds = createFolds(y = data$Species, k = 3)

#Train a support vector machine in parallel using foreach. Returns the predictions from the fitted model.

results = foreach(i = 1:length(folds), .packages = c("caret", "e1071", "tidyverse")) %dopar% {

  training = data[-folds[[i]], ]
  test = data[folds[[i]],]

  #Pre processing, if desired

  #Train the support vector machine. Parameters are just for example, ripped from a webpage that just so happened to be using the exact same dataset.
  model = svm(Species~., data=training,
          method="C-classification", kernal="radial",
          gamma=0.1, cost=10)

  tibble(Row = folds[[i]], Predicted.Class = predict(model, newdata = test))

}

#Bind all the data frames in the list "results"

processedResults = bind_rows(results) %>%
  arrange(-Row)

Notice how we are iterating over different training/test splits (with these lines):

training = data[-folds[[i]], ]
test = data[folds[[i]],]

but the function itself remains the same. Each core will receive a different train/test set in each iteration of the for loop. In your code, you aren't changing arguments to your function and so there is no point in running your code in parallel.

Now, if I understand mfe correctly, it is generating some summary statistics from your dataset. This could maybe run in parallel (maybe some statistics can be computed in parallel? I doubt there is any speed to be gained here), but I highly doubt it is much faster due to the overhead associated with running stuff in parallel (and hence, why the package creator did not choose to implement such a feature).

You mentioned random forest and how you were able to run it in parallel easily. This is because a random forest is easily made parallel (it is "embarrassingly parallel"). Each individual tree can be fit on a single worker (because each tree is fit independently of all other trees in the forest on a bootstrapped training set). The order in which the trees are fit doesn't matter either, because we just average over all of the trees predictions to get our final predictions.

My two cents: running in parallel is only useful if the algorithm itself can be run in parallel. You can't take advantage of all of your cores if the algorithm itself cannot be "divided" and then "combined". And even then, if the algorithm is trivial to run on a single thread then the computational overhead associated with running stuff in parallel makes it generally worse in these cases.


Yes, it sometimes does that. I think it depends on the data source. For RDBMS it is likely that it takes the data type from the database (so the column type is typically accurate). For text/csv, it probably does a scan of the top certain lines to guestimate the type. It is similar to the text import wizard of the SQL, etc. other similar tool.  (A Tableau user of 4+ years)
I am not sure if it is relevant to many of the Tableau users; many databases don't have that distinction either; I think it is more helpful for someone working in machine learning, etc.?


An ambiguity in SVM equations about misclassified data

Good point! Interesting consequence!

Problem is the $a_n=0$ assumption, i.e. assuming misclassified points are not support vectors.

Here is the flow. Slack variable $\xi_n$ is defined as
$$\xi_n := |t_n - y(\boldsymbol{x}_n)|$$
where $t_n \in \{+1, -1\}$ is the true label, and $y(\boldsymbol{x}_n)$ is the prediction. Therefore, for a misclassified point (on the wrong side) we have $$\xi_n > 1$$ by definition. Given $\mu_n \xi_n = 0$, therefore$$\mu_n=0$$
and given $a_n=C - \mu_n$, therefore $$a_n = C > 0$$ which means (given $a_n > 0$ only for support vectors)

  Every misclassified  point is a support vector.

This is a nice consequence and should have been stated in the book.

Although, a remotely! related point has been stated in the book:

  Points with $a_n = C$ can lie inside the margin and can either be
  correctly classified if $\xi_n \leq 1$ or misclassified if $\xi_n > 1$.


For those not familiar, item-item recommenders calculate similarities between items, as opposed to user-user (or user-based) recommenders, which calculate similarities between users. Although some algorithms can be used for both, this question is in regard to item-item algorithms (thanks for being specific in your question).

Accuracy or effectiveness of recommenders is evaluated based on comparing recommendations to a previously collected data set (training set). For example, I have shopping cart data from the last six months; I'll use the first 5 months as training data, then run my various algorithms, and compare the quality against what really happened during the 6th month.

The reason Mahout ships with so many algorithms is because different algorithms are more or less effective in each data set you may work with. So, ideally, you do some testing as I described with many algorithms and compare the accuracy, then choose the winner.

Interestingly, you can also take other factors into account, such as the need to minimize the data set (for performance reasons), and run your tests only with a certain portion of the training data available. In such a case, one algorithm may work better with the smaller data set, but another may work with the complete set. Then, you get to weigh performance VS accuracy VS challenge of implementation (such as deploying on a Hadoop cluster).

Therefore, different algorithms are suited for different project. However, there are some general rules:

All algorithms always do better with unreduced data sets (more data is better).
More complex algorithms aren't necessarily better.

I suggest starting with a simple algorithm and ensuring you have high quality data. If you have additional time, you can implement more complex algorithms and create a comparison which is unique to your data set.

Most of my info comes from This study. You'll find lots of detail about implementation there.


In principle you can use your approach.

However, you should not optimize on your test set (step 3). Instead you should select the best t using your validation set.
Then you compare it against KNN, also on the validation set. Finally, the best model should be evaluated on the test set.


Is there any optimal way on feature selection for more than one classification algorithms?

If you want to know what the feature importance of a dataset is, you can obtain it by training a random forest. After training the random forest, you can access the feature importance which is valid for any algorithm.

Note that the feature importance of other similar algorithms, such as some boosting algorithms, is strictly related to those algorithms; such thing is not true for random forest.

Hope this helps!

EDIT1:

The reason I said random forests give a sort of universal feature importance is that a rf is based on a lot of smaller decision trees in which each one uses bootstrap from the train set and a subset of attributes taken randomly. The bootstrap tries to avoid overfitting, whereas the subset of attributes helps determine which ones are the most important. A rf is capable of giving the importance of each feature averaging the oob precision from the trees that use the attribute. When a feature  is a great predictor, the trees that use it have better results that those that don't use it. With hundreds or thousands of trees, a rf can have a very good opinion on the predicting capacity of each attribute.

Note that each tree in a rf is a decision tree which bases the split on Gini impurity/information gain (entropy), or variance in the case of regression. This naturally selects the features that matter most in each case.

Here, an article which explores alternative ways of studying feature importance. Some more things are said about rf, but it's not limited to that method, so it may be useful for anyone reading this post.
https://towardsdatascience.com/explaining-feature-importance-by-example-of-a-random-forest-d9166011959e


For text processing, try using Python and the NLTK package.

For removing surplus symbols, you can use regular expressions. Install package 're' and use the built-in function re.sub to substitute symbols like '-' with empty character.

For suffix stripping, you can either use regular expressions again or use the built-in word stemming functionality in the NLTK package. This tutorial should help.


Your problem is a classification problem with 3 classes. The appropriate function for it is softmax function, whereas you are using Relu. Another thing is that you need a cost function. The function that you optimize and the one you do calculate derivatives of with respect to parameters.

The cost function and last layer activation function must match each other. So first you choose appropriate "last" activation funtion for your problem and then cost function.

For iris you can use softmax function with cross entropy function. The derivatives of these you can easily find. :)

Another thing are the dimensions. The result of softmax function is the vector of the size (number of classes, 1) for each observation whereas you have (1,3) dimensions.

Regarding the backpropagation algorithm for the other layers it is looks ok, but the last layer equation is wrong and should be like the one below:

where C is the cost function and we calculate derivative of C with respect to a (activation of last layer) and multiply element-wise by derivative of a (here it should be softmax function with respect to z). dw and db for last layers have correct equation.

I hope this hints will help you come up with final neural network implementation for iris classification problem.


One hot encoding of target variable containing classes 1 to 9 not including zero

I don't know it this is an optimal approach. The natural way to solve a SUDOKU with computer science is Linear Programming. I am curious if a CNN will solve the problem.
Can you add +1 or -1 in the post processing / pre processing of the predictions? It will be the easier way to solve it.


ValueError trying to use a pickled scikit-learn model

You have to use the same CountVectorizer instance on all data and have a method to handle out of training sample tokens.


How to correctly set a target for a time series based model?

You've got an interesting, if over analysed question. I would just say your question itself limits your potential profit. You want to give your computer more choice in this matter. the whole price_good, price_bad approach is price_bad. The computer should decide EVERYTHING in this game and you will want to build in a consideration of previous stock history, share earnings, maybe even public sentiment from twitter, etc.

This really seems like the optimal scenario to implement some variant of q-learning. You'll need a lot of data and the stock market is notorious for being difficult for the layman to derive any meaning from.

What is Q-learning?

Q learning is a deep learning method with allows a computer to find the ideal set of actions to maximize some final score. The system is composed of 3 parts: a deep learning network that predicts the target, a q table that contains the state at a given time and a possible action, and the change in the final score.

Read more here: https://towardsdatascience.com/simple-reinforcement-learning-q-learning-fcddc4b6fe56

Really, read that before continuing :)

Step 1:

You create the q-table (a table of many possible actions). In this case your state will be a vector of the number of shares you hold, the liquid funds you have for buying, and n number of consecutive prices for a stock (You'll also need the next price of the stock for a later step). You will then make a choice: how much of the stock do you want to sell? this number can be any real integer. a negative number indicates selling, a positive number indicates buying and a 0 indicates holding.

EDIT: I want to explain a bit more what's going on with the "n number of consecutive prices for a stock". This is where your "timeseries" bit comes into play. It is a vector that looks at a frame of prices at any given time, but it not necessarily the latest frame. It's just a way for you to train on what would have happened in the past if you were making that trade. For example if I have a vector of prices that looks like all_prices below, the frames would look like:

 all_prices = [12, 13, 14, 13, 14, 15, 16, 17, 18]
 frame1 = [12, 13, 14, 13]
 frame2 = [13, 14, 13, 14]
 frame3 = [14, 13, 14, 15]
 frame4 = [13, 14, 15, 16]

For each state you'll have to add rows to the table of every possible action (everything between sell all and buy with as much cash I have). Then for each row you'll calculate the reward for that action. Now, the reward is calculated based on the next price of the stock. It can be negative (bad) or positive (good).

So here's what that table would look like for a single stock:

 *n_shares* | *cash* | *price_at_t=1* | *price_at_t=2* | .... | *last_reward* | *action*
 50         | 1000   | 13.41          | 12.45          | .... | 0             | -10
 50         | 1000   | 13.41          | 12.45          | .... | 0             | -5
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 0
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 5
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 10

Step 2:

You'll train some deep learning network on each row (state) to predict the reward. yay!

Step 3:

When you actually want to make a decision on your data, you create a table of all possible states and actions and have your nueral network predict the reward (now it's unknown). You can select the action which maximizes long term reward.

Some caveats

Now the above really doesn't look at all that much data. You can add in soooo much more and potentially create a better model. I've never done this, but I don't have particularly high hopes for something like this. I'm definitely open to reconsider though. Perhaps if you look at this with each timepoint being the close price, and each day you get one change to sell, buy, or hold, it might be feasible. Otherwise, you'll need an API that allows you to trade (super cool, but I haven't heard of something like this for consumers).

Are there better solutions?
Yes, this is just a simple reinforcement learning solution. We can do better, but this seemed like a simple starting point to me.

Am I liable if you loose all your moneeeys?
Nope.

Is the above normal q-learning?
Not really. I took some shortcuts and glossed over some aspects of it. I do hope that it wasn't too complex and that it helped you understand how to better approach the problem.

Will you become a billionaire after this?
....maybe. I hope so. If it starts working, tell me and I'll invest with you.


Methods to reduce dimensionality within a feature?

You could try clustering the data with the carrier feature removed, add it back in, and look at the distributions of the carrier feature per cluster.


Are RNN or LSTM appropriate Neural Networks approaches for multivariate time-series regression?

Here is a really good source to begin multivariate time-series forecasting in Keras using LSTMs.

  I aim to train my model/regression on an interval $[t_0,t_n]$ and then be able to apply it on a larger different interval.

There's no harm in this as long as you perform the right kind of multi-step forecasting. If your problem requires you to train on $[t_0, t_n], \text{for some } n < 100$ and produce $y_1(t_{n+1}), y_2(t_{n+1})$ as outputs then APPLY it on $[t_n, t_n+100]$ then there will be issues in implementation, as most ML models will require you to provide the same shape as you did when you were training your model. Here's when sliding across time-series will help you. Simply put,

$$
\begin{align}
\text{Train on: }& [t_0, t_n] &\text{ Output: }& y_1(t_{n+1}), y_2(t_{n+1}) \\
\text{Predict on: }& [t_1, t_{n+1}] &\text{ Output: }& y_1(t_{n+2}), y_2(t_{n+2})\\
\text{Predict on: }& [t_2, t_{n+2}] &\text{ Output: }& y_1(t_{n+3}), y_2(t_{n+3})\\
& \vdots & & \vdots\\
\end{align}
$$

  I know that relation between my Y and X are non-linear, but also that in need to take into account lag or inertia. For example, Y1(t) is dependent of X1([t−dt1,t]) and X2([t−dt2,t]). All dtn are different, but I have an approximate idea of how 'far' I need to reach.

Well, you can model your time-series data as $X_1(t-k),\cdots,X_1(t), X_2(t-d),\cdots,X_2(t)$ as $X$ (input) and $X_1(t+1), X_2(t+1)$ as $y$ (target), where $k$ and $d$ refer to different lags for each variable.


What is the main difference between Hadoop and Spark?

Hadoop is a framework for the distributed storage and processing of big data on the Hadoop File System (HDFS) where data is stored in a cluster of "nodes" and can be set up to be fault tolerant. Since data is stored accross multiple nodes it can be processed in parallel, and Hadoop uses the MapReduce algorithm for doing so. This is basically achieved by each node in the cluster fetching the data it needs from disk, performing the neecessary computations which are then aggregated and returned.
Spark is a distributed processing framework for big data, but does not provide storage. Consequently it needs to work on top of distributed storage, which could be Hadoop. Spark is designed as an in-memory engine, and is therefore much faster than MapReduce on Hadoop. It is also fault tolerant. Spark includes a version of SQL which allows for much better querying of the underlying data compared to Hadoop/MapReduce. Since in practice, Spark is often still limited by memory availability when the underlying data is too large, so there is still some disk-bound operations, but at least in theory it is capable of 100% in-memory real time analytics. This is not possible with Hadoop.
Both Hadoop and Spark can work with columnar data, although Spark is columnar by default, and in Hadoop, the underlying data is often row-oriented, such as csv files (but can support columnar data such as parquet and ORC). Data stored in columnar format is much faster to query.
Hadoop is designed for large scale batch processing. Spark is also designed for batch processing but can also handle streaming and other data flows. Both are scalable technologies, but Hadoop scales nearly linearly, whereas with Spark, although it will generally be faster than Hadoop for similar sized data, there are limitations based on the memory available in the cluster, above which performance will deteriorate much faster than with Hadoop. That is, with Hadoop speed will decrease approximately linearly as the data size increases. With Spark, once the memory in the cluster is exceeded, relative performance will decline faster than with Hadoop.
Due to the memory needed in a Spark cluster, it will generally cost more to operate than Hadoop in terms of uptime. However for smaller datasets the much faster in-memory processing may completely offset the higher cost of compute time.
Some of the above are purposely general statements, there could be edge cases and other situations where they don't apply.


Is it possible to predict sentiment of unlabelled dataset using BERT?

Bert is a pre-trained language model with objectives like masked token prediction and next sentence prediction. So, it doesn't have any setup to do sentiment analysis,but you can use the pre-trained information to do the same.
There are couple of ways to solve your problem, they are

Annotate few of your documents and fine-tune the model for sentiment analysis.
Get open-source sentiment analysis datasets which fit your requirement, train Bert on the data and use the same classifier for your purpose (But this will work only if both data distributions look the same).


But when repeat same process with same values it generate different boundary values. Can anyone explain about it?

This is because you're not setting the random_seed in the in the train_test_split, which means that the training data is shuffled in a different way on each run.

With a quick check using one of sklearn's datasets, you can check that this is the issue:

from sklearn.datasets import load_boston

X,y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state=4)
tree_model = DecisionTreeClassifier(max_depth=2, random_state=2)
tree_model.fit(X_train, y_train)
y_pred = tree_model.predict_proba(X_train)[:,1]
X_train_df = pd.DataFrame(X_train, columns = ['sepal_len', 'sepal_wid',
                                              'petal_len', 'petal_wid'])
X_train_df['Age_tree'] = tree_model.predict_proba(X_train)[:,1]
X_train_df.Age_tree.unique()

This will produce the same boundaries in all runs, in this case array([0.        , 0.90697674, 0.03030303]). Whereas if you don't set the random seed you'll get different probabilities and boundaries on each run.


error in running exe file after changing .py to .exe

A quick google search of the error message reveals two issues describing the issue and potential solutions.
One of the issues is in the pyinstaller repo, and suggests adding the cv2 path to the pyinstaller invocation:
pyinstaller --onefile sample.py --paths "/root/opencv-4.3.0/release/lib/python3/cv2"

You can get the actual path by running a simple python program:
import cv2
print(cv2.__file__)

The other issue is in the opencv repo and may apply especially if you compiled opencv yourself.
UPDATE: with respect to the error ImportError: DLL load failed while importing cv2, according to the FAQ, you should install the Visual Studio C++ redistributable


Autoregressive (AR) models constants - Time Series Analysis

You have a slight typo in the notation of a AR(1) model. The correct signature is,

$y_t = \beta_0 + \beta_1 \times y_{t-1}+\epsilon_t$ or $y(t) = \beta_0 + \beta_1 \times y(t-1) + \epsilon(t)$,

where $y(t)$ and $\epsilon(t)$ are  random variables. If $y(t)$ is standard Gaussian you can estimate $\beta_{0,1}$ with a maximum likelihood estimator (MLE). If not you will need a more complex method. You can read more about it in this article on estimating at ARMA Process.


How to set weight in Weighted Kernel K-Means?

These weights should be introduced by a user. With a weight you tell the K-means algorithm, that one feature is more important than the other.

  [0] These might represent a measure of importance, a frequency count, or some other information. The intent is that a point with a weight of 5.0 is twice as "important" as a point with a weight of 2.5, for instance. This gives rise to the "weighted" K-Means problem.

[0] http://people.sc.fsu.edu/~jburkardt/m_src/kmeans/kmeans.html


Don't compress files that are already compressed (like how JPEG/PNG images or video files are). Their inherent compression is usually good enough, and you only trade some 5% in lower compressed size for a much lengthier and often non-seekable decompression that results in using twice the disk space.

If you need to batch the files together, just use tar.


Merging dataframes in Pandas is taking a surprisingly long time

There are a couple of points I can maybe give to help.

Firstly, Pandas is not great at merging multiple large dataframes in general because every time you merge a new dataframe to an old one, it makes a copy of both to make a third dataframe - this obviously starts taking a lot of time as your master dataframe grows in each step.

Secondly (untested), you could try to overcome any memory bottlenecks in the merging steps by trying something dumb, like merging two or three files, then writing it to disk. Repeat that until you have 30 larger CSV files, then do it again so you have, say 10 files, and so on. I am not sure at which point the python garbage collection would kick in with you single reduce function, so it could even be the case that you start using SWAP memory, which would make things really slow! This solution could at least prevent that from happening. You should perform each repeat in a new Python session.

One other idea, as you are reading the data from CSV files: you could use a terminal tool to simply paste all the CSV files into one file, then read that into pandas once. Then perform required clean up actions. The cleaning might be a pain (given your merge on requirements, but you would likely overcome the long waiting times for pandas to make the copies. If you files are all in one folder, you can run the following in terminal to put them all in one file

cat *.csv > merged.csv

There are other methods here, in case you need to leave out the header line etc.

Lastly I would recommend having a look at the data.table package, which is now functional in Python - it started in the R world. It has a much more robust memory management system (built on ideas from databases) and is able to store huge amounts of data in a single "DataTable", in cases where Pandas would crash. It is a great package to fill the gap between Pandas and Dask (both awesome packages!).

Here is a video, showing how Python data.table works and also shows some benchmarks.


K-fold-cross-validation if training dataset is much smaller than test dataset?

Generally speaking, when evaluating a model you either choose to do cross-validation or train-test splitting, but not both.  Your dataset appears to already be split between training and testing sets, so you seem to have implicitly chosen train-test splitting.

In your particular case, it may still make sense to run cross-validation if you have a reason to mistrust the results from the testing set.  Is the distribution of labels in the test data balanced or highly skewed?  Is the test data representative of examples you're likely to see in production?  If your test data is representative, then you'll likely get a better accuracy estimate evaluating on the test set rather than running cross-validation.

But I'm confused.  How did you come to have a training set of 500 images and testing set of 10,000 images?  If you 10,500 labeled examples available to you, then you can divide them however you like - right?  Why not run cross-validation with the entire set of 10,500 images?  That will give you the most robust accuracy estimate.


Detect if word is «common English» word or slang word

It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain you want and you do a simple lookup. Or you don't know in advance and you need to compute these common words from your documents (your short phrases). In that case, the more sentences you have the better.

An easy way to do this is using pure Python is to use a counter

from collections import Counter

documents = [] # here add your list of documents/phrases

counter = Counter()

for doc in documents:

    words = doc.split() # assuming that words can be split on whitespaces

    counter.update(words)

counter.most_common() # this will return words ranked by their frequency

Then it's up to you to apply a threshold to define what words are common and which aren't. A more advanced approach could be using their TFIDF weights but in that case, you are not necessarily keeping common words, you are keeping "important" ones.


Time series forecasting using multiple time series as training data

First  cluster the events that have the most similarities. Then use a comparable (or more than one of  then ) to forecast the sales of the new events that you do not have historical data. Use all other information you have as regressor. Here is a code to do the forecast in R. You will be able to combine different forecasting models with this code:

choose_model<-function(x,h,reg,new_reg,end_train,start_test){
      library(forecast)
      library(forecastHybrid)
      library(tidyverse)

  #train data

  x_train <- window(x, end = end_train )

  x_test <- window(x, start = start_test)

  #train and test for regressors

  reg_train <- window(reg, end = end_train )

  reg_test <- window(reg, start = start_test)

  h1=length(x_test)

  #model1

  stlf(x_train , method="arima",s.window= nrow(x_train),xreg = reg_train, newxreg = reg_test, h=h1)-> fc_stlf_xreg

  #model2
  auto.arima(x_train, stepwise = FALSE, approximation = FALSE,xreg=reg_train)%>%forecast(h=h1,xreg=reg_test) -> fc_arima_xreg

  #model3
  set.seed(12345)#for nnetar model
  nnetar(x_train, MaxNWts=nrow(x), xreg=reg_train)%>%forecast(h=h1, xreg=reg_test) -> fc_nnetar_xreg

  #model4
  stlf(x_train , method= "ets",s.window= 12, h=h1)-> fc_stlf_ets

  #Combination

  mod1 <- lm(x_test ~ 0 + fc_stlf_xreg$mean + fc_arima_xreg$mean + fc_nnetar_xreg$mean + fc_stlf_ets$mean)
  mod2 <- lm(x_test/I(sum(coef(mod1))) ~ 0 + fc_stlf_xreg$mean + fc_arima_xreg$mean + fc_nnetar_xreg$mean + fc_stlf_ets$mean)

  #model1

  stlf(x, method="arima",s.window= 12,xreg=reg, newxreg=new_reg, h=h)-> fc_stlf

  #model2
  auto.arima(x, stepwise = FALSE, approximation = FALSE,xreg=reg)%>%forecast(h=h,xreg=new_reg) -> fc_arima

  #model3
  set.seed(12345)#for nnetar model
  nnetar(x, MaxNWts=nrow(x), xreg=reg)%>%forecast(h=h, xreg=new_reg) -> fc_nnetar

  #model4
  stlf(x , method= "ets",s.window= 12, h=h)-> fc_stlf_e

  #Combination

  Combi <- (mod2$coefficients[[1]]*fc_stlf$mean + mod2$coefficients[[2]]*fc_arima$mean +
              mod2$coefficients[[3]]*fc_nnetar$mean + mod2$coefficients[[4]]*fc_stlf_e$mean)

  return(Combi)
}


You may have missing data (e.g. np.nans) in your COMMENTS field. This will throw an error in your email_extract function.
Try to filter the problematic rows out, e.g. with

filtered_data = data[~data.COMMENTS.isna()] #purge problematic comments

and applying the email extraction to the now "clean" column:

filtered_data.COMMENTS.apply(lambda x: email_extract(x))


What is the convergence criteria of a semi-supervised learning algorithm?

Most of the semi-supervised methods are heuristics and more or less are modifications of the standard supervised learning algorithms, where you are trying to take into account unlabeled data considering a small dataset of labeled data. If the data from these two datasets do not follow the same distribution, then you have to look for transfer learning methods.

I guess that you are asking for a stopping criterion for iterative methods/approaches of semi-supervised learning, otherwise you do not need a stopping criterion. It depends on the approach. You could define a threshold that reflects the confidence that the unlabeled data could be labeled correctly and use them as pseudo-labeled instances with the ones that are already labeled for training an accurate classifier. Another option could be to define a maximum number of iterations.


Steps for back propagation of convolutional layer in CNN

Extra Note: The matrix notation that I use is the following: [number of filters - depth - rows - columns]. For example, when I say [3 filters - 1 x 3 x 3] I mean that I have 3 filters with depth 1, 3 rows and 3 columns (depth is "equivalent" to "channels").

Steps in Conv layer:

Imagine I have an input X with shape [depth:1, rows: 28, columns: 28]

Then, I have the conv-layer, where I have 3 filters (you can call the filter as W) with shape [depth:1, rows:3, columns:3]

So, what I do is, using 1 filter a time, convolve input X , so the resulting matrix has the following shape => [depth:1, rows: 26, columns: 26] (this is the result of convolving 1 filter). So, when I do the convolution using the 3 filters, the resulting matrix is shaped [depth:3, rows: 26, columns: 26].

After this, I apply ReLU to each value of the resulting matrix (the 3x26x26 matrix), and then apply non-overlapping maxpooling with a window o 2 rows x 2 columns.

If there is any doubt of what I explained, please comment me.

Note: I'm editing my own answer beacuse I don't have 50 reputation for doing comments.


In R, can I integrate different classifying algorithms in one bagging model?

What you're looking for is called an ensemble model which means it is a compilation of several models to improve the results. This is a very common technique for winners in Kaggle competitions. Since you're using R and caret is a popular way to do ML in R, here's a package just for that purpose on caret:

https://cran.r-project.org/web/packages/caretEnsemble/vignettes/caretEnsemble-intro.html


How can I count the number of matching zero elements between two numpy arrays?

You can compare the predictions with the expected results directly, using simple comparisons, in this case just ==. This returns boolean values - True or False, which you can sum up because True == 1 and False = 0.
Here is an example for your case using some randomly generated dummy data:
In [1]: import numpy as np

In [2]: y = np.random.choice([-1, 0, 1], 10)

In [3]: preds = np.random.choice([-1, 0, 1], 10)

In [4]: y
Out[4]: array([ 1,  1,  1, -1,  1, -1, -1,  1,  1,  0])

In [5]: preds
Out[5]: array([ 0, -1,  1,  0,  1,  1, -1,  1, -1,  0])

The real part that checks where your predictions are correct is then done using two checks:

where are the predictions equal to the ground truth i.e. where were you correct in predicting the direction, and
where are the predictions equal to the direction you are interested in i.e. downwards (-1), no change (0) or upwards (1)

This can be done as follows:
In [6]: (y == preds) & (preds == 0)   #   &   means we needs both checks to be True
Out[6]:
array([False, False, False, False, False, False, False, False, False, True])

We can see only the final position is True after both these checks, because that is the only place that the prediction was True and the value direction was 0.
You can then write a loop to check all values and do something with them:
In [7]: n = len(preds)    # the number of test samples (= 10 in my dummy example)

In [8]: for direction in [-1, 0, 1]:
            score = sum((y == preds) & (preds == 0))
            accuracy = score / n
            print(f"Direction {direction:>2}: {score}/{n} = {accuracy * 100:.1f}%")

Which gives:
Direction -1: 1/10 = 10.0%
Direction  0: 1/10 = 10.0%
Direction  1: 1/10 = 10.0%


How is the target variable passed to the final estimator in this pipeline?

The Pipeline object performs a .fit_transform(X, y) sequentially based on how you structure your pipeline until just before the last step (the estimator). The estimator does not perform a transform because it's purpose is not to transform your input array X, but to make a prediction. The fit_transform method performs a fit, then a transform on that pipeline step when it is called. In your case that means:

Your standard scalar is fit based on X and y
Your standard scalar transforms your input data X
Your LinearSVC is fit based on your previously transformed X and your un-transformed target y

Since your LinearSVC is the last step of the pipeline object and doesn't have a transform method the process stops. Your Pipeline object that contains your StandardScalar and the LinearSVC now contains a fit version of both these objects. Now you can perform a .predict() method with this pipeline which will cause the fit StandardScalar to perform a .transform() on input X, then this transformed X will be passed to the final estimator LinearSVC for a .predict() which will return your predicted y array.


Find changes in variables into two states

Before doing anything more advanced, I would check the correlation of new value / old value pairs. If there seems to be a linear relationship, or some kind of correlation, that would be easy to solve. If that is not the case and the "new" value for each column depends on every other column in a nontrivial way, you may have to use more complicated methods.

Such a problem is called as a Multivariate Regression Analysis, and if you have a large enough dataset, you can try using using a neural network. I would have the old values for each variable as the inputs, and the new ones as outputs. A fully connected neural network - with size depending on the size of your dataset - could be a good first step. But to be honest, I don't have much experience with such models and don't know any tutorials available online. Another way might be to train separate regression models for each output. In such a setting, you can look for correlations with different inputs and perhaps eliminate any useless ones.

By the way, what are the last two columns title NA? If they carry information, I would add them to the input layer.


Train a GAN on "before and after" images of dental surgeries

It's a very specific problem and there's no right or wrong solution.  I'll just write what I'd do in your position and hope that it is useful.

  How many "before and after" images will I need?

You'll need a lot of images to consistently get good results, in the range of tens or preferably hundreds of thousands.

  What architecture should I use? i.e. StyleGAN, VAE?

StyleGAN seems to be the most popular option for these sorts of tasks these days, so I'd stick with this.

  As I will need my custom model; training a fresh model would need a good GPU. What platform should I go for? Colab, Floyhub, or something else

The platform you'll choose depends on whether or not you're willing to pay. Colab, Floydhub and Kaggle are the only free options with GPU afaik, but also offer paid versions (1, 2). All have their pros and cons. For example colab, which I'm more familiar with, offers only 12-hour sessions (after that it disconnects you and you have to reconnect).

  Approach

Regarding the process you said, that's an interesting approach. I would have gone with something more basic like a segmentation mask, but I think your idea is better.


Best clustering algorithm to identify clusters and determine the closet cluster each individual response is near?

Since they are categorical variables, I would cluster them using the k-medoids clustering method. Before applying this method, one-hot encode all the predictors.

See a tutorial here:
https://towardsdatascience.com/k-medoids-clustering-on-iris-data-set-1931bf781e05

Sklearn has an implementation:
https://scikit-learn-extra.readthedocs.io/en/latest/generated/sklearn_extra.cluster.KMedoids.html


Does the Bishop book imply that a neuron feeds to itself in chapter 5.3?

The equations are only working for a given layer.

If you want to generalize, you need to rewrite them as, for example :
$$a^l_j=\sum\limits_{i}w^l_{ji}z^{l-1}_i + b^l_j$$


Why is the optimal C chosen by GridSearchCV so small?

Have a look at svc_cv.cv_results_: there are many values of C that tied for best, with accuracy 99.6%, and the chosen C is the smallest of those.  The point is that the width of the margin doesn't affect the actual hyperplane very much, and so the accuracy score doesn't change very much.
A few suggestions:

With larger and less-separable data sets, this might get minimized, since small changes in the hyperplane would be more likely to classify points differently.

For this case, where many scores are exactly equal, you might prefer to tie-break for larger values of C.  This can be easily accomplished by just reversing the order of the list c_space, or more robustly by defining a custom scorer for the search, that takes the mean test score plus some function of C.

Something more refined than accuracy as the scorer for the search could help separate the values for different C, but it's not clear what would be best.  Something like log-loss would require calibrating probabilities; maybe AUROC?


Problem with chain rule in softmax layer when differentiated separately

I'm just gonna answer my own question here so that in the future I can find it easier, and hopefully it comes handy for others asking about the derivative of Softmax and gets confused by the fact that it's a matrix. So the shown matrix form for $\frac{\partial{Softmax(x)}}{\partial{x}} $ in the opening question is incorrect. The correct formula should be:
$$ \frac{\partial{Softmax(Z^{O})}}{\partial{Z^{O}}} := \begin{pmatrix}\frac{\partial{Softmax(Z^{O}_{0})}}{\partial{Z^{O}_{0}}}&\frac{\partial{Softmax(Z^{O}_{0})}}{\partial{Z^{O}_{1}}}&\frac{\partial{Softmax(Z^{O}_{0})}}{\partial{Z^{O}_{2}}}&...&\frac{\partial{Softmax(Z^{O}_{0})}}{\partial{Z^{O}_{N}}}\\\frac{\partial{Softmax(Z^{O}_{1})}}{\partial{Z^{O}_{0}}}&\frac{\partial{Softmax(Z^{O}_{1})}}{\partial{Z^{O}_{1}}}&\frac{\partial{Softmax(Z^{O}_{1})}}{\partial{Z^{O}_{2}}}&...&\frac{\partial{Softmax(Z^{O}_{1})}}{\partial{Z^{O}_{N}}}\\\frac{\partial{Softmax(Z^{O}_{2})}}{\partial{Z^{O}_{0}}}&\frac{\partial{Softmax(Z^{O}_{2})}}{\partial{Z^{O}_{1}}}&\frac{\partial{Softmax(Z^{O}_{2})}}{\partial{Z^{O}_{2}}}&...&\frac{\partial{Softmax(Z^{O}_{2})}}{\partial{Z^{O}_{N}}}\\...&...&...&&...\\\frac{\partial{Softmax(Z^{O}_{N})}}{\partial{Z^{O}_{0}}}&\frac{\partial{Softmax(Z^{O}_{N})}}{\partial{Z^{O}_{1}}}&\frac{\partial{Softmax(Z^{O}_{N})}}{\partial{Z^{O}_{2}}}&...&\frac{\partial{Softmax(Z^{O}_{N})}}{\partial{Z^{O}_{N}}}\end{pmatrix}$$
Which is:
$$ \begin{pmatrix}S(Z^{O}_{0})(1-S(Z^{O}_{0}))&-S(Z^{O}_{0})S(Z^{O}_{1})&-S(Z^{O}_{0})S(Z^{O}_{2})&...&-S(Z^{O}_{0})S(Z^{O}_{N})\\-S(Z^{O}_{1})S(Z^{O}_{0})&S(Z^{O}_{1})(1-S(Z^{O}_{1}))&-S(Z^{1}_{1})S(Z^{O}_{2})&...&-S(Z^{O}_{1})S(Z^{O}_{N})\\-S(Z^{O}_{2})S(Z^{O}_{0})&-S(Z^{O}_{2})S(Z^{O}_{1})&S(Z^{2}_{2})(1-S(Z^{2}_{2}))&...&-S(Z^{O}_{2})S(Z^{O}_{N})\\...&...&...&&...\\-S(Z^{O}_{N})S(Z^{O}_{0})&-S(Z^{O}_{N})S(Z^{O}_{1})&-S(Z^{O}_{N})S(Z^{O}_{2})&...&S(Z^{O}_{N})(1-S(Z^{O}_{N}))\end{pmatrix} $$
For which I incorrectly used the df2 function with z1 argument, as it should be:
$$ \begin{pmatrix}A^O_0(1-A^O_0)&-A^O_0A^O_1\\-A^O_0A^O_1&A^O_1(1-A^O_1)\end{pmatrix} $$
Using this, I have:
$$ \frac{\partial{A^{O}}}{\partial{Z^{O}}} = Softmax^{'}(A^{O}) = \begin{pmatrix}.25&-.25\\-.25&.25\end{pmatrix} $$
Finally doing the multiplication:
$$ \frac{\partial{E(A^{O}, T)}}{\partial{A^{O}}} * \frac{\partial{A^{O}}}{\partial{Z^{O}}} = \begin{pmatrix}0&-2\end{pmatrix}*\begin{pmatrix}0.25&-0.25\\-0.25&0.25\end{pmatrix} = \begin{pmatrix}0.5&-0.5\end{pmatrix} $$
Which matches the original:
$$ \frac{\partial{E(A^{O}, T)}}{\partial{Z^{O}}} = A^{O} - T = \begin{pmatrix}0.5&0.5\end{pmatrix} - \begin{pmatrix}0&1\end{pmatrix} = \begin{pmatrix}0.5&-0.5\end{pmatrix} $$
And this is where I messed up.


What is the bleu score of professional human translators?

The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this:

  The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even
  a human translator will not necessarily score 1. It
  is important to note that the more reference translations per sentence there are, the higher the score is. Thus, one must be cautious making even “rough”
  comparisons on evaluations with different numbers
  of reference translations: on a test corpus of about
  500 sentences (40 general news stories), a human
  translator scored 0.3468 against four references and
  scored 0.2571 against two references.

But as their table 1 (providing the numbers compared to two references, H2 is the one mentioned in the text above) shows there is variance among human BLEU scores:

Unfortunately, the paper does not qualify the skill level of the translators.


Text extraction from large pool of documents of different formats

One option is to use Apache SOLR + Apache TIKA.
Apache TIKA has support for most common file formats, it extracts test content from files. Extracted text can be stored in SOLR.  SOLR supports various kinds of text + aggregation queries.
Tutorials :
https://blog.webnersolutions.com/apache-solr-indexing-all-files-in-a-folder-recursively
https://lucene.apache.org/solr/guide/6_6/introduction-to-solr-indexing.html
https://lucene.apache.org/solr/guide/6_6/uploading-data-with-solr-cell-using-apache-tika.html


Time complexity of function minimizers for neural networks

The first paragraph in this tutorial, explains it pretty well.

Also the time complexity will not grow linearly if you increase the size of your network. It will grow squared and so will the number of parameters.


Key pixels, key "features" detection in CNNs

Class Activation Map is what I was looking for


In feature selection, I came across a situation where NaN were filled by median of the column values

There is no rule that only mean or median should be used. Based on the situation sometimes mean is better and sometimes median. In fact there are occasions when mode would be better.
These are not the only techniques to fill NaNs. There are several other imputation methods. If you are starting out, then an excellent hands-on training material would be the Titanic data set which contains a number of NaNs in the 'Age' feature. You can try your hands at finding out which is the best way to impute missing data there. You can refer to: https://www.kaggle.com/c/titanic/discussion/157929 - Missing Ages on the Titanic - Few perspectives from basic to the advanced for some of the advanced strategies (specific to the Titanic scenario)


neural networks error function: is global minimum desirable?

The figure describing momentum is a bit misleading because it only considers a very simple case. Usually momentum will help you not to get stuck too early in a local minimum, so you end up in a better but still local minimum.

Momentum can help you converge faster, but there are no guarantees to end up in a global minimum. In popular use cases of neural networks like image classification etc. the error function will be extremely complicated, so it will always be infeasible to reach a global minimum using gradient based methods. But practice has shown that driving down the error to a local minimum still leads to good results.

Also, momentum is compatible with other forms of regularization. You can apply momentum to any regularized cost function. The statement in Elements of Statistical Learning is more of general nature, meaning that even if you could achieve the global minimum it wouldn't be desirable because it is most likely not the solution that will give you the best generalization (even though it has the best training(!) error).


There is no problem with the fact that your regularization loss is going up.

The cost function of your model is a weighted sum of the regularization loss and the base loss, so during training the model looks to minimize them together, but eventually it comes to a point where it has to choose and minimize one on the expense of the other. The fact that it is prefers the base loss means that your regularization hyper-parameter is not too big (eventually your goal is to reduce the base loss).

In general terms: The goal of the regularization loss is to simplify the model during training - demanding smaller weights forces the model to increase the size of just the important weights, and not all of them. So some wights converge to very small and insignificant values, effectively reducing the number of weights in the model. Smaller model means less overfitting and more generalization.

In your case: During the training of your model, it starts from reducing the regularization loss (because it is an easy task), but at some point it starts to notice crucial features which it wants to emphasize so it begins to increase the weights of those features, which results in higher regularization loss but a smaller total loss (a good thing, it is learning but is also under control).


What are the common properties of all machine learning algorithms?

Machine Learning is enabling the machine to learn by its mistakes. As the machine is not human you need to find a way to tell it what it is doing wrong and a way to reduce its errors. Typically machine learning involves predictions of something (Numbers, Classes or whatever) All the fields you mentioned involve machine learning and of course statistics. Language is transformed into numeric to be processed by learning algorithm which is then able to predict a language from a text or keywords.
Hope been clear.


In a neural network with tensor input X, it seems there are times when it will never learn... Why?

In short

You are waaaaay undertraining. Increase the number of times you show the network your data. I am guessing training may take longer than you expect because typically networks train best with 0-mean data, which yours is not.
ReLU seems to cause problems with such a shallow network. Try increasing depth or using elu activation instead.
I confirmed having an activation in the last layer doesn't cause huge problems, but it is still a good idea of getting in the habit of knowing when you should and should not have an activation in the last layer.

Undertraining

I handled this by increasing the number of training examples by 10,000 times (you could increase the number of epochs instead but this results in better printing):

x_train=np.array([[1,2], [3,4], [5,6]] * 10000)
y_train=np.array([f(x) for x in x_train])[:, np.newaxis]

Problems with ReLU

The problems with ReLU can be handled in one of two ways, increase the number of layers when using ReLU, or use a different activation such as elu. Both trained just fine for me:

model = km.Sequential()
model.add(kl.Dense(units=3, activation='relu', input_shape=(in_dim,)))
model.add(kl.Dense(units=3, activation='relu'))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

or

model = km.Sequential()
model.add(kl.Dense(units=3, activation='elu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

Full working code

Below shows the code with elu, you can swap the block out for the ReLU version (shown above) instead and it prints very similar values.

import numpy as np
import keras.models as km
import keras.layers as kl
import keras.optimizers as ko
import keras.losses as kloss

# This will cause no learning
np.random.seed(1692585618)

def f(x):
    a = x[0]* 3.141 + x[1]
    return a;

# Create a sample dataset
# Input is (*, 2)
# Output is (*, )
x_train=np.array([[1,2], [3,4], [5,6]] * 10000)
y_train=np.array([f(x) for x in x_train])[:, np.newaxis]

# These are required by the shape of x_train and y_train
in_dim = x_train.shape[1]
out_dim = 1

model = km.Sequential()
model.add(kl.Dense(units=3, activation='elu', input_shape=(in_dim,)))
model.add(kl.Dense(units=out_dim))
model.compile(
  loss=kloss.mean_squared_error, optimizer=ko.Adam(lr=0.1)
)

model.fit(x_train, y_train, epochs=3, verbose=True)
print('predicted: {}'.format(model.predict(x_train)[:3, 0]))
print('actual   : {}'.format(y_train[:3, 0]))

prints

Epoch 1/3
30000/30000 [==============================] - 1s 29us/step - loss: 3.0553
Epoch 2/3
30000/30000 [==============================] - 1s 20us/step - loss: 5.0199e-06
Epoch 3/3
30000/30000 [==============================] - 1s 20us/step - loss: 5.4414e-06
predicted: [ 5.1426897 13.420667  21.707573 ]
actual   : [ 5.141 13.423 21.705]


What applications does linear programming have in data science?

Whenever you have an optimization problem the first question that you have to ask yourself is.

Can I make it a Linear Programming problem?

For python I normally use Gurobi. Here is a basic example to get started: https://www.gurobi.com/resources/food-manufacture-i/

You can also do Machine Learning with it, that is the hot topic nowadays, but if you can make it a linear programming problem you will achieve optimality.


Predicting contract churn/cancellation: Great model results does not work in the real world

If your model makes a prediction 6 months into the future, then it doesn't make sense to judge its performance before 6 months. If only 2 months have passed, then possibly 2/3 of the true positives have yet to reveal their true nature and you are arriving a premature conclusion.

To test this theory, I would train a new model to predict 2 months out and use that to get an approximation of live accuracy while your wait 4 more months for the first model. Of course, there could be other problems, but this is what I would try first.