Viterbi-like algorithm suggesting top-N probable state sequences implementation

Apparently, I misunderstood your question.

There are several methods for finding the k-best paths with extending versions of the Viterbi algorithm.

My first advice would be to look at this question on SO that is similar to yours and has a good illustrated answer.

Then, I would refer you to two articles/thesis that are publicly available and from where one can extend his/her research. (Disclaimer: these references may not be the "best" one but I've chosen them because they are publicly available and provide a good number of reference to deepen research on the topic)

The k-best paths in Hidden Markov Models. Algorithms and Applications to Transmembrane Protein Topology Recognition. Thesis by Golod, available here. (a thesis like this gives countless references on the topic)
Decoding HMMs using the k-best paths: algorithms and applications by Brown and Dolog, available here (a partial short version of what you will find in the aforementioned thesis)

My previous answer:

For anyone coming across this question looking for a way of computing some of the typical state sequences of an HMM (as I thought first this question was about), just know that such a concept of most probable sequence without specifying data is not really something used in any theory about HMMs, as far as I know. However, one can follow these steps:

As a first try, I would implement something like this:

Get the state at time $t=0$

Draw an initial state $s_0$ from the initial state probability mass function (pmf)

Get the state at time $t+1$

Draw the new state $s_1$ from the pmf defined by the $s_0$-th row of the transition matrix

Repeat this step as many times as needed to draw states up to $s_N$

Then you can repeat the entire procedure $X$ times in order to get as many sample paths as you wish.

This is very fast and easy to implement and it will give you what you want. Many scientific libraries in many languages have a built-in function for drawing a random sample from a pmf.


Identifying documents similar to specific clusters

First Idea: Go as usual. Calculate the distance according to of each new document to the centroids of 100 clusters and see to which cluster the belong (minimum distance wins).
Second Idea: You already know which clusters are your targets. Assign a label (e.g. 1) to these 10 and and another label (0) to the rest 90. Then train a classifier and try to predict the label for new documents. (The process of labeling in supervised learning is usually based on expert annotation i.e. an acceptable accuracy in labeling. But with this method you lose that accuracy so expect that not all the labels are perfectly fine as a clustering algorithm is blindly do that)


Bayesian regularization vs dropout for basic ann

It actually makes perfect sense to use both. Gal et al. provided a nice theory on how to interpret dropout through a Bayesian lense. In a nutshell, if you use dropout + regularization you are implicitly minimizing the same loss as for a Bayesian Neural Network (BNN), where you learn the posterior distribution over the network weights given the training data.

You can think of the fully Bayesian approach as taking it one step further on the road from MLE over MAP to Bayesian regression, as the former two provide only point estimates, while the latter gives access to the entire posterior distribution.

If you approximate a BNN with dropout, the introduced noise will take the role of drawing samples from the posterior distribution. The regularization will take the role of a prior, which is the same interpretation you would give it in the MAP-context. I don't think, however, that the represented priors are identical.

Either way, the prior helps to regulate the "spread" of the posterior. If you choose a large regularization parameter, which corresponds to a narrow prior distribution, the model will reduce variance in the posterior distribution as well.

If you only need a simple ANN with dropout and regularization, you probably don't care too much about using the approximate posterior after training, but I think this is still a nice perspective on things.


Should I connect my two GPUs with SLI or not? (for Keras + TensorFlow)

You need not connect GPUs via SLI.  Keras and TensorFlow will take care of distributing batches across GPUs

https://keras.io/utils/#multi_gpu_model

Instead of SLI, if you use NV-link, Keras can use use GPU for merge as well.
https://www.nvidia.com/en-us/data-center/nvlink/

From the documentation :

  Specifically, this function implements single-machine multi-GPU data
  parallelism. It works in the following way:

  Divide the model's input(s) into multiple sub-batches. Apply a model
  copy on each sub-batch. Every model copy is executed on a dedicated
  GPU. Concatenate the results (on CPU) into one big batch. E.g. if your
  batch_size is 64 and you use gpus=2, then we will divide the input
  into 2 sub-batches of 32 samples, process each sub-batch on one GPU,
  then return the full batch of 64 processed samples.

  This induces quasi-linear speedup on up to 8 GPUs.

  This function is only available with the TensorFlow backend for the
  time being.


How to use "tree boosting" with a data-driven loss function

First some prior and known aclarations (that you probably already know).
Metric is what we want to optimize.
Optimization Loss is what the model optimizes.
Obviously, we would like the Metric and the optimization loss to be the same, but this is always not possible. How to deal with this?

Run the right model. Some models can optimize different loss functions. In the case of XGBoost you have two loss functions, the one of the decision tree and the one of the boosting.

Preprocess the target and optimize another metric, this will be for example transforming the  target to the logarithmic of the target and then in that space applying a known loss function

Optimize another loss function and metric and then post-process the predictions.

Write your own cost functions. For xgboost we implement a single function that takes predictions and target values and computes the first and second-order derivatives.

Optimize another metric and use early stopping.

The last one almost always works.
In general for complex algorithms Neural Networks tend to work better due to the flexibility of the loss functions (more than in normal ML).


Best Python library for statistical inference

statsmodels is a good, and fairly standard, package to statistics.

For Bayesian interference you can go with PyMC - see as in Cam Davidson-Pilon, Probabilistic Programming & Bayesian Methods for Hackers.


Does it make sense to visualize data with a linear relationship using tSNE?

Your assumption is right, the results are in general misleading.

Suppose your (linearly correlated) data have missing points in some range:

Than for t-SNE the two subset of data will be two different clusters, even if they lie on the same linear distribution:

But, if you are actually interested in the fact that those two structures are separated, then t-SNE is a good choice to visualize it.

So a proper answer should be: it depends on what you need.

P.S.

Here the code used for this example:

import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE

def function(x, y):
    return 4+3*x + 4*y

x1=np.random.rand(100)
x2=np.random.rand(100)+1.2
X=np.concatenate([x1, x2])
Y=0.5+2*X

Z=function(X, Y)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(X, Y, Z, c='r', marker='o')
plt.show()

data=[]
for x, y, z in zip (X, Y, Z):
    data.append([x, y, z])

data_embedded = TSNE(n_components=2).fit_transform(data)

plt.scatter([x for x, y in data_embedded], [y for x, y in data_embedded], color='r')
plt.show()


Does building a corpus make sense on a documentation project?

Anyways is this problem I am trying to solve fit into the general NLP ML space?

Generally speaking, feeding the source data in bulk to a ML system is unlikely to give the kind of structured output you expect. It's likely that you would have to somehow guide the process in the direction of what you want to obtain, and this might take a lot of time and effort (depending on the requirements).

That being said, there are indeed NLP methods which are meant to extract specific pieces of information from text, and it usually works quite well with domain-specific data (provided it's done correctly, caveats apply). I'm just going to list a few typical tasks which can be done:

Named Entity Recognition would be the most common and probably the most simple, since there are many existing libraries. Most libraries use a pre-trained model, but it's likely to give much better results when it's trained on the kind of data it's applied to (of course that usually means manually annotating your own training set).
Text classification can be used to automatically assign documents to a category (class) among a set of predefined categories. This is supervised so you would also need a training set containing labelled documents. Again there is a number of algorithms and libraries available.
Simple information retrieval methods based on measuring semantic similarity (see e.g here) between terms and documents can be used to search e.g. documents relevant to a term.
Topic modeling is an unsupervised approach which groups similar documents together (clustering). Since it's unsupervised it doesn't require any training data, but on the other hand what the algorithm finds as "topics" (groups) is often different than what a human would expect.
Extracting relations between concepts (typically between named entities) is a more advanced task which usually requires more work in order to capture the specifics of the job. I'm not aware of any general library for that.

Overall there are many things possible, but the first step would be to try to design the system precisely, typically using some of the existing tasks as building blocks.

  Is it fine to include a specific document in multiple categories? For example an instance in AWS can be in category ec2 and a general category Computing unit

Yes it's fine, but if you want a ML classifier to be able to predict several classes you will need to use multi-label classification (the "standard" is single label).

  Is the starting point to this is to build a corpus? I tried it out nltk categorized corpus builder.

I'd recommend building a corpus only once you have a clear idea of how you're going to use it. Also it's usually an experimental process with lots of back and forth, so try to progress iteratively rather than starting with strong assumptions/decisions which might later turn out not relevant.


Comparing different classification results with different trainig and test data

Your question is unclear. If you're saying that you have datasets with different attributes then this answer is not relevant.

If you're saying that you've grouped your dataset by some similarity measure into different groups and then split every group into test and train subgroups then yes, you may have overfitting.

However, if you made a TEST dataset from the COMPLETE dataset which has mixed elements from the original groups and proportional to them, and then tested your model against this test dataset then your results should represent your complete dataset properly.


SMOTE-NC does not help to oversample my mixed continuous/categorical dataset

Before going through the process of oversampling, always see if the implementation of your algorithm supports assigning different weights to individual classes. The sklearn RandomForestClassifier has for example a class_weights parameter with which you can do that. I found this method to work better than over- or undersampling.

Also, I have to add an obligatory part: if you minority classes have only very few samples so that the charachteristics of the respective classes are not well captured, there is little you can do except collecting more data.


Is CRF suitable for multi-words Named Entity Recognition?

Absolutely. If you look at the training tutorial, it implies that this isn't an issue at all. When using multi-word entities, you typically need to use a IOB or BILUO tagging schemes, which helps your model train better.
But from a mathematical perspective, there aren't any restrictions for a CRF, as CRF models the likelihood of particular sequences/transitions. Often, people set restrictions for particular transitions if you know in advance that they aren't possible. But by default, all transitions are allowed. In sklearn-crf, allowing all transitions is done by using the all_possible_transitions=True argument.


The simplest is to keep your features separate and add a synthetic feature, feature cross, that captures the relationship between those features you mention that can possibly be nested.

For example, in a neuronal network based classifier (e.g., TensorFlow), the model will learn the 'correct' weight for those combinations of features' values that are impossible to happen (e.g., male and pregnant), excluding wrong data cases obviously.

In the end... you just want the cartesian product among the features that you need to 'cross'. And yes, your vector will grow.


how to train tensorflow chat application on updated dataset everyday

The distribution of your data won't change too much over a smaller period of time, retraining from scratch seems like a waste. Just add the new data to your set and do one or more epochs on all your data, not just the new ones. It can readjust the weights a little bit for changes in the distribution which is exactly what you want. Make sure you are not overreliant on early stopping as regularization method against overfitting because you continuously train like this, use some other regularizer. I would suggest periodically to retrain from scratch but this doesn't have to happen very regularly, maybe once every two months.


Can I use more features for my training data than my test data will supply?

The final purpose of a model is not the training but the actual application of it. If some features are not available during the final application and you cannot derive those features from other features, it is counter-productive to include these features during training.

Many classification methods will not even work if you don't give them the same set of features as they were trained on.


Find the time difference in an appropriate granularity(/ies):

daysDiffFeature = diffInDays(dateCol)
hoursDiffFeature = diffInHours(dateCol)

Then you can use whatever distance based metrics/calculations are required in your algorithsm


.1 after column name while read csv file from pandas

I rechecked and reloaded the data. I manually removed duplicate columns which solved the problem. But yes , it was showing .1 with non duplicate columns as well


Finding the minimum number of spheres likely makes this problem NP-hard, and a variant of the set cover problem. Good luck. It gets particularly difficult if you want to consider balls tha can be inbetween of data points.

There are two obvious clustering algorithms that will find an approximation: Leader and Complete Linkage Agglomerative Hierarchical Clustering. But they will not find the maximum, only an approximation, and probably with no guarantees.

Also, your assumptions are wrong: if you cover the data with balls of radius r, the maximum distance of two points will be 2*r.


When does decision tree perform better than the neural network?

Neural Networks, in my experience have several hyper-parameters (number of layers, neurons per layer, activation functions, optimizers, regularizers, etc.) and are very hard in finding the best configuration for each task. In fact in most cases it's not even worth it trying to find the optimal configuration as other classifiers can outperform Neural Networks with default hyper-parameters. Furthermore, NNs require caution as they are prone to overfitting.

For most tasks where you deal with structured data, I've found tree-based algorithms (especially boosted ones) to outperform NNs.

Some NN architectures are state-of-the-art tasks where we have a lot of unstructured data (e.g. CNNs for image-related tasks).

Finally, I'd like to say that there are no absolutes (e.g. SVMs will alawys outperform DTs). There is also a theorem along these lines: No Free Lunch Theorem.


Custom conditional loss function in Keras

You should be able to solve this with currying. Make a function that takes the label as input and returns a function which takes y_true and y_pred as input. Note that the label needs to be a constant or a tensor for this to work.

def conditional_loss_function(l):
    def loss(y_true, y_pred):
        if l == 0:
            return loss_funtion1(y_true, y_pred)
        else:
            return loss_funtion2(y_true, y_pred)
    return loss

model.compile(loss=conditional_loss_function(l), optimizer=...)

Small working example with different loss function depending on the label:

# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
data = dataset[:,0:8]
label = dataset[:,8]

X = Input(shape=(8,))
Y = Input(shape=(1,))
x = Dense(12, input_dim=8, activation='relu')(X)
x = Dense(8, activation='relu')(x)
predictions = Dense(1, activation='sigmoid')(x)

def custom_loss(l):
    def loss(y_true, y_pred):
        if l == 0:
            return binary_crossentropy(y_true, y_pred)
        else:
            return mean_squared_error(y_true, y_pred)
    return loss

# Compile model
model = Model(inputs=[X, Y], outputs=predictions)
model.compile(loss=custom_loss(Y), optimizer='adam', metrics=['accuracy'])

# Fit the model
model.fit(x=[data, label], y=label, epochs=150)

# evaluate the model
scores = model.evaluate([data, label], label)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))


Why doesn't AlphaGo expect its opponent to play the best possible moves?

It appears that AlphaGo did not rate the move as a best possible move for Lee Sedol, just as one that was within its search space. To put into context the board is 19x19, so a 1 in 10000 chance of a move is much lower than chance of the square being picked at random. That likely makes the move that it "found" not worth exploring much deeper.

It is important to note too that the probabilities assigned to moves are equivalent to AlphaGo's rating for quality of that move - i.e. AlphaGo predicted that this was a bad choice for its opponent. Another way of saying this, is "there is a probability p that this move is the best possible one, and therefore worth investigating further". There is no separate quality rating - AlphaGo does not model "opponent's chance of making a move" separately from "opponent's chance of gaining the highest score from this position if he/she makes that move". There is just one probability covering both those meanings 1

As I understand it, AlphaGo rates the probabilities of all possible moves at each game board state that it considers (starting with the current board), and employs the most search effort for deeper searches on the highest rated ones. I don't know the ratios or how many nodes are visited in a typical search, but expect that a 1 in 10000 rating would not have been explored in much detail if at all.

It is not surprising to see the probability calculation in the system logs, as the logs likely contain the ratings for all legal next moves, as well as ratings for things that didn't actually happen in the game but AlphaGo considered in its deeper searches.

It is also not surprising that AlphaGo failed to rate the move correctly. The neural network is not expected to be a perfect oracle that rates all moves perfectly (if it was, then there would be no need to search). In fact, the opposite could be said to be the case - it is surprising (and of course an amazing feat of engineering) just how good the predictions are, good enough to beat a world-class champion. This is not the same as solving the game though. Go remains "unsolved", even if machines can beat humans, there is an unknown amount of additional room for better and better players - and in the immediate future that could be human or machine.

There are in fact two networks evaluating two different things - the "policy network" evaluates potential moves, and the output of that affects the Monte Carlo search. There is also a "value network" which assesses board states to score the end point of the search. It is the policy network that predicted the low probability of the move, which meant that the search had little or no chance of exploring game states past Lee Sedol's move (if it had, maybe the value network would of detected a poor end result from playing that through). In reinforcement learning, a policy is set of rules, based on known state, that decide between actions that an agent can take.


diseases risk prediction using social media comments

You should also have comments/symptoms of a reference group of non-diabetes patients to be able to make a distinction yes/no diabetes.

Do you have users of which you know they don't have diabetes?
If not, "Is often thirsty?" will always map to diabetes, because you have no other data.

Non-realistic example:

Suppose 99% of diabetes patients eat 10 candy bars a day, and 1% does not eat candy bars.
Then suppose Suppose 99% of non-diabetes patients eat 10 candy bars a day, and 1% does not eat candy bars.
There is a strong correlation between eating candy bars and having diabetes.
Your data of only diabetes patients shows that both people who eat 10 candy bars and people who eat no candy bars develop diabetes.
Data that includes non-diabetes patients will reveal that people that eat 0 candy bars are much more likely to not have diabetes.

Or you should have comments and diabetes type data to be able to make a distinction diabetes type 1 or type 2.
"How often do you use medication?" is a strong indication between the two, but you'd need the data.

Furthermore, expect a risk score with at least weighted numbers that can be either positive or negative, for weak or strong (inverse) correlation between an answer and indication for diabetes.

There are a lot more advanced techniques (e.g. with neural networks) that will have better results, including finding correlations between answers.

If you are doing anything more serious than 'Go see the doctor' on >50% of positive indicators, you should really do more research on data science models, but also on how to formulate the questions etc.


Which is better: Cross validation or a validation set for hyperparameter optimization?

The $k$-fold cross-validation (CV) process (method 2) actually does the same thing as method 1, but it repeats the steps on the training and validation sets $k$ times. So with CV the performance is averaged across the $k$ runs before selecting the best hyper-parameter values. This makes the performance and value selection more reliable in general, since there is less risk to obtain the best result by chance. However it takes much longer (since repeating $k$ times), so if the training process is long it's not always practical to use CV.


The ranking is derived form the size of the eigenvalue of the principal component (largest on top) and the scores represent 1 - cumulative sum of the variance.


sklearn.metrics.classification_report provides precision and recall for all classes along with F-score and support. It might prove to be helpful in your case of 3 classes.


What are the alternatives to Python + Spark (pyspark)?

Try checking dask. It's a distributed library which is native with Python and it uses pandas and numpy. So its like using pandas with some wrapper for distribution computation.


How to deal with Nominal categorical with label encoding?

Label encoder is better suited for output classes. You are looking to handle categorical variables. The term is "one hot encoding" which means creating a binary column for each unique value of categorical data. Check get_dummies function in Pandas
It is hard to give any suggestion on model to use without knowing the objective, amount of data available, accuracy requirements. You are better off trying multiple models and checking test set errors to see which one works better.
Variables with high degree of missing values or low fill rate can be removed from the dataset.


How important is outcome variable scaling in SVM regression?

This is a cross-posting from CrossValidated:
In support vector regression (with linear loss), we minimise the objective function:
\begin{align}
    \min_{\mathbf{w}, b, \mathbf{\xi}} \quad & \frac{1}{2}\| \mathbf{w}\|^2 + C\sum_i \mathbf{\xi}_i + \mathbf{\hat \xi}_i \\
    \text{s.t.} \quad & (\mathbf{w} \cdot \mathbf{x}_i + b) - y_i \leq \varepsilon + \xi_i \\
    \quad &  y_i - (\mathbf{w} \cdot \mathbf{x}_i + b) \leq \varepsilon + \hat \xi_i \\
    \quad & \xi_i, \hat \xi_i \geq 0
\end{align}
As you can see from the conditions, $\varepsilon$ and the slack variables $\xi_i, \hat \xi_i$ are on the same scale as the output variables $y_i$. So, as you scale $y$, you need to scale $\varepsilon$. On the other hand, in the objective function, $\mathbf{w}$ enters quadratically, and the slack variables only linearly. To keep the balance unchanged, you also need to scale the multiplicative factor $C$.
Assuming you've adjusted the hyperparameters accordingly, the scaling has no effect:
import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm

def svmTest(x, y1, sc=1):
  N = x.shape[0]
  x = x.reshape([-1, 1])
  regr = svm.SVR(C=sc*N/100, epsilon=sc*.2)
  y2 = sc*y1
  regr.fit(x, y2)
  y2p = regr.predict(x)

  plt.plot(x, y2, '.', )
  plt.scatter(x[regr.support_], y2[regr.support_], s=80, facecolors='none', edgecolors='r')
  plt.plot(x, y2p, '-')
  plt.gca().set_title(f'Outcome variable scaled by {sc}')

x = np.arange(-1, 1, .01)
y = np.cos(2*np.pi*x)*np.exp(-np.abs(x)) + np.random.normal(0, .1, x.shape[0])
x = x.reshape([-1, 1])

svmTest(x, y, 1) produces:

while svmTest(x, y, 1e6) produces:


Datasets with only single feature variable

Finding a dataset with just two variables might be difficult because these would be a very naive dataset.

Therefore, what I would suggest, is for your to take a more complex dataset and pick two columns that you would like to analyse.

For example, you could look at this dataset: https://www.kaggle.com/sohier/calcofi#bottle.csv

Then, pick the column Depthm as your feature variable and the column T_degC are your target variable. These two columns seem to match the type of data that you expect.

Here is how these two variables look like (for Depthm > 2000)

or you could look at  another two variables instead, for example Depthm vs Salnty:


spaCy - Text Preprocessing - Keeping "Pronouns" in text

What you are doing seems fine in terms of preprocessing. Removing less informative words like stopwords, punctuation etc. is a very common technique. Here are some of my notes:

probably best for speed to load your "nlp" object outside of the function call
"-PRON-" must be the lemma for "you're" in this case. So you shouldn't remove it as it is following the correct logic. Though, you should investigate whether your model improves if you change your preprocessing logic

if you use SpaCy to create your classification model, I'd recommend experimenting training a model without using any preprocessing. As far as I know, SpaCy handles some of this implicitly through feature engineering


Is there any advanced collocation tool based on NLP+DL?

You are describing the "Language Modeling" problem in NLP. Language modeling finds the probability distribution over sequences of words - given the words I have seen so far, what are the most likely words that will come next or are missing.

word2vec can find reasonable collocations/cooccurrence for individual words. However, word2vec will do a poor job with sentence completion because it does not model grammatical dependencies.

You can find a word2vec nearest individual word demo here.


I haven't seen a library or module that will do this for you, but it should be simple enough to handle. If you saved the image names in your original dataset while loading, you can add the predictions to that same dataset and then loop through row by row, renaming the image based on the classification category.


Offline evaluation of recommender systems

There is always a bias in offline testing in recommender systems. Real evaluation happens in A/B testing. But, that should not discourage one from doing offline testing.

There is ongoing research on this topic using multi-armed bandits. I recommend reading offline testing procedures using reco-gym.

Following workshop talks about removing these biases and developing efficient offline estimators for recommender systems. They also conducted an online course on efficiently doing an offline evaluation of recommender systems.

You can also follow a video tutorial on this website on counterfactual evaluation


Impact of varying sequence length in ensemble GRU model

The number of units in an RNN layer determines the amount of "memory" of that layer. Higher number of units make the model (potentially) able to remember longer sequences, and to explain current values based on information that dates more back in time. If on the other side the number of units is shorter, the model at that layer won't be able to recover information way back in the past. This is tru for any Recurren architecture (simple recurrent, LSTM, GRU).

If you increase the number of units of a GRU layer and you see that there's no improvement in the quality of your model, that probably means that your time series data have "shorter memory", i.e. that you don't need to go that far back in time in order to explain current values. Of course I didn't take a look at your data, this is just the most likely reason IMHO.


What is the meaning of semi-handcrafted features?

It means part of it handled by the human at first, and then, base on them running a machinery learning algorithms to extract new features. For example, gather some features from the human about some pictures, and using some machinery algorithms to find some new features by some non-linear combinations of those features.


Numerical example of Confusion in understanding learning rate in xgboost

Each iteration is supposed to provide an improvement to the training loss.
Such improvement is multiplied with the learning rate in order to perform smaller updates.

Smaller updates allow to overfit slower the data, but requires more iterations for training.

For instance, doing 5 iteations at a learning rate of 0.1 approximately would require doing 5000 iterations at a learning rate of 0.001, which might be obnoxious for large datasets.

Typically, we use a learning rate of 0.05 or lower for training, while a learning rate of 0.10 or larger is used for tinkering the hyperparameters...


Data Visualization with multiple dimension, and linear separability

You basically need a t-SNE plot, the t-SNE will convert the high dimensional feature vector (several features in your case) to a 2d point and then you can use matplotlib to plot, while plotting you need to send in the class of the corresponding feature to get different colour for different classes of data points. once you do this, you will be able to judge if they are linearly separable.

the code goes as follows

import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=0)
transformed_data = tsne.fit_transform(features)
k = np.array(transformed_data)
plt.scatter(k[:, 0], k[:, 1], c=class_labels, zorder=10, s=0.4)

and the above code should produce an image as shown below

Since you're looking for MATLAB code you can get the t-SNE with the function tsne(x) click on it for documentation. Also matplotlib scatter plot is pretty easy to do on Matlab, the documentation is extensive!

vote me if i was able to help ;)


How to validate a clustering model without a ground truth?

In my opinion there are two ways:

Ask a few experts to assess the quality of the clusters based on a sample (after the clustering has been done, much easier than pre-annotating the whole data especially in the case of clustering)
If the clustering is done in the perspective of using the result in another task, the performance of this other task will reflect the quality of the clustering.

Imho any measure based on the distance between clusters or other technical measure would be a flawed evaluation, because it would depend on the quality of the representation. Such measures might provide some useful indications though, just not a proper evaluation for the task.


How do CNNs find different feature maps?

You shouldn't initialize all the weights with zero, use random initialization instead. Otherwise you can't break the symmetry and all the outputs in the network would be same.


Welcome to the community!
There are points coming to my mind:

Check the amount of data in each block and then their distribution. First experience might be due to the lack of enough data in reserved block (i.e. you literally just trained but did not validated resulting to a complete overfitting) or having a totally different distribution (i.e. the difference between blocks is NOT random).
Be sure about wht you called "Cross Validation". If you splitted 49 blocks into a training and a testing set, then it is minimal version of Cross Validation. 49 blocks should be splitted to $k$ different set, each time one of them plays the test set role and then the final error is averaged over all $k$ runs. That is statistically more robust (first experience shows problem in robustness of statistical inference)
Second experiment uses 100 blocks instead of 50 (where were those other 50 in first experiment?) so first, you have more data, and second, you did not split according to "blocks" (whatever they are) but merged all and randomly chose train, validation and test. This procedure reduces the danger of distributional bias within train/valid/test and also might be an indicator of the fact that statistically those "blocks" are different (their distribution)

hope it helped. Please share your results so everyone can learn from it.
Good Luck!


SKLearn has an amazing array of HMM implementations, and because the library is very heavily used, odds are you can find tutorials and other StackOverflow comments about it, so definitely a good start.

http://scikit-learn.sourceforge.net/stable/modules/hmm.html

PS: Right now its outdated in June 2019

As of July 2019 you can use hmmlearn (pip3 install hmmlearn)


How to find coherence between a large number of sentences

You could use clustering with a more basic similarity measure, for example cosine or even simply the proportion of words in common (e.g. Jaccard, overlap coefficient). This should gives you groups of sentences which are "quite similar" against each other, whereas sentences in different clusters are supposed to be very different. This way you would only have to compute WMD distance between smaller groups of sentences. By increasing the number of clusters the clusters would be smaller so there would be less WMD computation needed, however there is more risk to miss a pair of sentences since they could end up in different clusters.


Is it OK to try to find the best PCA k parameter as we do with other hyperparameters?

Your emphasis on using a validation set rather than the training set for selecting $k$ is a good practice and should be followed. However, we can do even better!

The parameter $k$ in $\text{PCA}$ is more special than a general hyper-parameter. Because, the solution to $\text{PCA}(k)$ already exists in $\text{PCA}(K)$, for $K > k$, which is the first $k$ Eigenvectors (corresponding to $k$ largest Eigenvalues) in $\text{PCA}(K)$. Therefore, instead of running $\text{PCA}(1)$, $\text{PCA}(4)$, ..., $\text{PCA}(K)$ separately on training data, as we do for a hyper-parameter in general, we only need to run $\text{PCA}(K)$ to have the solution for all $k \in \{1,..,K\}$.

As a result, the process would be as follows:

Run $\text{PCA}$ for the largest acceptable $K$ on training set,
Plot, or prepare ($k$, variance) on validation set,
Select the $k$ that gives the minimum acceptable variance, e.g. 90% or 99%.

And, N-fold cross validation would be as follows:

Run $\text{PCA}$ for the largest acceptable $K$ on N training folds,
Plot, or prepare ($k$, average of N variances) on held-out folds,
Select the $k$ that gives the minimum acceptable average variance, e.g. 90% or 99%.

Also, here is a related post that asks "why do we choose principal components based on maximum variance explained?".


Understanding neural network probability

What you have declared here is actually a 2-class multi-label classifier. It is trying to learn the probability of being each class independently where they are not mutually exclusive.

While that is a valid classifier problem to solve, I'm guessing what you mean is that you have an exclusive 2-class problem: it's either positive, or negative. In that case, you want the final layer to be Dense(1, activation='sigmoid')


Feature Extraction Technique - Summarizing a Sequence of Data

would LOVE to see a box written which collected case studies on feature engineering / extraction

Please advise if this helps

Discretization of Time Series Data
http://arxiv.org/ftp/q-bio/papers/0505/0505028.pdf
Optimizing Time Series Discretization for Knowledge Discovery
https://www.uni-marburg.de/fb12/datenbionik/pdf/pubs/2005/moerchen05optimizing
Experiencing SAX: a Novel Symbolic Representation of Time Series
http://cs.gmu.edu/~jessica/SAX_DAMI_preprint.pdf
Indexing for Interactive Exploration of Big Data Series
http://acs.ict.ac.cn/storage/slides/Indexing_for_Interactive_Exploration_of_Big_Data_Series.pdf
Generalized Feature Extraction for Structural Pattern Recognition in
Time-series Data
http://www.semanticscholar.org/paper/Generalized-Feature-Extraction-for-Structural-Olszewski-Maxion/7838bcd87bb6616e9fd3ffd92d4676a7082da34c
Computing and Visualizing Dynamic Time Warping Alignments in R: The dtw Package
https://cran.r-project.org/web/packages/dtw/vignettes/dtw.pdf


Learning cost function for linear regression

There is a worked out example in the Wikipedia page for "simple linear regression"

Just for the sake of it, let me plug in your example into the formulas:

The fitted model should be a straight line with parameters $\alpha$ (value at $x = 0$) and $\beta$ (the slope):

$$f(x) = \alpha + \beta x$$

The values for these parameters that minimize the distance between line and data points are called $\hat{\alpha}$ and $\hat{\beta}$.  They can be computed out of the data point values by using these formulae, derived here:

$$\begin{align}
\hat{\beta} & = \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } ,
\\
\\
\hat{\alpha} & = \bar{y} - \hat{\beta}\bar{x}
\end{align}
$$

where an expression with an overline $\overline{xy}$ means the sample average of that expression: $\overline{xy} = \tfrac{1}{n} \sum_{i=1}^n{x_iy_i}$.

Here are the values I find for the datapoints you have listed in your question:

$$
\begin{align}
\overline{xy} &= \frac{1}{4} \sum{<(1 \times 4000), (2 \times 10000), (3 \times 22000), (4 \times 30000)>} \\
              &= \frac{1}{4}(4000 + 20000 + 66000 + 120000) = 52500, \\
\overline{x} &= \frac{1}{4} \sum{<1, 2, 3, 4>} = 2.5 ,\\
\overline{y} &= \frac{1}{4} \sum{<4000, 10000, 22000, 30000>} = 16500 , \\
\overline{x^2} &= \frac{1}{4} \sum{<1^2, 2^2, 3^2, 4^2>} = 7.5 , \\
\overline{x}^2 &= 2.5^2 = 6.25
\end{align}
$$

and the fitted line should be:

$$
\begin{align}\
\hat{\beta} &= \frac{52500 - 2.5 \times 16500}{7.5 - 6.25} = \frac{41250}{1.25} = 33000 , \\
\hat{\alpha} &= 16500 - 33000 \times 2.5 = - 66000 , \\
\Rightarrow f(x) &= - 66000 + 33000 x
\end{align}
$$

Therefore, the model would predict, for a house with 10 rooms, a rent of:

$$ f(x) = -66000 + 33000 \times 10 = 264000 $$


You can/should use a generic Java Architecture for XML Binding (JAXB) approach.

Simply put, call Row#withContent(Object...) with instances of org.w3c.dom.Element that represent the desired XML content.

For example:

Document document = documentBuilder.newDocument();
Element shortForm = document.createElement("shortForm");
shortForm.setTextContent("m");
Element longForm = document.createElement("longForm");
longForm.setTextContent("male");
row = row.withContent(shortForm, longForm);


Does scikit-learn use regularization by default?

Please take a look at the documentation. The first line shows the default parameters, which include penalty='l2' and C=1.0.

You actually cannot disable regularization completely, you can only regularize less... try setting C=1e10 for example.


Machine Learning/Text Analysis/Algorithm Pointers

Yes, This absolutely makes sense. This is a common NLP (Natural Language Processing) problem. You should use word embedding models alingside LSTM (Long Short Term Memory) and deep neural networks. Actually, first you should represent every word as a vector of fixed dimension (e.g. 100) using word2vec. Then you should build a deep neural network architecture. The inputs of this network are your word vectors for each bill that are concatenated to form a sequence of vectors. The output of this network is a label that indicate which type of bill you are considering. I highly recommend that you use Keras which is a great python package for dealing with deep learning and NLP. Also, you can use existing word2vec sets that contain vector representations of most of words of a language (e.g. English). For example you can use GoogleNews word2vec set.


How does XGBoost use softmax as an objective function?

It's not softmax to be minimized, but crossentropy loss function, which is based on softmax. Crossentropy is calculated on a softmax output, that's why they are a standard couple in ML. Tree-based classifiers find "cuts", or portions of the variables' space in a way that minimizes the entropy of a dataset.

If you want to explore the relationship between softmax and crossentropy further, you can start with the nice explanations provided here. If you want to dig deeper, you can find a very detailed and technical explanation here.


Model-agnostic variable importance metric

Have you looked at the permutation importance approach in the eli5 package?

The idea is that instead of retraining the model without the feature, which is computationally expensive, they replace each feature in turn with random noise in the test set. To get random noise that is drawn from the same distribution as the original feature, they just randomly shuffle that feature.

Note that, as with feature importance in decision trees, this measure is biased against categorical variables with low cardinality.


Why can't Policy Gradient Algorithm be seen as an Actor-Critic Method?

In RL we have:

Actor-only methods such as REINFORCE in which the output is a probability distributions over actions. REINFORCE is a policy gradient method but doesnt use a critic.
Critic-only methods such as Q-learning in which the output is the expected reward for every available action ($Q(s,a)$ $ \forall a\in A $)
Actor-Critic methods that involve both Actor and Critic estimations. For example the popular DDPG and A3C algorithms. Both algorithms are policy gradient methods. By reading the papers you will start getting a sense on why the simple REINFORCE introduces variance in gradient estimations and how a critic can reduce it.

Policy Gradient methods are based on the Policy Gradient theorem. A standard implementation is an Actor-Critic algorithm, and we use both Actor (probability distribution over actions) and Critic (Value functions) in order to trade-off bias and variance in your gradient estimations.


From my understanding you have dataframe containing a list of Buyer ID's, the product they bought and how many of it they bought. You want to find out what percentage of the total of each product bought each customer was responsible.

For instance if buyer 1 bought all of A then they would 100% and if buyer 2 bought half of all B then they would be 50%.

Each buyer only buys one type of product at a time.

I generated a dataset to replicate this as so:

options = 'ABCDEF'

product = []
count = []
buyer = []

for i in range(0,100):
    product.append(test[np.random.randint(0,5)])
    count.append(np.random.randint(0,10))
    buyer.append(i)

df = pd.DataFrame(data = [buyer, product, count]).transpose()

df.columns = ['buyer','product','count']
print(df.head())
>>> buyer product count
0     0       B     4
1     1       B     6
2     2       A     2
3     3       D     2
4     4       D     5

In order to calculate each customers percentage you will need the total count of each product bought

totals = {'A' : 0,
         'B' : 0,
         'C' : 0,
         'D' : 0,
         'E' : 0,
         'F' : 0}

for i in range(0, len(df)):
    currentProduct = df.iloc[i,1]
    totals[currentProduct] = totals[currentProduct] + 1

Now all you need to do is calculate count/total for each product and save to a new column in the dataframe.

for i in range(0, len(df)): # Iterate over every row in
    # new column | count bought | total bought by all
    df.iloc[i,3] = df['count'][i]/totals[df['product'][i]]
print(df.head())
>>> buyer product count  contribution
0     0       B     4      0.210526
1     1       B     6      0.315789
2     2       A     2      0.080000
3     3       D     2      0.111111
4     4       D     5      0.277778
print(totals)
>>> {'A': 25, 'B': 19, 'C': 15, 'D': 18, 'E': 23, 'F': 0}

I know it's not the same method that you where using but it should still work, if you want please post a sample of your code so that we can look bugs, alternatively just tick this answer and change the title to represent the question that was answered.

If i misunderstood anything just say so


why is an optimizer used in something that is not a neural network?

In general, there is more than one kind of optimiser. The gradient-based optimisers such as gradient descent, stochastic gradient descent, Adam, Adagrad, RMSProp, form one broad category of optimiser that can find minima or maxima of scalar functions provided it is possible to calculate a gradient.
In your question however, you are effectively focusing just on gradient-based optimisers - why can they be used to alter input images or latent spaces, as well as neural network weights?
The answer is that these optimisers are not specific to neural network weights, or even to neural networks. They are appropriate whenever:

You can measure success at a task with a scalar function - typically this is a loss or cost function to minimise, but it can also be a numerical score that you wish to maximise.

The task involves parameters that you can control. This is a set of variables that can be changed in order to solve a problem. The weights of a neural network are one example, but so are the pixels in an image if your goal is to find an image that matches some criteria (such as looking like a photo of a face).

You have a way to estimate gradients of that function with respect to the parameters that you can control. Usually some form of back propagation is used to get from the loss function to the parameters you care about. That is the case in style transfer or stylegan, but it is not a requirement.

The basic requirement is being able to estimate the gradient of your scoring function with respect to parameters that can be changed. Then you can minimise or maximise that function - or at least find local minima and maxima. Neural network weights are just one common example of the approach.
Likewise, using minibatches of input/output pair examples is one common way of estimating gradients. It is commonly used when learning how to generalise from a data set, but is not the only possible way to get gradients - which approaches are available and make sense depends on the optimisation problem you need to solve.


How to reverse engineer a logarithmic equation

You can determine the most likely values for $\log x$ and $\log y$ by linear regression since your relationship implies:
$\log a = b.\log y + \log x$
So run regression on $b$ as input and $\log a$ as output. The gradient will be $\log y$ and the intercept $\log x$


Why does applying PCA on targets causes underfitting?

First, thanks for the edits to your original question since we now know that you are applying the same transformation to all of your data.

Q: Why do perceptrons perform so much better than generalized linear models for some problems? A: Because they are inherently nonlinear models, with a great deal of flexibility.  The drawback is that the additional knobs require more data to correctly tune.

Big picture:

Less data can cause high-bias.  High-bias can be overcome by more data.  You have reduced your data from a 4800 feature dataset to a 38 feature dataset, so should expect to see increased bias.  Neural networks require more data than models without hidden layers.

Linearity vs. Nonlinearity

Your artificial neural network (perceptron) is an inherently nonlinear model, yet you are deciding to remove features from your dataset with a linear model (PCA).  The presence of a single hidden layer explicitly creates second order terms in your data, and there are two additional nonlinear transformations (input==>hidden and hidden==>prediction), which add sigmoidal nonlinearity at each step.

The fact that the training data and the target data are both multiplied by the same matrix to achieve the reduced dimensionality really just means that your perceptron needs to learn the matrix if it wants to reconstruct nonlinear aspects of the original data.  This requires more data or for you to reduce less of the dimensionality.

An Experiment:

I suggest trying an experiment where you perform second order polynomial feature extraction on the full dataset and then performing PCA on that enhanced dataset.  See how many features you end up with while retaining 99% of the variance of the enhanced dataset.  If it is larger than the dimensionality of your initial dataset then stick with the un-enhanced and un-reduced dataset.  If it is betweeen the dimensionality of the original data and 38 then try training the perceptron with that data.

A better idea:

Rather than using the (linear) variance to determine the feature reduction of your PCA projection, try training and cross validating your model with different amounts of PCA dimensionality reduction.  You will likely find that there is a sweet spot for a given set of images.  For instance, an SVM on the MNIST digit data performs best when the 784 pixel features are reduced down to 50 linearly independent features using PCA.  Though this is not obvious from analyzing the variance of the principal components.

Other options:

There are nonlinear dimensionality reduction techniques, such as isomap.  You could investigate the use of nonlinear feature reduction since you are clearly loosing information with the linear PCA that you have applied.

You could also look into image specific feature extraction techniques to add some nonlinearity before reducing the dimensionality.

Hope this helps!


High error machine learning regressor algorithm in Python - XGBOOST Regressor

It is not necessary to one-hot-encode your categorical features when using tree-based methods. Basically the idea is that the tree has to make many splits to figure out the category. Instead, you can use ordinal encoder (even if categories are not ordered).
I also would not have set 'YEAR_BUILT' as a categorical variable, even though it is discrete. By one-hot encoding this variable, you end up with a very splitted dataset where the tree has to consider every YEAR_BUILT as a subset and it probably has too few observations to do so. By considering it as continuous it enables the algorithm to determine 2 or 3 pivot years that are relevant (changed currency, economic crisis, ...).

I think the second point can really improve performance.


2D Pytorch tensor doesn't have independent random values

You don't use the parameters "loc" and "scale" the right way. They are not suppose to be tensors. Bellow the right syntax :

dist = torch.distributions.StudentT(10, 0, 1)
dist = torch.distributions.StudentT(10) # 0 and 1 are the default parameters

Then you can sample multiple values like that :

t = dist.rsample(torch.Size([n,m]))
t = dist.rsample(torch.Size([n]))


Given rankings and the factors that determined the rankings, what is the best method to reverse engineer the rankings?

Depending on the complexity of the ranking scheme, this might be more or less difficult, but if you have all the inputs and the outputs, it is very possible.

Start by plotting your data and try to see if you can see a pattern, or any obvious correlation, even though not necessarily linear. Do that for each variable.

You can also look at the distribution of your variables (what you call factors) for each of your rank, and see how they differ.

Once you have a better understanding on the relationship between the dependent variable (rank) and the independent variables (factors), then you can either:

Come up with the model right there
Apply a machine learning algorithm (linear/polynomial regression, decision trees, etc.) which will learn the model's parameters for you. Which one you chose will depend on your analysis.


Whenever I work with xgboost I often make my own homebrew parameter search but you can do it with the caret package as well like KrisP just mentioned.

Caret

See this answer on Cross Validated for a thorough explanation on how to use the caret package for hyperparameter search on xgboost.
How to tune hyperparameters of xgboost trees?

Custom Grid Search

I often begin with a few assumptions based on Owen Zhang's slides on tips for data science P. 14

Here you can see that you'll mostly need to tune row sampling, column sampling and maybe maximum tree depth. This is how I do a custom row sampling and column sampling search for a problem I am working on at the moment:

searchGridSubCol <- expand.grid(subsample = c(0.5, 0.75, 1),
                                colsample_bytree = c(0.6, 0.8, 1))
ntrees <- 100

#Build a xgb.DMatrix object
DMMatrixTrain <- xgb.DMatrix(data = yourMatrix, label = yourTarget)

rmseErrorsHyperparameters <- apply(searchGridSubCol, 1, function(parameterList){

    #Extract Parameters to test
    currentSubsampleRate <- parameterList[["subsample"]]
    currentColsampleRate <- parameterList[["colsample_bytree"]]

    xgboostModelCV <- xgb.cv(data =  DMMatrixTrain, nrounds = ntrees, nfold = 5, showsd = TRUE,
                           metrics = "rmse", verbose = TRUE, "eval_metric" = "rmse",
                           "objective" = "reg:linear", "max.depth" = 15, "eta" = 2/ntrees,
                           "subsample" = currentSubsampleRate, "colsample_bytree" = currentColsampleRate)

    xvalidationScores <- as.data.frame(xgboostModelCV)
    #Save rmse of the last iteration
    rmse <- tail(xvalidationScores$test.rmse.mean, 1)

    return(c(rmse, currentSubsampleRate, currentColsampleRate))

})

And combined with some ggplot2 magic using the results of that apply function you can plot a graphical representation of the search.

In this plot lighter colors represent lower error and each block represents a unique combination of column sampling and row sampling. So if you want to perform an additional search of say eta (or tree depth) you will end up with one of these plots for each eta parameters tested.

I see you have a different evaluation metric (RMPSE), just plug that in the cross validation function and you'll get the desired result. Besides that I wouldn't worry too much about fine tuning the other parameters because doing so won't improve performance too much, at least not so much compared to spending more time engineering features or cleaning the data.

Others

Random search and Bayesian parameter selection are also possible but I haven't made/found an implementation of them yet.

Here is a good primer on bayesian Optimization of hyperparameters by Max Kuhn creator of caret.

http://blog.revolutionanalytics.com/2016/06/bayesian-optimization-of-machine-learning-models.html


How to decide which images to label next?

This type of problem is considered to be part of 'active learning'. There is a lot of research being done on this topic at the moment, but some first approaches are relatively easy, depending on the type of model that you are using. Since you mentioned that you are using deep learning bounding box detectors, I will showcase a few examples of how to approach this problem using Convolutional neural networks.

The core idea is that we want some measure of potential gain of an unlabeled sample. That way we can train our model on our labeled training set, predict the labels for our unlabeled set and measure which examples will be most useful to label.

In case of classification you could use the sigmoid/softmax output and get some kind of uncertainty from there, however deep learning models are usually fairly certain about their predictions and a high probability doesn't automatically mean that it predicts it well.

Another approach is to use dropout in your model during training, and then apply dropout to your predictions on your unlabeled set as well. By sampling multiple dropout masks and comparing all the different predictions, you could measure how different the outputs are. If the outputs are very similar, it's unlikely that your model will learn much more if you label this, but if the outputs vary wildly, maybe this example lives in a part of your feature space that your model doesn't know or understand very well yet.

There are a lot of ways to approach this, what I have written here is just an introduction to the concept of 'active learning'. There are a lot of papers available about this topic! EDIT: I haven't actually read a lot of this research, but here are a few:

https://arxiv.org/pdf/1703.02910.pdf

https://arxiv.org/pdf/1707.05928.pdf

https://arxiv.org/pdf/1701.03551.pdf


SelectFromModel vs RFE - huge difference in model performance

To start with, let me repeat here what I have already answered to another OP wondering if Recursive feature selection may not yield higher performance?:

There is simply no guarantee that any kind of feature selection
(backward, forward, recursive - you name it) will actually lead to
better performance in general. None at all. Such tools are there for
convenience only - they may work, or they may not. Best guide and
ultimate judge is always the experiment.
Apart from some very specific cases in linear or logistic regression,
most notably the Lasso (which, no coincidence, actually comes from
statistics), or somewhat extreme cases with too many features (aka
The curse of dimensionality), even when it works (or doesn't), there is not necessarily much to explain as to why (or why not).

Having clarified that, let's see your case in more detail.

None of the feature selection procedures here takes into account the model performance; in classification settings, the sole criterion by which features are deemed as "important" or not is the mean decrease in the Gini impurity achieved by splitting in the respective feature; for some background, see the following threads (although they are about Random Forests, the rationale is identical):

How is the 'feature_importance_' value calculated in sklearn random forest regressor?
Relative importance of a set of predictors in a random forests classification in R
How are feature_importances in RandomForestClassifier determined?

Although it is often implicitly assumed that a reduction of the features using this importance as a criterion may lead to gains in the performance metric, this is by no means certain and far from straightforward (I am actually repeating my intro here).
Given that, it would seem that the actual question here should be why the two methods end up selecting different features, for which the thread you have linked yourself, Difference between RFE and SelectFromModel in Scikit-Learn, is arguably relevant. In any case, the expectation that they should offer similar results is arguably not well-founded; the relative feature importance changes when features are removed from the model (RFE case), hence it cannot be directly compared with the SelectFromModel approach (use all features, remove those with importance below a threshold). Similarly, the question "Is there any known use case where RFE will fare better?" is ill-posed - you have not shown that RFE is consistently inferior, and the results of a single experiment with a single dataset and a single parameter setting (such as the no. of required features and the threshold involved in SelectFromModel) should not be generalized light-heartedly.
In any case, selecting features based on their Gini importance (Mean Decrease in Impurity - MDI) has started falling out of fashion, mainly because the calculated importance is spurious in cases of categorical features with high cardinality (see the academic paper The revival of the Gini importance?); in fact, there is already a relevant warning in the scikit-learn classifiers inorporating this "classical" feature_importances attribute:

Warning: impurity-based feature importances can be misleading for high cardinality features (many unique values). See sklearn.inspection.permutation_importance as an alternative.

See the scikit-learn vignette Permutation Importance vs Random Forest Feature Importance (MDI) for a concrete example.

Irrelevant to the exact question, and if the classes in your dataset are imbalanced (as they seem to be), you could (and should) inform your models about this, using the class_weight argument of ExtraTreesClassifier (docs), i.e. change the model in both cases to
ExtraTreesClassifier(class_weight='balanced')

(h/t to Ben Reiniger for a constructive comment that helped to improve the answer)


Are stacked NN the second generation of NN?

No.

Kumar, Satish. Neural networks: a classroom approach. Tata McGraw-Hill Education, 2004.:

  The first generation of neural network models employed McCulloch-Pitts TLN type neuron. They were essentially switching networks. The second generation of neural network models employed a smooth sigmoids. The sigmoidal function was motivated by the increase in firing rate of a neuron with the net input, which is saturates at a certain maximum frequency. The third generation of neural network models will use spiking neurons which encode information in spike trains.


Adaboost - Show that adjusting weights brings error of current iteration to 0.5

You're nearly there.  The quantity $err_t/(1-err_t)$ is exactly what you need it to be.  It might be easier to see if you think about
$$\sum_{i=1}^N w_i \Pi(h_t(x^{(i)})=t^{(i)})$$
as
$$\sum_{i:\ x^{(i)}\text{ is correctly classified}} w_i$$
(just using the indicator function to reduce the summation range).


What does a blue empty cell mean [ ] in Anaconda if the cell above is still [*]?

The cell with [*] means it is currently running, you can queue other cells for execution but until this cell finishes they will not run.
When it is empty [] it means you haven't run this cell.


Can someone explain to me the structure of a plain Recurrent Neural Network?

The architecture of a RNN is called recurrent because it applies the same function at each step. So all the cells on the graph actually represent the same computation, but not the same state. Each green square in your figure represent the computation.
$$
s^{(t)} = f(s^{(t-1)}, x^{(t)}, \theta)
$$
Where $f$ is the function of the RNN, $\theta$ are parameters, $s^{(t)}$ is the RNN state at step $t$ and $x^{(t)}$ is the input at step $t$ in the input sequence.
What you see represented in the figure is actually what is called the unfolded representation, that is the same RNN cell applied to the input sequence one input vector at a time.
I recommend you to read the chapter 10 on RNN Deep Learning book. The following figure is from this book and summarize the idea.


Why is the cost increasing in the linear regression method?

The reason is your learning rate alpha is too large for this optimization problem. Start with a really small value (< 0.000001) and you will observe a decrease in your cost function.

Keep in mind that when the learning rate is too large, the gradient descent algorithm will miss the global minimum (global because MSE cost function is convex) and will diverge. This is why you have observed still increasing cost function values with alpha = 0.001.


Sourcing (discounted) products customers want

Clustering is not well suited for prediction.

Use a recommender system instead.


You need to take a step back in order to decide which model would suit best for your use case. Before doing that predict_proba is definitely important to calculate the posterior probability of the class labels, but it isn't great for comparing against other model outputs (especially to decide which models suits best for your prediction).

Logistic Regression, QDA and LDA all have different approaches. Logistic regression is based on maximum likelihood estimation while LDA and QDA are based on Bayes theorem. To understand which classifier best suits your model we need to go over the assumptions (assuming you know the mathematical expressions) and then you can judge which best apply to you.

1. Logistic Regression

In Logistic regression, it is possible to directly get the probability of an observation for a class (Y=k) for a particular observation (X=x). There is nothing to assume to run logistic regression for classification. Its generally a safe go to method which is not exigent and is robust.

2. LDA & QDA

LDA and QDA algorithm is based on Bayes theorem and classification of an observation is done in following two steps.

Identify the distribution for input X for each of the class (or groups ex Y=k1, k2, k3 etc )
Flip the distribution using Bayes theorem to calculate the probability Pr(Y=k|X=x)

Following are the assumption required for LDA and QDA:

LDA Assumption:

Common covariance across all response classes σ2 ( for ex σk1 = σk2 =
σk3 for k1, k2 , k3 response classes )
Distribution of observation in each of the response classes is normal
with a class-specific mean (µk) and common covariance σ.

QDA Assumption:

Different covariance for each of the response classes. For ex – σk1,
σk2, σk3 for response class k1, k2, k3 etc.
Distribution of observation in each of the response class is normal
with a class-specific mean (µk) and class-specific covariance (σk2).

Notes:

LDA (Linear Discriminant Analysis) is used when a linear boundary is required between classifiers.
QDA (Quadratic Discriminant Analysis) is used to find a non-linear boundary between classifiers.
LDA/QDA, when all its requirements met classifies better than logistic regression (more efficient).
Logistic regression is not sensitive to outliers while LDA/QDA is.

To conclude:

LDA and QDA work well when class separation and normality assumption holds.
Logistic regression has an edge on LDA/QDA for dataset that is not normal.


The data that you are showing is typical survival data. If you want to model the event depending on time and value you should look into survival models with time dependent covariates. If you want to do this without any assumptions on the distribution you can start with a Kaplan Meier estimate as explained here. If you want to use parametric models you can look at Weibull or Gamma regression.

If you are new to this topic I can highly recommend to browse the examples of the packages in the CRAN Survival Task View.


Generally, MAE is more robust to outliers, so if your data set has outliers, then you can use MAE. But if the outliers represent anomalies in data and it is important that you want to find these anomalies and report it, then we should use MSE. But, if the outliers are just the corrupt data that acts as noise in the data set, then you can use MAE.

RMSE is another very common loss function that can be used for the linear regression :


Creating a metric based on some features

Indeed, there are methodologies that have been tested elsewhere, some with greater and less success.

I will propose one of them to build a prediction of job satisfaction, which you can then enter as an explanatory variable in a supervised model of employee resignation, whose methodology you can review in this tutorial with Python code that I did some time ago: HR analytics MVP

Methodology to generate a satisfaction level prediction: Deduce the importance of the variables from a score that represents the satisfaction declared by a subset of the members of your company

I think the best way to start doing a good MVP (minimum viable product) with which you can deliver relatively fast results having a result that incorporates elements of your company is one in which you derive the importance of the features from a dataset in which have your explanatory variables and a target with a declarative satisfaction survey made to the workers from which the score that is the variable explained was calculated. For this you must follow the following steps:

1.-You design a satisfaction survey that will be answered by the workers and that will allow you to calculate a Score from it. Here the important thing is that the design of the survey is as complete as possible, that the number of respondents allows you to draw conclusions at a statistical level and, most importantly, that of those who answer the survey have how to extract the raw data that allows you later deduce which are the most relevant variables. Here are some resources that can give you some ideas of how to generate the satisfaction level index

2.-Then, using that dataset generated in step 1, you can make a feature engineer and establish which variables have the greatest impact on the satisfaction declared by the workers.

3.-Solved the point 2 you can generate predictions on the score and apply your model to the future and with other workers of the same company.

Important: Whenever you run the prediction for the next period you should do a few satisfaction surveys in each iteration to confirm that the model is still valid and to use that data as a permanent retraining. In general, the model should be useful as long as the context of the company does not undergo major changes (mergers, significant deterioration of the work environment due to massive dismissals, etc.), since in such cases you should try to capture the short and long term effects of these shock

Although this methodology is a good starting point, it is omitting many things that are difficult to detect for a company because it corresponds to exogenous variables to it, such as:

a.- That the person changes his interests and / or goals in terms of career. Example: a software developer who wants to change the focus of his career towards a more commercial facet or another specialty such as Data Science or Data Engineer

b.-That the person change their objectives and / or prioritize them in their life. Example: A person who wants to begin to dedicate more time to his personal life because he went through a crisis with his partner

Here is an example of where they used that methodology: Mining the drivers of job satisfaction using algorithmic variable importance measures

PD: There are other lines of research that avoid extracting the satisfaction index from the direct query to the employee and occupy other variables such as equivalent income or time spent in the company as equivalent metric. It is not my favorite line, but here I leave an example of that: Using equivalent income as metric


How to calculate the elapsed time of a flag status per day?

Python's pandas package has the capability to answer all those questions, assuming the data can fit into the computer's memory.

Pandas can process timeseries data, find time deltas, and remove duplicates.


Understanding outputs from ANN and how to improve validation loss

500K is decent data size. So, we can have a larger test set and also the network should be bigger.
Try the below listed suggestions -

Have a bigger model e.g. 200 Neurons, 5-6 layers
The test set should be 25%-30% since you have a decent dataset size
Keep LR to default
Increase batch size a bit as it's still fluctuating
Data should be standardized
Try adding Regularization when it starts overfitting


How to deal with an attribute including a lot of character or factor values when using caret package in R

Yeah, some (not all) libraries have that numeric input limitation.

What you're looking for is called "one-hot encoding" which can basically convert an N level factor like your product categories to N binary columns. Here is a post describing a few ways of doing that:

https://discuss.analyticsvidhya.com/t/how-to-do-one-hot-encoding-in-r/6982/5


Aggregating small values in a frequency bar plot

Here is an example where I create a new row called Other, which contains the sum of all values below a given threshold.

I then remove the rows that were below that threshold (and so included in the sum), so the final Pandas Series only has values above the threshold, plus the new Other row.

This is all performed on a randomly generated Pandas Series, as follows:

In [1]: import pandas as pd, numpy as np

In [2]: X = pd.Series(np.random.random(20))

In [3]: X
Out[3]:
0     0.507151
1     0.366259
2     0.444196
3     0.027280    this will be in the sum
4     0.132785    this will be in the sum
5     0.993170
6     0.614712
7     0.942894
8     0.516255
9     0.107436
10    0.710416
11    0.512221
12    0.502685
13    0.753515
14    0.894735
15    0.780213
16    0.998315
17    0.971558
18    0.504525
19    0.224767
dtype: float64

In [4]: threshold = 0.2

In [5]: X.loc['Other'] = X[X < threshold].sum()

In [6]: X.drop(X[X < threshold].index, inplace=True)

In [7]: X
Out[7]:
0        0.507151
1        0.366259
2        0.444196
5        0.993170
6        0.614712
7        0.942894
8        0.516255
10       0.710416
11       0.512221
12       0.502685
13       0.753515
14       0.894735
15       0.780213
16       0.998315
17       0.971558
18       0.504525
19       0.224767
Other    0.267501
dtype: float64

Notice that the indices still match the original values, i.e. they are not continguously numbered - indeices 3 and 4 have been dropped/removed.
If you want to get continuously numbered indices back, you can do the following:

X.reset_index()

This creates a new index and puts the original one shown above in a new column called index, which can then itself be dropped if desired:

X.drop('index', axis=1)

That will remove the label Other, which was in the original index.


XGBoost for binary classification: choosing the right threshold

You have to decide what you want to maximize.

Classifying by comparing the probability to 0.5 is appropriate if you want to maximize accuracy.  It's not appropriate if you want to maximize the f1 metric.

If you want to maximize accuracy, always predicting zero is the optimal classifier.

Alternatively, given a probability score $p$, another option is to randomly flip a biased coin; with probability $p$, output classification 1, otherwise output classification 0.  This doesn't always predict zero.  However it's probably not actually any better in any useful way.

If you want to maximize f1 metric, one approach is to train your classifier to predict a probability, then choose a threshold that maximizes the f1 score.  The threshold probably won't be 0.5.

Another option is to understand the cost of type I errors vs type II errors, and then assign class weights accordingly.


The maximum a posteriori definition usually goes like this:

$$ \arg\max_{\theta} p(\theta|x) = \arg\max_{\theta} \frac{p(x|\theta)\cdot p(\theta)}{p(x)} $$

and given that $p(x)$ is independent of $\theta$, it's not needed for finding the $\arg\max_{\theta}$, then you have

$$ \arg\max_{\theta} p(\theta|x) = \arg\max_{\theta} \frac{p(x|\theta)\cdot p(\theta)}{p(x)} =  \arg\max_{\theta} p(x|\theta)\cdot p(\theta)$$


Is feature importance from classification a good way to select features for clustering?

It might make sense, but it depends what you're trying to do:

If the goal is to predict the binary target for any instance, a classifier will perform much better.
If the goal is to group instances by their similarity, loosely taking the binary target into account indirectly, then clustering in this way makes sense. This would correspond to a more exploratory task where the goal is to discover patterns in the data, focusing on the features which are good indicators of the target (it depends how good they actually are).


You should definitely use a sliding window.

An n-gram language model represents the probabilities for all the n-grams. If it doesn't see a particular n-gram in the training data, for example "sliding cat", it will assume that this n-gram has probability zero (actually zero probabilities are usually replaced with very low probability by smoothing, in order to account for out-of-vocabulary n-grams). This would result in a zero probability for a sentence which was actually in the training case (or a very low probability with smoothing).

Also it's common to use "padding" at the beginning and end of every sentence, like this:

#SENT_START# The
The sliding
sliding cat
cat is
is not
...
to dance
dance #SENT_END#

This gives the model indications about the words more likely to be at the beginning or end (it also balances the number of n-grams by word in a sentence: exactly $n$ even for the first/last word).


Elements of Statistical Learning - question on p. 12

You are correct, at p+1 it is returning to talk about the K=1 case again. p inputs produce 1 output. The linear function $\beta$ you fit maps p dimensional X to a p dimensional subspace of the p+1 dimensional space that includes the 'output' and it's specifically just a hyperplane. Try plotting $z = 3x + 2y$ in a graphing calculator for a quick intuition of what single output from p=2 inputs feels like.


Determine if a piece of text is a job ad

For text classification in your case, it can be solved by training a binary classes classification network.
Main steps as follows:

1. Training data preparation

Two classes data are needed, one class contain job ad text, and the other does not contain job ad.

2. Building the network

Building a binary classification network with Keras, Caffe or any other DL library you like.

3. Train the network

Training the network on the data prepared above.

4. Text class prediction

Use the trained network to predict the text class you want to deal with.

It’s a easy classification task for neural network: )


How to use a dataset where attribute names are changed?

What piece of knowledge am I missing here to use this dataset?

By anonymising the data and altering the attribute names, the providers of this dataset have made it into an abstract machine learning exercise, where data has similar qualities to real-world credit approval data, so that approaches to train models that work well on it are likely to work well in a real-world scenario. However, those models could not be used in a production system. You are not in a position to train a model based on it and then input real world data you have collected elsewhere to make a prediction.

You can still train a model on this data to predict A16 (the class attribute). You can measure the accuracy - or any other metric - by holding out a test set. You can experiment with feature engineering, feature selection etc, in an abstract way without being able to apply much domain knowledge. You can try out different model classes, different hyper-parameters, different approaches to imputing missing values or cross-validation etc.

Your project can draw conclusions about the approach you have taken, based on test results. What you cannot do is explore real-world scenarios with simulated customer data. This might make it less compelling as a demonstration - for instance users could not explore exactly what factors might lead to an application being approved or not, instead you would just have to show some graphs that demonstrate you have selected a model with good performance.


One attribute includes another attribute

The two features "voice mail calls" and "voice mail plan" are related but they are not linearly correlated. "Voice mail plan" still contains some information that is not available from other features. Why do you want to remove "Voice mail plan" in first place? If you need to decrease your number of features, you can try dimension reduction (linear or non-linear), this way you make sure most of the variance of your feature set is considered for building your model.


Does REPEATED K-fold cross validation make sense with Random Forest?

To the title question, yes, repeated k-fold makes sense with random forests; and to the question body, no, the results will not generally be the same as repeated model fits with one k-fold split.
The reason is that fixing one k-fold split and then repeatedly fitting random forests (with different random seeds) still only gives each forest access to $(k-1)/k$ of the data at a time.  It may be easiest to think about the case when the number of trees is astronomical: the random choices for the bagging get averaged out, to the point where different random seeds don't actually matter: the forests converge to the same result, given a training split.  Then the average of the forests' scores are the same for each of the splits, and so you average just $k$ scores.  Compare that to repeated $k$-fold, where each of the forests converges, but are all on different training sets, and so the average happens with more variety.
Whether that has a sizable impact, or in which direction, is harder to say.  Repeated $k$-fold seems like it should give more stable results, even when the number of trees is something more reasonable, because the forests are less correlated.


I don't know, there might be disasters which happened due to ML but my guess is that it might be hard to really consider ML as the cause: ML is always deployed in a setting designed by humans for a very specific task, and the design must take into account the statistical nature of ML. So I would tend to interpret most problems as caused by a wrong design, not by ML itself.
If you are looking more generally into problems in the applications of ML, as far as I know the most common problems cited are:

Increasing existing bias, for instance face recognition being is more accurate with white than black faces.
Lack of transparency, especially in critical applications like medicine, military, law enforcement
Lack of a legal responsibility framework, e.g. who is at fault if a self-driving car causes an accident?


I think your question is off-topic, but I'll still try to answer because it might help you. Getting a job in data science isn't something everybody can do, it's a competitive field. Well done, for landing yourself in data science. However, data science is a big area, so you'll need to think what you want to do and plan ahead.

Do you want to get yourself into machine learning? Do you want to get yourself into statistics? Right now, you're more like a software engineer in the field, still not quite being a data scientist. Do you want to become a real data scientist? Or are you happy being a software engineer in the field? If you're planning to become a data scientist, you should train yourself hard in statistics, machine learning etc. Being a Hadoop user is simply not enough to step yourself fully into the field.

I give you my own experience. I was a software engineer myself. It wasn't easy to get a job in data science. I had to do a postgraduate degree in mathematics. Trained myself as much as possible. In the end, I landed myself a job as a bioinformatician.

Recommendation

Don't leave just because nobody mentors you. Learn as much as possible and prepare yourself. Once you're comfortable, move on.


Finding the best "depth" of ICD9 codes with pseudo-hierarchical clustering

It's called a prefix tree.

And because the ICD codes are human made, and humans tend to think in categories, there is indeed such a structure in this codes.

It's just that humans first thought of the structure, then made these codes to reflect that structure, not the other way round. What you are now doing is reverse-engineering this...


Using partially defined features in an unified deep learning model

You can fill the B features with zeroes where they are not available, and add an extra 0/1 feature indicating whether the B features are present or not.
Then, you can fit whatever neural architecture is more suitable for the nature of the input data, e.g. dense layers with ReLU activations, LSTMs.


Could not convert string to float error on KDDCup99 dataset

I notice you mentioned that you used Label encoding but I did it myself and the code runs just fine. I used the 10 percent version of the dataset . Just put this piece of code after you load the dataset:

for column in dataset.columns:
    if dataset[column].dtype == type(object):
        le = LabelEncoder()
        dataset[column] = le.fit_transform(dataset[column])

After label encoding you should use a One Hot Encoder to improve the performance of some algorithms. You should also avoid using cross_validation module as it is deprecated, it will be removed in version 0.20.


One Hot Label Encoding Scikit_learn convert back to Data Frame

Should i convert it back to a data frame? why not?

  If you have some specific requirements like, saving data in
  a file or want to perform some specific operations which can be run
  better on DataFrame, then its a good choice to convert it back to
  dataframe. Otherwise it should be ok to go with numpy array, even
  Scikit_learn different algo takes numpy array as an input.

what is the best practice to merge X with my 1 numerical feature now ?

  I can share my experience and what exactly I did.

  Save separately and drop the categorical feature and move rest of the features in to numpy array.
  Convert categorical features in to OneHot encoding.
  Concatenate OneHot Encoding numpy array with rest of the features and consume this array for model training.


The n_neurons (LSTM_nums) on the first input layer should match those of n_features or in your case data_dim.

Further reading: LSTM Neural Network for Time Series Prediction


Is there any domain where Spiking Neural Networks outperform other algorithms (non-spiking)?

My answer comes from experience more than from experiments or benchmarks published.

As far as I know, Spiking Neural Networks do not outperform other algorithms in any task. There have been advances in robotics and reservoir computing but  reservoir computing algorithms are as good as other algorithms (like reinforcement learning) according to recent publications.  There are rumours that some companies are interested in these algorithms because they've hired a few reservoir computing researchers recently but these are only rumours.

Here is one of the most recent publications detailing advances and limitations of reservoir computing in robotics
https://www.cs.unm.edu/~afaust/MLPC15_proceedings/MLPC15_paper_Polydoros.pdf

I began experimenting with Liquid State Machines in college using Wolfgang Maass' proposed architecture. It looked promising, specially the idea of inhibitory neurons forming part of the circuit. But in reality using these algorithms in real life data applications (language classification, image classification among others) was not enough to get close to the benchmarks like RNNs, ANNs, SVMs. Sometimes even vanilla Multilayer Neural Networks perform better than Liquid State Machines. My understanding is that these kind of models are good for robotics and other autonomous related tasks like sensors and autonomous navigation (but that was not my area of research) but not so much for other types of data. There are a few labs, mainly in Europe working with this algorithm but so far I haven't heard of many advances in this area in the past years.

I do believe that brain inspired algorithms are the next big step in AI, and while many companies like Numenta and Deepmind are researching towards this direction, as of today there is still a lot of work to be done in order to have the next breakthrough in AI.


How long would it take to become proficient in machine learning for someone with a non-statistical mathematical background?

I think you already know enough applied mathematics to begin with. You can pick-up rest of it as required.

One option is :

Start with an online course that provides high level overview of machine learning and types of algorithms (E.g.: https://www.coursera.org/learn/machine-learning)
Start applying the knowledge in real world problems as soon as possible.
Learn various types of neural networks (deeplearning.ai is one place to start)
Apply knowledge to real world problems (Such as Audio/Video classification , Natural Language)
Get an internship

This will take 5 - 6 months.


Plot RDD data using a pyspark dataframe from csv file

There is no need of two rdd values,

plotdf=df.select(_c3,_c4);
plotdf.plot( style=[])
plt.show()

If you still face the error, please follow the link and set index.


Create a Predictive model to find users that would book an exam

Your problem is a kind of Binary Classification which predicts the probability of a target variable to be Yes(1) or No(0).
But, first of all, you need some ground-truth data, because most of the state-of-the-art solutions to this problem are supervised.

If you're the admin of this system or have access to their server, you can collect these data by logging everything you need from every user and label their traces with 1 and 0 when they book or not.

After a while, you have a big labeled dataset and you can normalize and shape your inputs in a way that it fits for example to a deep neural network for prediction.


Best ML technique to suggest predictor variables

STEP 1

When you do Regression on this dataset, what you obtain is:

$\hspace{60mm}f(x_1,x_2,x_3,x_4) = ERI$

where your $x_1,x_2,x_3,x_4, ERI$ are O2 level, Cylinder Pressure, Fuel Flow,, Engine Temp, and Engine Running Indexrespecetively.
In case of Multiple Regression, you will get 1 coefficient per feature (total 4 in coef_, {$w_1,w_2,w_3,w_4$}) and one intercept_ ($w_0$) term for the intercept. Your equation would then come to look like:

$\hspace{40mm}ERI = w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0$,

where $w_0$ - $w_4$ are fixed values. Your objective then is to find the best $x_1$ - $x_4$ which maximize $ERI$.

STEP 2

The above is better expressed by the optimization problem:

$\max_{(x_1,x_2,x_3,x_4)} w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0$
with constraints that $x_1,x_2,x_3,x_4 \geq 0$

I added those constraints by default as all those variables would be positive only. You could have upper limits on each of them as well. It is important to add these constraints because you don't want any of your value to reach $\infty$ or even some unrealistic values like Engine Temp = $10^6$.

The above function is the objective function which you want to maximize, OR you can minimize the negative of the same. Both of them are equivalent. Thus your objective becomes:

$\min_{(x_1,x_2,x_3,x_4)} -(w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0)$
with constraints that $x_1,x_2,x_3,x_4 \geq 0$

From here, you can use Scipy's Minimization function with the Objective function we just came up with.


I think that you can approach this problem in a better way. To do so you need some Customer Demographics. Even if you try doing some recommendation using customer ID it won't give you ideal Basket/Recommendation, because from Customer ID alone you can't decide anything.

Is there any possibility to get Transaction Id, so that you can unique baskets to apply Aprori Algo/ any association mining techniques by which you can give good recommendation using Lift and Support.

In the time stamp do you have time? As your sample record doesn't show time in it(just for confirmation), Let us consider a scenario you have it. Now ['Celery', 'Beets', 'Cheese'] - transform as Basket-1, ['Celery', 'Bread', 'Cheese'] as Basket-2 and so on... Now you see which Baskets are sold the most WRT timestamp. This is to see if there is any seasonality in the data and you can even find some trends, if there are any.

This is just one way of looking at the problem but this is not the concrete solution.


How does dropout work during testing in neural network?

During training, p neuron activations (usually, p=0.5, so 50%) are dropped. Doing this at the testing stage is not our goal (the goal is to achieve a better generalization). From the other hand, keeping all activations will lead to an input that is unexpected to the network, more precisely, too high (50% higher) input activations for the following layer.

Consider the neurons at the output layer. During training, each neuron usually get activations only from two neurons from the hidden layer (while being connected to four), due to dropout. Now, imagine we finished the training and remove dropout. Now activations of the output neurons will be computed based on four values from the hidden layer. This is likely to put the output neurons in unusual regime, so they will produce too large absolute values, being overexcited.

To avoid this, the trick is to multiply the input connections' weights of the last layer by 1-p (so, by 0.5). Alternatively, one can multiply the outputs of the hidden layer by 1-p, which is basically the same.


Paramaeter estimation in noisy conditions with Machine Learning, possible?

Statistical approach

This question is more related to statistics than data science, perharps you would have better answers on Cross Validated. As you suggested in the question, a purely statistical approach should work for sure (though it requires some assumptions).

As i understand your problem, your prior knowledge is that $\alpha$ (or $\beta$) should be always the same, but you get noise on the realizations of $\alpha$, and you suppose this noise comes from the input ($\theta$), not the process ($f_1$).

Let us assume that the true $\alpha$ corresponds to the mean value of $\theta$, i.e. we have $\alpha_{real} = f_1(\bar{\theta})$. Just taking $\bar{\alpha}$ is generally wrong as you are assimilating $f_1(\bar{\theta})$ with $\overline{f_1(\theta)}$: those are not the same if the problem is not linear.

I suppose you have no idea what the distribution of $\theta$ looks like. But the good news is that you know the $f_1$ and $f_2$ processes, so you can infer the distribution parameters from your data (by maximum likelihood estimation). For instance, you can assume $\theta \sim \mathcal{N}(\mu, \sigma^2)$ (or whatever distribution seems likely), and determine which $\mu$ and $\sigma$ explain the realizations of $\alpha$ and $\beta$ best. Having two different outputs is comfortable, this makes the approach more robust.

Then, your true $\alpha$ and $\beta$ would simply be $f_1(\mu)$ and $f_2(\mu)$.

An incomplete "data-driven" idea

If we make the assumption that $\theta$ is evenly distributed and you have enough samples, its median is the same as the mean. So basically if we find the median, we get a good estimator of the true $\alpha$ and $\beta$ values (again assuming that the true value is the one at the mean of $\theta$).

If either $f_1$ or $f_2$ process is monotonous, this is straightforward, because you just have to sort $\alpha$ or $\beta$ realizations. In the same way, if one process is easily invertible (analytically or numerically), the problem becomes almost trivial. If both are highly non-monotonous and non-invertible, you will need to find a way to sort them in a machine-learning fashion. In the following, I will assume that both processes are at least continuous.

First option: try to perform a binary clustering of your data ($\alpha$, $\beta$). You will need to use the kernel trick, and this will only work if:

you have enough samples to map your processes in a continuous way
there is no "crossing" zone if both processes, i.e. $\forall \theta, \theta^*, [f_1(\theta^*) = f_1(\theta)\ and\ f_2(\theta^*) = f_2(\theta)] \iff \theta = \theta^*$

This may work if your processes are not too "complex".

Second option (which should work better on complex processes): start with the samples in a random order, and compute the euclidian distance (or any other relevant distance) from one sample to the next one, then sum: this gives you the total distance of the pathway going through all samples. Then, use an optimizing procedure (simulated annealing for instance) interverting the index of two samples, until you find an order that minimizes this metric.

This second option is basically solving the travelling salesman problem.

On both cases, I cannot guarantee that it will work, but I think it's worth trying, just for the fun of using standard methods that are not at all suited to the initial problem!


What is the complexity of a recurrent neural network?

I've found some time ago two interesting papers about recurrent neural networks and their complexity. I guess you can use those as a reference points at least:

Architectural Complexity Measures of Recurrent Neural Networks
Bounds on the complexity of recurrent neural network implementations of finite state machines


Scheduler for activation layer parameter using Keras callback

You will need to write a custom callback for this, that implements the on_epoch_end method. Roughly it should look something like this
class CustomCallback(keras.callbacks.Callback):

    def __init__(self, freq):
        super().__init__()
        self.freq = freq   # how often to change the parameter

    def on_epoch_end(self, epoch):

        if epoch % freq == 0 and epoch > 0:

            weights = self.model.get_weights()

            # here you change the weight you want, e.g. it is the 5th layer
            weights[4] = weights[4] / 10

            self.model.set_weights(weights)


Architecture for multivariate multi-time-series model where some features are TS specific and some features are global

Stacked LSTM is one option in this scenario

This assumes that First two LSTMs have different frequencies and City has static features (Like lat/long, one-hot-encoded value etc). If City is also time-series like series of population , mean income; it will be an LSTM as well.
Code example for stacked LSTM.


There are a variety of techniques that you could use, depending on what you would like to do.
If your goal is to gain insight into the phrases that are being used in each group, then I'd recommend looking for the most frequent N-grams of different lengths that appear in each class. Here is a related stackoverflow question that shows how you can use nltk and sklearn to extract these.
If your goal is to predict the outcome (accept/reject) given a phrase, then I'd recommend setting this up as a binary classification problem. Since those phrases are quite short, you could start with a Bag of Words approach - the scikit-learn documentation for working with text data is a good example that guides you through the steps.