How to do give input to CNN when doing a text processing?

So, here the question is asking how you use text in a CNN architecture.
Just like when we process images with a CNN, the text data will also be a 2D matrix, where the rows will represent the text features and the columns the sequence of characters (which make up a name) for example.
Now, of course, if we simply use one-hot encoding, we get data sparsity, which is not not particularly computational efficient, especially when using CNNs. So, preferably, we would use word embeddings, which you have described in your post. Here, word embeddings aim to collapse the high dimensionality of the input so that reduces the chance of data sparsity.
However, there is a potential problem with using word embeddings in your problem. The principal behind word embeddings is that it will assign similar word embeddings to words which can be easily replaced in similar contexts (e.g. Sam is kind & Simon is kind) and that don't affect the coherence of a sentence. As you can see, when it comes to proper nouns, we might end up with similar vectors for all names since they are very interchangeable in any context and do not affect the overall coherence of the sentence.
Therefore, although this is contradictory, I would advise using one-hot encoded sequence of characters as a start point for this problem and then when you implement this, you might come up with a more effective way to represent the names.


Can feature importance change a lot between models?

Is this normal?

It is not surprising.

First, you are using different measures of feature importance. It’s like measuring the importance of people (or simply sorting them) using their a) weight, b) height, c) wealth and d) IQ. With a and b you might get quite similar results, but these results are likely to be different from results obtained with c and d.

Second, the performance of your models is likely to be different. In extreme case, output of one of your models could be completely rubbish (in your case it is more likely to be NB). Then the feature importance metrics produced with such model is not credible. In less extreme scenarios when the difference in models‘ performance is not so dramatic the trustworthiness of importances produced by two different models is more comparable. Still the importances might be quite different due to the first argument, i.e. different language used to capture the importance.

Edit:

You have not asked about it in your question, but there are feature importance  approaches which are model agnostic, i.e. can be applied to any predictive model. For example, check the permutation importance approach, described in chapter "15.3.2 Variable Importance" in The Elements of Statistical Learning


Text annotating process, quality vs quantity?

It depends on the context which you consider. For example, suppose there is a situation that all possible states can be covered by 10K different texts. It is trivial, if these texts are all annotated, then for 1000 tests, at least we can classify 500 of them truly (as we have two classes, and probability of wrong annotation for each text is at most 0.5).

Now, suppose 1K of 10K texts are annotated. Then, as annotations are exact, we can classify 1/10 of 1000 texts truly (because we have no idea about the other 9K possible states).

Therefore, in this situation quantity is more important than quality.

Also, we can consider these cases, when the possible states are 1K. It could be straightforward to show that in this case (if the power of annotators is same as the former case) quality can be more important than quantity. However, in the most cases this number is not realistic.

In sum, as in the most cases the variety of texts are more than the power of annotators, we prefer quantity to quality, as we can cover more text space and machine can learn more. Although the accuracy can be less, but for two classes classification is negligible.


Pytorch Luong global attention: what is the shape of the alignment vector supposed to be?

You answered yourself [sequence length, 1] is correct assuming you work with a single sentence. (Or actually, the 1 dimension depends on implementation.)
In practice, the data is typically batched, so it will be [batch, sequence length 1]. This can be element-wise multiplied with the encoder states of dimension [batch, sequence length, hidden size] and summed in the middle dimension to get a context vector of dimension [batch, hidden_size].
Note also that if the sentences do not have the same length, you need to deal with padding, i.e., set the padded positions to -inf before softmax.


Finding the relation between two dimensions in a multi-dimensional problem

You might take a look at canonical correlation analysis. It tries to find correlation between two sets of data. I guess you could adapt it to explore your data.


Solutions for Labelling Training Data for Binary Classification Problems

Ofcourse not. Here is a simple possible solution.
Do unsupervised learning. If you do it good and efficiently you will only see these two groups in your data (binary classification). And your silhuette score will be high. Hence you can automatically than label these groups/clusters.


You can make the plots by looping over the groups from groupby...
Or this should also work..
import matplotlib
import matplotlib.pyplot as plt
df['company_score'].groupby(df['company_id']).rolling(10).mean().unstack().plot()

See this for more information on plotting with pandas dataframes and this for looping over a groupby-object. Finally this link is super helpful.


Keras: Provide One-Hot-Encoded input values to neural network

what I understand from your code is you are fitting a one-hot encoder on your training set, which may not include all words that appear in your test set. So when you get a new word in your evaluation method, your transformer cannot hash it, and hence throw an error.
the easiest way to solve this would be to use the unknown_error argument in one hot encoder
def prepare_targets(y_train, y_test):
  ohe = OneHotEncoder(handle_unknown='ignore')
  ohe.fit(y_train)
  y_train_enc = ohe.transform(y_train)
  y_test_enc = ohe.transform(y_test)
  return y_train_enc, y_test_enc

now when you encounter a new word in the test set your encoder will output an array of all zeros, and if your try inverse transforms on all zeros, you will get none.
The downside is obvious here, your Neural Network does not what to do for zero vector as it has never seen such an example in training. So expect a weird response when a new word is evaluated.
The smart way of solving this problem would be to assume there are some unknown words in the training set by synthesizing few rare (unique) words as 'unknown' (replace words with the word 'unknown') and then do the one-hot encoding. while evaluating you will then first check if the word is not part of words set in training data, then you will replace that word with 'unknown'.


what ETL technique should i use for text documents using Hadoop?

I would suggest a data fabric. That would meet your need for data acquisition, preprocessing, data quality, master data management, etc.

Given I work for Talend, I would suggest our data fabric. =)

Here’s a case study with the Panama Papers. https://www.talend.com/blog/2017/01/17/talend-data-masters-2016-icij-decoded-panama-papers-talend/

The concept in that case study of combining a data fabric with analytic tools is the general concept to use regardless of which data fabric you use.

You can find the trial and open versions of Talend at https://www.talend.com/download/.

Edit: Here’s another example, which implements a backend and UI showing the ebooks of the Gutenberg project. It allows to import the whole gutenberg index using a camel route. https://github.com/Talend/tesb-rt-se/tree/master/examples/tesb/ebook


Using different timesteps for training data and target value

is it ok to use daily prices?

You can use the daily prices as time series to forecast an arbitrary horizon.

  LSTM vs NN

In short, vanilla NN have no memory of previous inputs whereas LSTM NN have. This is particularly useful in your problem because LSTM can learn the temporal (sequential) characteristics of your dataset and therefore forecast several steps ahead. LSTM RNN can learn the dynamics of your problem (vanilla NN cannot). A useful Python/Keras tutorial for your problem can be found here.

  Regression vs Classification

In short, classification is the problem of discriminating a sample into discrete categories. Regression refers to using past values of a signal (or more) to estimate future values of the same (or other) signals. In your case, regression is what you ask for.


How do I choose number of clusters when Eigengap heuristic suggest 1 for spectral clustering?

The first eigenvalue in your plot is non-zero, whereas the last is one.
This seems to suggest that the Laplacian whose eigenvalues you are computing is different from the standard Laplacian $ L = D - W$, where $D$ is the matrix whose diagonal entries are degrees of nodes and $W$ is the weighted adjacency matrix.
So in your plot, $\lambda_{19} = 1$ and $\lambda_{18} \neq 1$ but is $\approx 1$  which is indicative of the graph being one connected component.
So the correct way to interpret your plot would be to invert it ($\lambda \mapsto 1 - \lambda$) and then arrange them in increasing order, or to arrange them in decreasing order as-is.
Then you can use the eigengap heuristic to choose a suitable value for the number of clusters (7 seems to be a good choice as $\lambda_{13}-\lambda_{12}$ is large compared to $\lambda_{14}-\lambda_{13}$, or maybe 3 if the number of clusters is a constraint).
This is described in greater detail in Luxborg's excellent tutorial on Spectral Clustering, and you can find a footnote in Ng et al's seminal paper on the same.


How do i iterate functions until convergence in R?

Generally you want to check that the change in your value of interest becomes very small, eg $abs(X_n - X_{n-1}) < eps$ where eps is a small number like 0.0001.


In "Attention Is All You Need", why are the FFNs in (2) the same as two convolutions with kernel size 1?

Convolution is actually applying a sliding window projection over a tensor. With the kernel size of 1, the convolution reduces a to a for loop over the vectors, i.e., each of the vectors gets multiplied by a filter matrix. But this is actually a definition of matrix multiplication which is how feed-forward layers are implemented.


How big does a Neural Net need to be to learn anything

The size of the Neural Network is usually treated as a hyper-parameter. You need to estimate it from a separate cross-validation set.

For Neural nets there are no theoretical proven bounds (there are some papers here and there, but they are not applicable in practice).

You need to use the good old "cross-validation" method to search for the best fit value of the size of NN.

P.S. : There is a paper from Yoshua Bengio with all trick and traits for training such machines.

https://arxiv.org/abs/1206.5533


Without labels for which data points are anomalies, you have no way to know how many anomalies you should get.

The closest thing to adding anomalies to the dataset is using synthetic data for anomaly detection. I have seen some practitioners doing this, and failing. When you create the synthetic data, it matches your inductive bias*. The real world very rarely matches that inductive bias*. Afterwards, you can easily find a clustering algorithm with the same inductive bias* which can flag your synthetic data as anomalies. However, when that algorithm is deployed it will fail miserably.

*in plain language, inductive bias is the assumptions you have of how the world should be.


Why do we only care about convex functions when doing Gradient Descent/SGD?

Probably in the 1-dimensional case this more precise class of functions you aim at would be something like:

Piece-wise monotonous (thus approachable with gradient descent locally) with up to one flip (thus you know that one optimum would be where the pieces meet, the other would be at the extremes)

However, even if one is able to come up with such a definition for the simpler space of 1-dimensional functions, it becomes much more complicated to come up with a generalisation that would be valid in the more generic multidimensional case.
Despite this difficulty, the effort still might be worthwhile if there's going to be a distinctive benefit of doing it. In this particular moment, I personally wonder if this might be the case. Surely, it depends on the context of your problem. Wouldn't a possible approach be to break down your problem into steps:

Try to break down your surfaces/functions in piecewise convex
Apply gradient descent on the pieces
Compare the optima that you have identified


How do I calculate the accuracy for graph mining in terms of (top 1%)?

You have a mining problem which is basically a search problem. In search problem you have TP, FP, FN but not TN (you have it but it is trivial, imagine search in google. Basically bilions of webpages are TN in every search. right?)
I also assume that you are not interested in ranking results as you did not mention it in the question.
In this case you can easily go with Precision and recall:

TP: Graphs which are right and you could find.
FP: Graphs which are not right but your model found.
FN: Graphs which are right but your model did not find.

Then you simply calculate:
Precision: $$\frac{TP}{TP+FP}$$
Recall: $$\frac{TP}{TP+FN}$$
And as a combined score, you may calculate F1:
$$\frac{2TP}{2TP+FP+FN}$$
About Accuracy
When the volume of search space is huge, as I said above, accuray is not meaning much. According to the formulation:
$$ACC=\frac{TP+TN}{TP+TN+FP+FN}$$
when $TN$ is very large then the whole fraction approaches zero and does not inform much.
But if your search space is relatively small, you can calculate accuracy with the formulation above. However it was just a theoretical justification and still your best metrics are precision and recall and their related metrics.
Update For Your Specific Question
Your set of top 1% results may have different sizes due to different size of your model output (If it is always 4200 please write in the comments but in "mining" questions usually the size of output is variant).
In your case TP and FP are pretty straight-forward and meaningful. About FN, if your test set is also ranked, then you can evaluate your results according to top 1% of your ground-truth which includes 90 samples and go on as you already did, or you can fix the size of ground-truth set equal to your 1% (i.e. take 42 top test results) and again calculate same scores. The first one is more stable and significant for comparing different models.
A more precise metric which is related to the ranked results is called NDCG.
If the test set is not ranked then go this way:

$|correctly\_found\_graphs|$: Number of graphs that you correctly found.
$|wrongly\_found\_graphs|$: Number of graphs you found but they are not in ground-truth
$|all\_found\_graphs|$: here 42
$|all\_correct\_graphs|$: here 900

Then calculate Precision@k (here precision@42) metric which calculates the precision only on the top k results of a search output:
$Precision@42 = \frac{|correctly\_found\_graphs|}{|all\_found\_graphs|}=\frac{38}{42}$
This is a well-established information retrieval metric.
$Recall = \frac{|correctly\_found\_graphs|}{|all\_correct\_graphs|}=\frac{38}{900}$
How can I improve the recall?
As long as size of your top 1% results is less than ground-truth, your recall is limited to an upper-bound i.e. even if all 42 all relevant, still your recall is 42/900. Knowing this fact you can use recall but I do not see the point. Recall makes more sense if you take all 4200 outputs into account.
Your calculation of True Negative in the comments is also wrong. TN in your case is the number of all graphs that your model should not have returned and it didn't i.e. 900-42
Moreover, all samples are positive in my test dataset. Not single sample is negatives
See the problem from an Information Retrieval point of view. In this approach, any graph which is not in 900 set is a negative sample.
You can comment in case it is still unclear.


Explained definition of the norm in Ordinary Least Squares

The norm indicates the Eucleadan norm, which gives the ordinary distance between the points.
https://en.wikipedia.org/wiki/Norm_(mathematics)

The subscript 2 with norm sign indicates "square root of the sum of squares".
The superscript 2 indicates sign power.


Counting the number of trainable parameters in a gradient boosted tree

Maybe that can be the size of your grid, but not the number of possible trainable  hyperparameter of the Gradient Boosting Regressor of scikit learn.
I add some more hyperparameters of GBDT Regressor:

loss{‘ls’, ‘lad’, ‘huber’, ‘quantile’},
learning_rate
criterion{‘friedman_mse’, ‘mse’, ‘mae’}
max_features{‘auto’, ‘sqrt’, ‘log2’},

This is just some of the hyperparameters needed, this is extracted from the documentation
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html


As I understand it, you don't want to use the dot product, because it is unbounded.  Yet you also don't want to use the cosine similarity metric:

, which is just the dot product normalized by the length of the two vectors being dotted because it does not take magnitude into account e.g. $(1,1)\cdot(1,1)=1=(1,1)\cdot(2,2)$

Perhaps you could therefor use the sorensen dice coefficient or possibly 1-soresen:

This varies from 0 to 1 and accounts for magnitudes of the constituent vectors e.g.:

$$s_{\nu}[(1,1),(1,1)]=1$$

, where as

$$s_{\nu}[(1,1),(2,2)]=\frac{4}{5}$$

Hope this helps!


Find the order of importance of random variables in their ability to explain a variance of Y

Decision trees are by nature immune to multi-collinearity. So by that principal so is Random Forest. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features. - follow this link to know more.

For other models, solutions can be many, but here are the popular choices -

Regularization

This is an automatic way of handling high correlations in your Xs. It decreases the value of the co-efficients by penalizing them against the loss function and introduces bias in the system. Popular implementations are L1, L2 and Elastic Net(in case of linear and logistic models), Pruning(in case of Decision Trees), Dropout and max pooling in case of deep learning.

Almost all sklearn implementations support a regularization parameter in 1 way or the other. Depends on the algorithm you are using.

Using VIF to detect multi-collinearity

Variance Inflation Factor is a metric that you can use with all your Xs and then start dropping variables with the highest VIF one by one (usually the variable with the highest VIF is dropped and then the exercise is repeated till there is no multi-collinearity left or domain experts come in and choose what important variables to keep/drop)

Principal Component Analysis

Although the model might lose its interpretability (you won't be able to tell exactly what feature is how important), PCA has been another effective way of removing multi-collinearity. Basically it projects the data in a way that the output columns are orthogonal (independent)

Some Examples

Ridge Regression

 from sklearn.linear_model import Ridge
 import numpy as np
 n_samples, n_features = 10, 5
 np.random.seed(0)
 y = np.random.randn(n_samples)
 X = np.random.randn(n_samples, n_features)
 clf = Ridge(alpha=1.0)
 clf.fit(X, y)

Lasso Regression

 from sklearn import linear_model
 clf = linear_model.Lasso(alpha=0.1)
 clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])

Elastic Net

 from sklearn.linear_model import ElasticNet
 from sklearn.datasets import make_regression

 X, y = make_regression(n_features=2, random_state=0)
 regr = ElasticNet(random_state=0)
 regr.fit(X, y)

Random Forest
 - Almost all parameters in scikit-learn can account for regularization here

Statsmodels has a rather good implementation of VIF

PCA

import numpy as np
from sklearn.decomposition import PCA
from sklearn.svm import SVC

X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
pca = PCA(n_components=2) # adjust yourself
pca.fit(X)

X_t_train = pca.transform(X)
clf = SVC()
clf.fit(X_t_train, y)


How to merge personalized models together

Feature engineering is the name of the game when it comes to this cases. I stumbled upon a similar problem a few years ago and it can be baffling to have a model nor generalize well for all cases. However, one model for one user is never the way to go, after all you, in many cases have only one data point for that user in particular. Additionally, you will never be able to make predictions if a new user appears.

Therefore, you need to use one model for as many as possible users. There are many more approaches but these are the ones worth considering  in my opinion.

You can cluster users by location, city, neighborhood, region. Usually people in the same areas frequent the same places. If one model is incapable of generalization for all possible users then splitting the data into similar clusters is an approach.
In the case a model can generalize well for most cases but there is a big cluster when it cannot generalize, use a second model for the hard cases.
Basic feature engineering: users living nearby are going to go to similar places based on their current location and are going to use the same ways. There is a lot of information similar users will share. Use starting location as a feature as well.
Advanced feature engineering: Use old targets from the same users along with time features in a meaningful way as training features to predict future targets.
Use users ids: user ids can be used to predict targets in modern machine learning algorithms, sparse matrices do the trick, think of them as NLP words. Some might be useful others don't but let the algorithm do its work. Search for algorithms that support sparse matrices if you take this approach.
Research GIS (Geographic Information System) forecast: there are other tricks and feature engineering methods in that field that are not applicable for other ML problems which are useful for forecasting these kind of problems.
Try different targets. Longitude - -latitude is often not possible but a raw estimate of the region can be done. You can treat the targets as a multilabel preferably (not multiclass) classification with each region/neighborhood/block be one label. Remember, algorithms can handle many targets like in the case of image classification, research those approaches.
Consider simpler methods if you don't have enough data: If user Bob goes to spot A every Monday then probably Bob will be at A the next Monday as well.
Use dates/days of the week/holidays as features. People change transit behaviors during holidays.

tl.dr. Find a way to use many users for a model that generalizes well because training one model for one person never works due to users having often just one point of data. Because, all algorithms need massive amounts of data to generalize well.


clustering people according to answers on survey

Maybe you can

assign numeric values to the response of each question (e.x. 0 disagree, 1 for neutral, 2 for agree) if there is a natural order in them,
perform one hot encoding for the rest where no such order exist (ex. country of origin)

and then try your luck with a classical algorithm like k-means or hierarchical clustering. There exist a lot of bibliography against that approach but maybe you can create clusters that make sense to your case.


Are there any open-source text annotation for multi label classification tools?

There are many manual text annotation tools available, but you will probably have to search around in order to find the one which suits your precise needs.
Here are a few pointers:

Gate
Text annotation tools
Doccano
a recent review


GTX 1660 Ti vs. RTX 2060 for a deep learning pc

I would have gone with the RTX.
If I was to design deep learning models,I would be prefer more number of cores in the GPU , where RTX is the winner.

Lets not forget the Tensor cores which are present in RTX , which makes it stand out for Deep Learning processing. (Faster Tensor operations).

Go with RTX if you can afford it.


You've not provided the dataset but I will try to answer based on your descriptions.

  how many (multiple of 200?) rows do I choose for train/test sets?

Actually, the train/validation/test splits do not really depend on the length of the time series. It is more dependent on the number of examples, time series here. $300k / 200 = 1.5k$ which means the size of your dataset is small and you can use the customary percentages for your splits. Something like 60/20/20 for train/validation/test.

  Is my batch size 200?

$200$ is not your batch size, it is the length of your signal. In LSTM networks, you usually deal with temporal data. For each sample, you have 12 features for each time step. $200$ means each sample has $200$ steps. In vectorise implementations of LSTMs it is customary to define batch size. That is, you stack $m = batch size$ samples for acceleration in training, it has details that I skip.

  What is the basis for choosing number of neurons?

Take a look at here. Although the details of MLPs are described, they can be generalised to LSTMs too. Consider RNNs like simple MLPs which take the inputs of the current time step and the outputs of the previous step.


Tensorflow dataset batching for complex data

From what I can understand from your code, it seems like you need to use the initializable iterator. Here is why:

Your are creating a dataset from a placeholder.

Here is my solution:

batch_size = 100
handle_mix = tf.placeholder(tf.float64, shape=[])
handle_src0 = tf.placeholder(tf.float64, shape=[])
handle_src1 = tf.placeholder(tf.float64, shape=[])
handle_src2 = tf.placeholder(tf.float64, shape=[])
handle_src3 = tf.placeholder(tf.float64, shape=[])

PASS A TUPLE TO  .from_tensor_slices method of the tf.data.Dataset class

dataset = tf.data.Dataset.from_tensor_slices((handle_mix, handle_src0,
                                               handle_src1, handle_src2,
                                               handle_src3))

dataset = dataset.shuffle(1000).repeat().batch(batch_size)

iter = dataset.make_initializable_iterator()

# unpack five values since dataset was created from five placeholders
a, b, c, d, e = iter.get_next()

create a session object and initialize the iter making sure to pass "your data" to the placeholders. 'your data' can be numpy arrays

sess = tf.Session()
sess.run(iter.initializer, feed_dict={handle_mix:'your data',
         handle_src0:'your data',handle_src1:'your data',
         handle_src2:'your data',handle_src3:'your data'})

run the following to print the contents from the iterator

print "%r, %r, %r, %r, %r" % sess.run([a,b,c,d,e])

relevant details or clarification are available from the official TensorFlow documentation on how to use the dataset API.


Why not always use the ADAM optimization technique?

Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM.
There is often a value to using more than one method (an ensemble), because every method has a weakness.


Thesis on avalanche prediction using One Class SVM

Your problem seems to belong to novelty detection in the general area of OCC problems.

So, the short answer is: yes. You can apply SVDD (Support Vector Data Description) method to get the smallest hypersphere containing samples in the dataset and then assess whether a new observation is an outlier or not.

Of course, the less representative your dataset is, the less accurate your classifier will be.


Which one is better performer on wrangling big data, R or Python?

I think this may help you:
Is the R language suitable for Big Data

I think it really depends on what you are comfortable with and what your objectives are.  In Python, I can execute SQL queries directly into Pandas data frames.  From there, data cleaning and visualizing (I like seaborn) is fairly straightforward.  I can easily perform matrix manipulations using Numpy, and the sklearn is a well documented ML package.

That being said, I still like R and there are definitely times when I will use R.  Learn both!


In fact, this means that the quality of the feature f1 is very high. Usually, you should be worried if you get all the features giving the same importance level, and that level is low.

It is very important to understand that random forest does two level of randomization: at the data level and the feature level and it is very difficult to over fit.

Since you have this large number of trees in the forest, I do not think you have over fitting problem.


Calculating the Number of Parameters of a 2D CNN Layer

When we say the filter size if 5x5 it is for an image with 1 input channel, for three the filter size is 5x5x3 (but at a lot of places this additional info is skipped to make things easy to understand). When you apply a kernel of 5x5x3 to an image the output is just one channel. To get an output of 8 channels you need 8 such kernels. In that case, the number of parameters would be 5x5x3x8.


What are the advantages of contrastive divergence vs the gradient of the quadratic difference between the original data and the reconstructed data?

It is natural that RBM requires some time to converge, but if you see that the reconstruction is decent, then your algorithm works well.

I don't understand the idea of optimizing the model by your objective function:
cost = T.sum((data - vis)**2).mean().
I think you just cannot train RBM (or any other stochastic model) that way because vis is a nonlinear function of the data. If you omit sampling of the hidden vector you might use this formula and end up having model similar to the feed forward network that learns an identity transformation. It fact it can be used as a feature extractor.

Also notice that your second network no longer will be a proper generative model, because its learning rule doesn't follow the gradient of RBM's cost function. Hence it won't properly model either an energy function or a joint probability P(v, h) of your visible and hidden vector.

You may also be interested in reading this:

  Although it is convenient, the reconstruction error is actually a very poor measure of the progress of learning.  It is not the function that CD learning is approximately optimizing, [...] and it systematically confounds two different quantities that are changing during the learning.  The
  frst  is  the  difference  between  the  empirical  distribution  of  the  training  data  and  the  equilibrium
  distribution of the RBM. The second is the mixing rate of the alternating Gibbs Markov chain.  If
  the mixing rate is very low, the reconstruction error will be very small even when the distributions of
  the data and the model are very different.  As the weights increase the mixing rate falls, so decreases
  in reconstruction error do not necessarily mean that the model is improving and, conversely, small
  increases do not necessarily mean the model is getting worse.

So what is the cost function that RBM tries to optimize? good description can be found here
$$-\frac{\partial log(p(x))}{\partial \theta }=\frac{\partial F(x)}{\partial \theta}-\sum_{\tilde{x}}p(\tilde{x})\frac{\partial F(\tilde{x})}{\partial \theta}$$
On the left hand side we have minus log probability of the data which we must minimize in order to increase the probability of the data. On the right hand side we have two terms. The function $F(x)$ coresponds to the free energy of the network given the data $x$ and current weights $\theta$ so minimization of this function increases the probability of our data. On the other hand we must take into account the normalizing factor $Z$- its role is to give correct probability distribution of the input vectors and this sum in the equation corresponds to the normalizing factor. In other words we must maximize the free energy (decrease probability) over all set of inputs proportionally to its current probability $p(\tilde{x})$. If we omit to maximize this sum our model won't converge to the optimal solution. Contrastive Divergence is only crude approximation of this objective funtion but it works very well in practice. I recommend that you watch this film by Geoffrey Hinton


How to convert sequence of words in to numbers which are input to RNN/LSTM?

from keras.preprocessing import Tokenizer
samples = ["grss is green and sun is hot"]
tokenizer = Tokenizer(num_words=1000)
tokenizer.fit_on_texts(samples)
sequences = tokenizer.texts_to_sequences(samples)

The Keras library uses it's tokenizer function but you have other well known libraries like nltk, gensim to convert them into sequences and pass it into your neural network. There are other ways like TF/IDF and CountVectorizer in Sklearn for more simpler algorithms. num_words takes the most 1000 frequent words and tokenizes them.

Link : Keras text processing


How to use unigram and bigram as an feature on SVM or logistic regression

You need to create a vocabulary of the n-grams, i.e., a numbered inventory of bigrams that you are going to use as features. Typically, these are the most frequent ones. When you create the feature vector, you start with a zero vector and put one (or add one) if the n-gram with the corresponding index appears is in your sentence.
Machine learning libraries typically have functions that do that. For instance, in scikit-learn, you can use CountVectorizer to do the job. The fit method has an ngram_range argument that controls the length of n-grams that are considered in the feature vectors.


Multivariate linear regression in Python

You can still use sklearn.linear_model.LinearRegression.  Simply make the output y a matrix with as many columns as you have dependent variables.  If you want something non-linear, you can try different basis functions, use polynomial features, or use a different method for regression (like a NN).


Is it legal to scrape YouTube videos for training data?

It will depend on the rights of the videos themselves, although probably the terms of service of youtube wouldn't agree with it anyway.

But you have the YouTube 8M dataset, released by Google for research purposes.

  YouTube-8M is a large-scale labeled video dataset that consists of millions of YouTube video IDs and associated labels from a diverse vocabulary of 4700+ visual entities. It comes with precomputed state-of-the-art audio-visual features from billions of frames and audio segments, designed to fit on a single hard disk. This makes it possible to get started on this dataset by training a baseline video model in less than a day on a single machine! At the same time, the dataset's scale and diversity can enable deep exploration of complex audio-visual models that can take weeks to train even in a distributed fashion.

Edit: And Facebook and MIT just released SLAC (dataset to be released soon, as well as pretrained models for transfer learning).

  This project presents a novel video dataset, named SLAC (Sparsely Labeled ACtions), for action recognition and localization. It consists of over 520K untrimmed videos and 1.75M clip annotations spanning 200 action categories.


Your question is way to vague. What did you try, what do you want to achieve in detail. However, find below some R examples how to approach topic modeling. This book might also be helpful: Text Mining with R.

library(topicmodels)
data("AssociatedPress")
AssociatedPress

# LDA (Latent Dirichlet Allocation)
ap_lda <- LDA(AssociatedPress, k = 2, control = list(seed = 1234))
ap_lda

# Get "topics"
library(tidytext)
ap_topics <- tidy(ap_lda, matrix = "beta")
ap_topics

library(ggplot2)
library(dplyr)

# Plot topics
ap_top_terms <- ap_topics %>%
  group_by(topic) %>%
  top_n(10, beta) %>%
  ungroup() %>%
  arrange(topic, -beta)

ap_top_terms %>%
  mutate(term = reorder(term, beta)) %>%
  ggplot(aes(term, beta, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free") +
  coord_flip()

# (words with) Greatest difference between categories
library(tidyr)

beta_spread <- ap_topics %>%
  mutate(topic = paste0("topic", topic)) %>%
  spread(topic, beta) %>%
  filter(topic1 > .001 | topic2 > .001) %>%
  mutate(log_ratio = log2(topic2 / topic1))

beta_spread

list(beta_spread$term)

# Select 10 largest and smallest values of "log_ratio")
df1 <- subset(beta_spread, beta_spread$log_ratio<=max(tail(sort(beta_spread$log_ratio, decreasing = T),10)))
df2 <- subset(beta_spread, beta_spread$log_ratio>=min(tail(sort(beta_spread$log_ratio, decreasing = F),10)))
df = rbind(df1,df2)
df

# Plot
barplot(sort(df$log_ratio), names.arg=df$term[order(df$log_ratio)], las=2)

# Gamma contains the probability of document X to belong to topic Y
ap_documents <- tidy(ap_lda, matrix = "gamma")
ap_documents
# the model estimates that only about 24.8% of the words in document 1 were generated from topic 1


Which feature to use in feature selection?

My guess would be that the features are highly correlated. Check this. Regarding feature selection, I suggest starting with Logit and Lasso (L1 regulation). This method (Lasso, l1) can „shrink“ features to zero (so kick them out basically). This happens automatically based on feature importance.

Many methods (e.g. Boosting, Neural Nets) allow you to use L1 regularization. So there may be no need to „manually“ select features.

If you want to do that manually, you may use stepwise feature selection, as e.g. described in Introduction to Statistical Learning, Chapter 6.

Here is some Python code from the chapter: https://github.com/JWarmenhoven/ISLR-python

You may also have a quick look at this post related to Lasso: https://datascience.stackexchange.com/a/55702/71442


How to create interactive plot of thousands of images as output of t-SNE?

It is certainly possible to create interactive plots of many thousands of images, as Google has done in their interactive art t-SNE Map.

However, as far as I have found, there is not any canned way to do this.  https://github.com/YaleDHLab/pix-plot is an open source application that also does an impressive job at plotting thousands of images, and if you are using python (e.g. with scikit-learn's t-SNE), it does not look too bad to modify to your use case.


Is it acceptable to select a random child node when using a Decision Tree (trained via ID3) to predict if an unknown attribute value is encountered

Do your leaf nodes return a probability distribution rather than a single class value e.g., the majority class of training instances that arrived at the node? If so, a better approach would be to send the test instance down every child path and average the probability distributions that are returned by the leaves.

Your good score of 95.45% on iris data suggests to me that it's not a very important detail in this example. Try some more challenging datasets and see if it makes a difference.


Flattening output before calculating metrics

You need to evaluate the percentage of pixels in the output map which were correctly classified, and it doesn't matter whether it is represented as 2D or 1D. The demand is just that mask's and the ground truth pixels have to match each other.

You can find an example about Pixel Accuracy here.

So, if you would like, you can convert (stretch 2D) into a 1D vector. However, you can also implement computation of precision and recall by yourself. If you use python, it's easy to leverage construction with np.sum(mask), np.sum(target), or np.sum(mask*target) to calculate it for 2D:

TP = np.sum(mask*target)
FP = np.sum(mask*np.where(target == 0, 1, 0))
FN = np.sum(np.where(mask == 0, 1, 0)*target)
Precision = TP / ( TP + FP )
Recall = TP / ( TP + FN )
F1_measure = 2 * Precision * Recall /(Precision+Recall)


What kind of research I can do with my data set?

There are several things that can be done, however as @Dawny33 mentioned, start with the problem you want solved.

As an example, If I was trying to solve: Which factors, influence the prices and how?

A simple analysis could be to run a linear regression model (lm) with PriceSqFt as a dependent variable and others as regressors. You can then select top 10 or 15 or how so ever many you like and look at their relationship.

This may help you answer some simple questions like:

How do prices vary by dealers, by property type, by location, by size?
Does move in ready date make a difference to rent / prices?
Do specific kind of properties have later than usual ready dates?
Is there a negative or positive correlation between size and pricesqft?
Are certain dealers over or under priced compared to other identical
properties and location?

You can also create a plots using pairs or other functions to look at pairwise relationship for initial analysis and building on top of what you see.

If you really want to take it a to the next level, you can create a prediction model to predict the price based on select parameters.


Adam works in the same way as SGD does in this regard, it updates the weights at the end of each iteration, so at the end of an epoch multiple weight updates have been applied.
Inherently neither Adam nor SGD do anything to counteract the noisy labels, they just try to find the best parameters that minimize a loss function. I don't think anyone can answer apriori if it will be better to use Adam or SGD for your problem.


Clustering algorithms based on lists of most similar items?

You can go for K-means clustering to cluster the data based on the pattern.


Fast introduction to deep learning in Python, with advanced math and some machine learning backgrounds, but not much Python experience

Stanfords CS231n has a great python+numpy tutorial.
http://cs231n.github.io/python-numpy-tutorial/

As for Neural Networks and Keras + Tensorflow, I can recommend the Deep Learning specialization on Coursera. It is free for one week. If you do not want the certificates you can download all videos and stop.

If you want the certficates you can enroll in all courses simultaneously and finish them faster than advertised. They assume 3-6h per week for 16 weeks. If you have 6h per day, it is possible to go through a weeks content per day. Andrew Ng speaks clearly and the videos are understandable at 1.5x-2x speed.

As for the roadmap, i would suggest to

start with python + numpy until you have a basic understanding of array slicing and read up on unknown commands when you encounter them.
start a DL course of your choice but do not skip classes. That is I think it is important to know the basics even if you will only use high level frameworks later on.


One thing you could try is to use a "normal" Logit, as it is computationally not very expensive. Using Lasso, Elastic Net, or Ridge can yield good results (often similar to boosting) shown here.

Here is a recent code example for Logit/Lasso.

If you would like to stick with boosting, you may check LightGBM as it tends to be faster. XGBoost tends to be "heavy" in terms of data handling. LightGBM (as the name indicates) aims at resolving this problem and I have had good experiences so far.

Here is the link to the the LightGBM docs.


The below piece of code will generate the following Image(your's is Subplotting Three of them, so you will get 3 different axe's and per axes you have to use fill-between) (Kindly ignore the Axis Label's..)

plt.style.use('ggplot') #Change/Remove This If you Want

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(trees_grid, train_acc.mean(axis=1), alpha=0.5, color='blue', label='train', linewidth = 4.0)
ax.plot(trees_grid, test_acc.mean(axis=1), alpha=0.5, color='red', label='cv', linewidth = 1.0)
ax.fill_between(trees_grid, test_acc.mean(axis=1) - test_acc.std(axis=1), test_acc.mean(axis=1) + test_acc.std(axis=1), color='#888888', alpha=0.4)
ax.fill_between(trees_grid, test_acc.mean(axis=1) - 2*test_acc.std(axis=1), test_acc.mean(axis=1) + 2*test_acc.std(axis=1), color='#888888', alpha=0.2)
ax.legend(loc='best')
ax.set_ylim([0.88,1.02])
ax.set_ylabel("Accuracy")
ax.set_xlabel("N_estimators")


Given a machine learning algorithm, what is the minimum size of the training set for it?

You can have a model with a single training example.  The real question should be, how good is your model with only a single training example. The answer? A learning algorithm $h(\theta)$ is an approximation of the actual relationship between your features and your target. However, the performance on how good $h(\theta)$ relies on good data. Your model is only as good as your data. If your data shows clear patterns, you can have a good model with a few data points. That's why Exploratory Data Analysis is essential.

I'd recommend watching Statistical Learning theory on youtube, specifically the lecture "Is learning feasible". The class is offered by Caltech.


R: substituting only whole statements (not parts of them)

I believe it will help if you add the exact regex anchors to the pattern you are trying to match. In your case adding "^I like football$" as the pattern you are trying to match will look for strings starting and ending exactly like that. That way you have:

>grepl("^I like football$", "I like football, W Lazio")
[1] FALSE

This will eliminate at least the problem you are facing now.
 I am not sure if I understand the rest of the question though...as I get it, you have some other character vector called text for example, and inside you are trying to look for elements of the vector pattern and, if matched, replace them with the corresponding vector element from replacement right?


Identifying whether training dataset is imbalanced in CART

Yes, your data is not balanced.

Before applying any kind of modeling it is suggestible to balance the data, you can use packages like SMOTE,ROSE to overcome the imbalance in your data.

For your next question, which index to use: It is subjective to your data. Please go through these links Link-1, Link-2 for better understanding but suggestible to try both Entropy and Gini Impurity. There is a good chance to get similar trees and similar outcome.

Gain Ratio works better to reduce bias towards highly branching features like time, salary etc. (Information about the features of the dataset isn’t mentioned)

Do let me know if you have any additional questions.

Finally, how big is your test dataset?


Training Object Detection model on just 10 images

Consider using data Augmentation first, you'll have a couple of hundred new images for you to work with, then add some noisy images since to avoid over-fitting. This way you'll have better results.


Based on the explanation you gave I think you are talking about k-fold Cross validation.

Assuming it as k-fold cross validation then you need to take mean of all the error rates. And the mean outcome would be your final error rate.


How to extract assignment from natural language text?

It's indeed a very specific type of relation extraction. Generally relation extraction is much more complex because it's not only about simple sentences Subject Verb Object and not only with the verb is.

It's not clear to me if your examples are representative of the real cases you're dealing with: if yes, you probably don't need full blown relation extraction, basic pattern matching rules will do the trick.


What are the most suitable machine learning algorithms according to type of data?

Given the list of algorithms you provided, these falls under 3 major classification of ML algorithms.

1) Classification Algorithms - Naive Bayes Classification, Decision Tree, Random Forest, kNN, Support Vector Machine (SVM), Neural Networks, etc.

2) Regression Algorithms - Linear Regression, Logistic Regression, Lasso Regression, etc.

(Note: Although Logistic Regression has Regression in its name, it is essentially a classification algorithm.

3) Clustering Algorithms - K-Means Clustering, Fuzzy C Means, Mixture of Gaussian, etc.

You might also be knowing there are 4 types of ML algorithms:

1) Supervised Learning

2) Unsupervised Learning

3) Reinforcement Learning

4) Semi-supervised Learning

Among these 4. first and second are the most important ones.

Supervised Learning is applied when we have a labelled data set i.e., we already our output variable/dependent variable. For example, a data set which contains the size of the house (independent variable) and corresponding house price (dependent variable). We can predict the house price of new data points with respect to the size of the house. Another example is, determining if a tumor is harmful or not harmful when we already have a list of tumors which are harmful or not.
In supervised learning we know the problem statement and have all the necessary features to get answer.

In Unsupervised Learning, we do not have labelled data. We do not have any output variable. We do not know the problem statement. It is applied when we need to find a structure in the data set and extract meaningful insights out of it. For example, a data set of Walmart containing its customer's buying pattern. Given this, Walmart will ask its data scientists to extract some meaning. The data scientist may choose to apply K-Means Clustering and find how customers are segmented. Group A customers -- buys X,Y,Z products; Group B customers -- buys U,V,X products.

Classification and regression algorithms are used when dealing with a supervised learning problem and clustering algorithms are used when dealing with unsupervised learning.

Now coming back to your original query,
1) Naive Bayes -- best applied to a data set containing multiple features (independent variable) and an output variable which takes two discrete value (Yes/No). Thus, categorical data.

2) SVM -- best applied to a data set containing infinite number of features and you need to reduce these features down to a number so that it can be computed. Since it's a classification algorithm so it best works upon categorical data.

3) Regression -- Linear Regression is applied to a continuous numerical data set in which the dependent and independent variable exhibits linear relationship. For example, size of the house vs house price.  Logistic Regression is a classification algorithm so it is best applied to categorical data.

3) K-Means -- K-Means can applied to many types of data sets. What it does is segmenting data points into clusters. Data points with similar features are clustered together.

4) Neural Networks -- Neural Networks can be shallow neural networks and deep neural networks and both of these could be applied to supervised or unsupervised problem as it has separate algorithms for both the cases. It is the most powerful and popular class of ML algorithms. It can be used in every problem statement. Main intuition behind it learning from its own error. I do not have much knowledge about neural network so I will not write further more.

If you want to learn more about ML and Neural Network you can apply for Machine Learning by Andrew Ng class. It's the best course out there for beginners like us.


Is there any way to use (update) a pre-trained logistic regression model for data with new set of columns?

As far I know, if you want to extend your model training, then number of feature should be same as previous one.

  In case you have more features, then you have two options:

  Drop those new features
  Retrain your model from scratch

  In case you have less features, then again you have two options:

  Place some values in missing features. Yo can use Impute classes, Refer Link for more information
  Retrain your model from scratch


How to group categorical columns into similar types?

although i don't quite understand your question. you don't have any business in worrying about data values in train and test sets, they are two different datasets after the split was done, as long as the features are the same, the content shouldn't be your focus. this is why scaling is done after splitting the dataset


Useful metrics to compare network-output image to true image?

Widely adopted method to measure similarity between two images is structural similarity:

Wikipedia
Useful links
Implementation

However, if you aim on training a network that should produce images of desired properties, I would consider GANs framework. In this case, the similarity between the generated and target image is defined by a special discriminator network.
Depending on what is your task, there are many GAN sub-types.


I'm sorry that this is not an exact solution to your problem but a work around I did, since I also tried to convert the models, but I didn't find any magic library to make it happen and I can't spend much time on this.

If you already trained the model you will also need to convert the weights.
Keras and TF weights are in hdf5 format, while pytorch weights are pickle, so you need to convert the weights and import the model.

Check out this post: How to convert my tensorflow model to pytorch model?. There you can see how you should add the weights one by one for each layer. It seems too much trouble.

Another option for the weights is this one (to be implemented while you train the model in keras): Pickling Keras Models ... but you will still need to convert the ANN architecture to Pytorch.

On the other hand, implementing models in keras is pretty easy and nice to get started with ANN architectures, so you could give it a go.

To sum up: I just found a keras library that implements the ANN backbone model that I needed, in order to train the model and make inference later, which is pretty straight forward.

Please let me know if you find a better option.


You might try it without the double quotes around the np.NaN and do not forget to import numpy (import numpy as np)

As a sidenote, you should use SimpleImputer instead of Imputer:

Imputer was deprecated in version 0.20 and will be removed in 0.22. Import impute.SimpleImputer from sklearn instead.


How do I predict new/unknown data in Bayesian linear regression?

Let me help you get started -

you don't get a single value here like you get in linear regression,rather posterior distribution of the parameters. yes, The output here is generated from a normal (Gaussian) Distribution characterized by a mean and variance , nothing special I am afraid. But depending on the mean and variance you can determine which price is more likely to be.
yes its not a real valued number like price here but the probability ,output is the likelihood multiplied by the prior probability of the data.

Here is a very simple explanation, I couldn't put it in better words-

https://towardsdatascience.com/introduction-to-bayesian-linear-regression-e66e60791ea7

And here is the more mathematical background with this chapter from stanford-

http://cs229.stanford.edu/section/cs229-gaussian_processes.pdf

and you are not alone as these Author's agree-

"Gaussian process regression models, though perhaps somewhat tricky to understand
conceptually, nonetheless lead to simple and straightforward linear algebra implementations"


What's the difference between fit and fit_transform in scikit-learn models?

To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation:

$$x' = \frac{x-\mu}{\sigma}$$

You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before forecast. But you have to use the exact same two parameters $\mu$ and $\sigma$ (values) that you used for centering the training set.

Hence, every sklearn's transform's fit() just calculates the parameters (e.g. $\mu$ and $\sigma$ in case of StandardScaler) and saves them as an internal object's state. Afterwards, you can call its transform() method to apply the transformation to any particular set of examples.

fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set $x$, while also returning the transformed $x'$. Internally, the transformer object just calls first fit() and then transform() on the same data.


Why am I getting crossvalidation scores of 0 only

I think that perhaps the problem is that cross_val_score() in its default options for the parameter cv the documentation says:

  cv : int, cross-validation generator or an iterable, optional
  Determines the cross-validation splitting strategy. Possible inputs for cv are:

  None, to use the default 3-fold cross validation,
  integer, to specify the number of folds in a (Stratified)KFold,
  An object to be used as a cross-validation generator.
  An iterable yielding train, test splits.
  For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

So my guess is that, if cv not specified the split is being done without stratification. This, coupled with the fact that by default in the iris dataset the targets are perfectly sorted (50 label 0, then 50 label 1 and then 50 label 2) means that in each 3 k-fold you are training with two classes and predicting the third one and that's why the scores are 0.

Two solutions:

A) Shuffle the data:

from sklearn.datasets import load_iris
import pandas as pd
iris = load_iris()
from catboost import CatBoostClassifier
model = CatBoostClassifier(iterations=50,
                      learning_rate=0.1, depth=4,
                      loss_function='MultiClass')
from sklearn.model_selection import cross_val_score
df = pd.DataFrame({'X0':iris.data[:,0],'X1':iris.data[:,1], 'X2':iris.data[:,2],'X3':iris.data[:,3],'Y':iris.target})
df = df.sample(frac=1).reset_index(drop=True)
scores = cross_val_score(model, df[['X0', 'X1', 'X2', 'X3']], df['Y'])
print(scores)

Out:

[0.96 0.94 0.94]

B) Modify the cv parameter:

from sklearn.datasets import load_iris
iris = load_iris()
from catboost import CatBoostClassifier
model = CatBoostClassifier(iterations=50,
                      learning_rate=0.1, depth=4,
                      loss_function='MultiClass')
from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, iris.data, iris.target, cv = 4)
print(scores)

Out:

[1.         0.92105263 0.91891892 0.78378378]


Does dropout require multiple passes of the same data set, as a sort of ensemble method?

Dropout is applied over one network. Sometimes (like with non-dropout networks) you will run your data through it multiple times before it converged and this number will be a bit higher on average with dropout but it is one network. Per layer you have a dropout probability during training and during testing/prediction you use the full network without dropout by multiplying the weights by (1-p) where p is the dropout probability. This is because during training (1-p) of the nodes are actually used.

Why this is related to ensembles is that every training instance is basically trained on a different network, by randomly dropping out nodes, forcing it to learn specific things using different nodes because it will not have all the nodes available at all times. It is not a traditional ensemble in that you combine multiple networks, just during training it acts a bit like it.


How to transform many columns of TRUE/FALSE/NA to just TRUE/FALSE?

I found that this works (though I'm interested if there's a better way - eg. avoiding the loop, or being able to exclude the columns by name instead of being just first 8).

for (i in 8:ncol(data)){
  data[,i] <- sapply(data[,i], isTRUE)
}


MSE relevance as a metric when errors < 1

I'd use relative RMSE
$\sqrt{\frac{1}{n} \sum \frac{(Preicted - True)^2}{True^2}}$.
In this case, close to 0 implies a good model, regardless of the scale of the true values.
Similarly, you can try relative MAE.


No module named "" error when loading the pickle file

According to the scikit-learn model persistence docs, it may be better to use joblib instead:

Save model

from joblib import dump

dump(model, 'filename.joblib')

Load model

from joblib import load

model = load('filename.joblib')


Massive variation in results with tensorflow and keras

I commented previously, now some idea as an answer:

1) I think that the validation data may include "lucky shots" which match the trained patterns well in some cases. Also a small validation set can be a problem. As far as I know, validation data is the last X% of the data (in Keras). You can set shuffle=True to mix validation data or False to not mix it (as I understand). I would try this.
https://github.com/keras-team/keras/issues/597

2) An option to get a good idea of how the model performs is cross validation. If you can afford it, train 3 or 5 or 10 models with different validation data and have a look at the average accuracy. Should give you also an idea of how volatile the results are.

3) You have a small sample. Thus, adding more validation data can be a problem. However, I would also try with 20%.

4) Since you have a small sample, the NN might have trouble getting all the relevant features. Instead of training a convnet from scratch, you can use a pretrained model. This has already learned a lot of features. Given a small sample, this could be THE way to go. Note that there are a number of different pretrained models which you can try.
https://keras.io/applications/

5) Get more data (if possible).

I recently used a pretrained model in a multiclass setting. Here is my code, which is a slightly modified version of a tutorial code by F. Chollet (https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb).

from keras.applications import VGG16
import os, datetime, statistics
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.utils import to_categorical
from keras import models
from keras import layers
from keras import optimizers
from keras.layers.core import Dense, Dropout, Activation
from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

###############################################
# DIR with training images
base_dir = 'C:/kerasimages'
# Number training images
ntrain = 2000
# Number validation images
nval  = 500
# Batch size
batch_size = 20
# Epochs fine tuning
ep = 600
# Epochs first step
ep_first = 50
# Number of classes (for training, output layer)
nclasses = 22
###############################################

conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(150, 150, 3))
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')

datagen = ImageDataGenerator(rescale=1./255)

def extract_features(directory, sample_count):
    features = np.zeros(shape=(sample_count, 4, 4, 512))
    labels = np.zeros(shape=(sample_count))
    generator = datagen.flow_from_directory(
        directory,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='binary')
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base.predict(inputs_batch)
        features[i * batch_size : (i + 1) * batch_size] = features_batch
        labels[i * batch_size : (i + 1) * batch_size] = labels_batch
        i += 1
        if i * batch_size >= sample_count:
            break
    return features, labels

# Lables and features
train_features, train_labels = extract_features(train_dir, ntrain)
validation_features, validation_labels = extract_features(validation_dir, nval)
train_labels = to_categorical(train_labels)
validation_labels = to_categorical(validation_labels)
train_features = np.reshape(train_features, (ntrain, 4 * 4 * 512))
validation_features = np.reshape(validation_features, (nval, 4 * 4 * 512))

#######################################
# Model
model = models.Sequential()
model.add(conv_base)
model.add(layers.Flatten())
model.add(layers.Dense(2048, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(1024, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(512, activation='relu'))#256
model.add(Dropout(0.20))
model.add(layers.Dense(nclasses, activation='softmax'))
conv_base.trainable = False

#######################################
# Data generators
train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        train_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

# Model compile / fit
model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=2e-5),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,
      epochs=ep_first,
      validation_data=validation_generator,
      validation_steps=50,
      verbose=2)

#######################################
# Fine tuning
conv_base.trainable = True

set_trainable = False
for layer in conv_base.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

model.compile(loss='categorical_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-5),
              metrics=['acc'])

history = model.fit_generator(
      train_generator,
      steps_per_epoch=100,
      epochs=ep,
      validation_data=validation_generator,
      validation_steps=50)


Why TF-IDF is working with Sentiment Analysis?

One important factor to take into account is how you use the numerical representation of words / embeddings from either TF-IDF or Word2Vec to then compute sentiments. Without knowing how you do this, it is difficult to give a concrete answer. Also, which task are you working on, what does a result of 90% mean?

Regardless of how you compute TF-IDF (there are several definitions - shown below), it is essentially assigning a numerical value to a word, thus creating a mappng of sorts. Word2Vec technically created an embedding, as it maps individual words into a vector space.

Images taken from here

The final TF_IDF is simply the multiplication of the term frequency and the inverse document frequency.

I won't go into details as to how the vectors in Word2Vec are computed, but they also define a way to assign a numerical vector (an embedding) to a single word. In essence, both of these are saying how important a word is, in the context of your documents (your corpus), with Word2Vec also having the interpretability of comparison between word vectors. For example, doing this with the associated vectors of the words actually works really well:

King - Man + Woman = Queen

Perhaps the sentiments you are computing, based on either of these methods , does something similar to taking an average over many words appearing in a sentence, and you end up with a normalised and similar results.

TF-IDF takes a more intuitive approach, looking at how many times a word appears in general, in how many of the documents does it appear and how many times. Word2Vec instead looks at which words often appear together (there is a related quote that normally gets brought up here: "You can judge a person by the company they keep"). So the intuition behind each on is slightly different, but you have numerical values for each word. Perhaps a closer comparison of TF-IDF would be to look at Doc2Vec.

There is also the GLoVE embedding model, which scores well in many NLP tasks - on the same level as Word2Vec embeddings.


How to get mean of column using groupby() and another condition


Text classification- What to do when train and test data have different features

It's perfectly fine to have different features in your training and testing partitions. If the two groups are randomly selected from the same population, one would expect, if there are many possible features, as is frequently the case in text classification, that differences would be observed. To handle this, the standard practice in the field is to train your model on the training partition, and then evaluate on the testing partition with any new features in that grouping being discarded. If you encounter the edge case of a test observation with no features after this procedure, it is common practice to use a heuristic to classify it, bypassing the model entirely (e.g., an observation with no features is classified as neutral).


Predicting survival on Haberman's dataset

You can use any classification algorithm. Read up on Logistic Regression and Support Vector Machines, if you have sufficient data, you can try neural networks. The threshold score is of extreme importance here for good precision and recall. So lookup, precision-recall curves and ROC curves which help in determining a reasonable threshold.


How does a 3D, 4D,.. hyperplane look like (visualization)?

As you can see from your examples a hyperplane of dimension $n$ is visualized in $n+1$ dimensional space. This goes basically back to the definition of hyperplane:

  In geometry, a hyperplane is a subspace whose dimension is one less than that of its ambient space. If a space is 3-dimensional then its hyperplanes are the 2-dimensional planes, while if the space is 2-dimensional, its hyperplanes are the 1-dimensional lines. This notion can be used in any general space in which the concept of the dimension of a subspace is defined.

Source: Wikipedia

Since we exist in 3D space (or 4D spacetime if you include time) one cannot visualize anything directly in 4D or higher. So you can't visualize a hyperplane with more than 2 dimensions directly either.


I do not agree that Bootstrapping is generally superior to using a separate test data set for model assessment.

First of all, it is important here to differentiate between model selection and assessment. In "The Elements of Statistical Learning" (1) the authors put it as following:

  Model selection: estimating the performance of different models in
  order to choose the best one.

  Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.

They continue to state:

  If we are in a data-rich situation, the best approach for both
  problems is to randomly divide the dataset into three parts: a
  training set, a validation set, and a test set. The training set is
  used to fit the models; the validation set is used to estimate
  prediction error for model selection; the test set is used for
  assessment of the generalization error of the final chosen model.
  Ideally, the test set should be kept in a “vault,” and be brought out
  only at the end of the data analysis. Suppose instead that we use the
  test-set repeatedly, choosing the model with smallest test-set error.
  Then the test set error of the final chosen model will underestimate
  the true test error, sometimes substantially.

In line with this the author of "Machine Learning - A Probabilistic Perspective" (2) writes:

  Often we use about 80% of the data for the training set, and 20% for the validation set. But if the number of training cases is small, this technique runs into problems, because the mode won’t have enough data to train on, and we won’t have enough data to make a reliable estimate of the future performance. A simple but popular solution to this is to use cross validation (CV)

Since in model selection you are comparing different models (different hyperparameters for a given technique or even different techniques) the absolute level of the prediction error estimate is not that important. As long as the relative error, i.e. how does model A compare to B, is sufficiently well estimated. In contrast, when assessing your final model you would like to obtain an estimate of the absolute prediction error. Also note that in the text quoted by Carlos Mougan the authors write

  [...] it can be evaluated on these instances, without the need for a
  separate validation set.[...]"

i.e. we are talking validation (model selection) and not testing (models assessment) here. Both, Cross-Validation and Bootstrapping are more common for model selection (i.e. estimate the validation error) but less common for model assessment (i.e. estimate the test error).

To summarize:

Cross-Validation and Bootstrapping are usually applied when data limited
Cross-Validation and Bootstrapping are usually applied for model selection/estimation of validation error. They are less common for test error estimates.

Nevertheless Bootstrapping can be used to estimate the validation and even the test error. There are at least 3 ways to apply it (I will now refer to "prediction error" which could either mean "validation error" or "test error" depending on the way you use Bootstrapping):

1. Simple Bootstrap

You produce $B$ Bootstrap samples for your data by randomly selecting $N$ (sample size) data points with repetition. You then train your data on each Bootstrap sample and calculate the prediction error on the whole data set. This is then averaged across all Bootstrap samples $B$. Since the prediction error is based on the whole data set, incl. the examples in the Bootstrap samples, it means that training data is also used to estimate the prediction error. Which can lead to overfitting.

2. Leave-one-out Bootstrap

Here, the approach is similar to Cross-Validation: for a predictor $\hat{f^b}$ trained on Bootstrap sample $b$ the prediction error is estimated based on all data except the examples in the Bootstrap sample $b$. (However, this is not the same as Cross-Validation as Bootstrap samples are produced by drawing with repetition.) This way the problem of overfitting is solved as training and prediction error estimation are based on different sub-data sets for each bootstrap sample.

3. .632 Bootstrap

This approach uses a linear combination of the Leave-one-out Bootstrap prediction error $error_{leave{\text -}one{\text -}out}$ and the simple training error $error_{training}$:
$$error_{.632} = 0.368 \cdot error_{training} + 0.632 \cdot error_{leave{\text -}one{\text -}out} $$

Note that $0.632$ is the probability that when drawing a Bootstrap sample a data point $(x_n,y_n)$ from the overall population is included in that Bootstrap sample. A very casual interpretation of this equation is that you use an estimate that is somewhere between the test error and the leave-one-out Bootstrap error with more weight being put on the latter.

For more details I suggest to read section 7.11 in "The Elements of Statistical Learning - Data Mining, Inference, and Prediction" (1).

(1) "The Elements of Statistical Learning - Data Mining, Inference, and Prediction"; Hastie et al; 2nd Ed

(2) "Machine Learning-A Probabilistic Perspective"; Murphy; 1st Ed


Suitable methodology to assign job titles to categories

This is regular text classification, but with very little text (only the job title). You could start with a simple one-hot encoding over the words in the job title, then apply your favourite algorithm (e.g. Naive Bayes, Decision Trees, etc). It will probably work better with some form of normalization of the words (at least using the lemma in order to mach variants of the same word).
Word embeddings are probably a good choice too, but my guess is that this would help only if you have a quite high amount of examples.


How to assign a new level to many levels of a categorical variable

Say your data is in a DataFrame df, and the column of interest is named column.

treshold = 5 #or whatever value makes sense to you

def set_levels(df):

    for i df.column.unique():
       if len(df.loc[df.column == i]) < treshold:
          df.loc[df.column == i] = 'other'

    return df

new_df = set_levels(df)


Are there known techniques to transform features X classified as C to features Y classified as C'

This problem can be addressed from an Economics perspective, even with existing data and without complicated algorithms.  There is some existing research in the Labor Economics field about transferability of skills between occupations.  For example, this "The estimation methods of occupational skills transferability" paper: https://link.springer.com/article/10.1007/s12651-016-0216-y
There is one method it discusses, referencing another 2016 paper by Ormiston, that could be helpful to you.  It involves calculating "the ratio of shared, standardized knowledge, skill and ability categories across occupations".
Here are some relevant excerpts:

It "employs a “skills” approach by explicitly estimating the proportion of knowledge, skills and abilities (KSAs) utilized in occupation i that can also be applied in occupation j."
"To estimate this proportion, it used data from O*NET – the U.S. Department of Labor’s occupational database – to examine the commonality of occupations across 120 standardized knowledge, skill and ability dimensions."
"O*NET examines each occupation and measures each component on the level (0–7 scale) of proficiency needed to function within the profession and the importance (1–5 scale) of each component to the occupation; multiplying these values together generates a comprehensive score, $S_{im}$ , for each component m of occupation i."

There is a formula given that

"estimates the skills transferability across 500 occupations based on the three-digit occupational codes and then aggregates these transferability estimates into 22 occupations in the two-digit level by taking the arithmetic mean within each two-digit category for the comparison with Shaw’s skills transferability. As a result, this study derives the 22 × 22 occupational skills transferability matrix based on two-digit U.S. census code, representing the estimated proportion of occupational human capital transferability."

There is an example given:

The data indicate that HR managers can apply only 16.01 of their 19.68 points utilized in their current profession into a new role as an industrial engineer. However, given differences in occupational skill portfolios, HR managers could apply all of their required knowledge in engineering and technology (1.03 points) and sales and marketing (5.99) in a new position as an industrial engineer. Summing the amount of shared points between occupations i and j (23.03) and dividing by the total points employed in occupation i (26.70) results in an estimated transferability rate of 86.30 % (t ij  = 0.863).

You could use this data and methods to then figure out the areas where someone's skills are lacking and make recommendations from those findings.


Transform multi-label problem to multi-class problem

A multi-label problem is when an instance can have several labels, for instance a system which classifies news articles by topic could do this:

instance 1: politics, society
instance 2: sports
instance 3: culture, sports
instance 4: society
...

To turn this into a multiclass problem and still do the exact same task, one needs to create one class for each possible subset existing in the data, for instance:

{ politics-society, sports, culture-sports, society, ...}

If the original multi-labels problem contains $N$ labels, the number of classes in the multiclass problem is $2^N$ in the worst case (number of partitions of the set).

The main problem is that the classifier needs a representative sample of every class in order to perform well. The classes sports and culture-sports (for example) are now independent from the perspective of the classifier, so the class sport cannot benefit anymore from the instances belonging to culture-sports, as it would be the case in the multi-label problem.

So in general one would need many more instances to train a multiclass classifier than a multi-label classifier for doing the same thing.


Sensor fusion using recurrent neural network: obtaining a smoothed output

To remove the noise I'd suggest you utilize Fourier Transformation to get the frequency spectrum of the signal. Then you can apply a threshold to remove high-frequency signal components which are mainly noises.

As I understood you are using python. The code below creates an artificial noisy signal and removes it using FFT (Fast Fourier Transformation) algorithm.

import numpy as np
import matplotlib.pyplot as plt

x = np.arange(0, 10, 0.01)

y = np.sin(x)
y_noisy = y + np.random.uniform(low=0, high=0.5, size=len(y))

fft_values = np.fft.fft(y_noisy)

mean_value = np.mean(np.abs(fft_values))

threshold = 10 * mean_value  # Fine-tune this

fft_values[np.abs(fft_values) < threshold] = 0

filtered_samples = np.fft.ifft(fft_values)

plt.plot(x, filtered_samples)
plt.show()

Good Luck!


Not unless you have an idea of what makes two people similar to each other. By choosing a distance function you are defining what it means for two people to be similar to one another. The question of whether one metric is quantitatively better than another may not be answerable without a defined objective.

For example, compare the Manhattan distance (L1 norm) to the euclidean distance (L2 norm). The former may be more effective for a pedestrian trying to navigate a city on foot, whereas the latter may be more effective for an athlete running across a sports field.

  Now how do I quantitatively assert that one is better than the other?

You would need some examples which embody what it means for two people to be similar (or not) to one another. This would have to be generated by a person or group of people.

Rather than generate a "similarity value" directly between two people, one could instead generate a ranked list of people which are most similar to a given person of interest:
For person A, the most similar other person is B, the second most similar person is C, etc.
Better yet, presented with a person of interest and a list of other people someone would just have to rank each person in the list by their similarity to the person of interest.

If more than one person generate ranks of a given list of people for a given person of interest, you could agglomerate the results by summing the ranks for each person and reordering the list by summed rank.

If you have some data in this form, then you may evaluate your similarity model by how well it recreates the ranked lists. Some possible metrics for evaluation are:

Recall (at rank n): What fraction of the top n people are returned in the model's top n results?
Precision (at rank n): What fraction of the model's top n results are in the true top n results?

Look here for other methods of evaluating ranking methods.

I hope this helps. Let me know if I'm not understanding your question.


Printing Feature Contributions in a Random Forest algorithm from the Treeinterpreter library leading to errors

Try this in the last part of your code:

for i in range(len(x_test)):
    print ("Instance", i)
    print ("Bias (trainset mean)", clf_bias[i])
    print ("Feature contributions:")
    for c, feature in sorted(zip(contributions[i,:,0], data.columns),key=lambda x: -abs(x[0])):
        print (feature, round(c, 2))
    print ("-"*20)

The problem is that you are sorting contributions without taking into account that contributions is a 3D array and the column names is accesible with data.columns, not data.feature_names.


how to create sklearn pipeline object using predtrained standardscalar object

This works the way you would want out of the box.

pipeline takes standard scaler class

No, pipelines get initialized with estimator instances, not the classes.  (This is why you need the parentheses in the steps, e.g. StandardScaler().)
That is, the following works:
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

X, y = load_breast_cancer(return_X_y=True)

scaler = StandardScaler()
lr = LogisticRegression()

X_sc = scaler.fit_transform(X)
lr.fit(X_sc, y)

pipe = Pipeline(steps=[('scale', scaler),
                       ('lr', lr)])

# Predicting would fail if the pipeline had unfitted estimators:
pipe.predict_proba(X)


Train/Test/Validation Set Splitting in Sklearn

You could just use sklearn.model_selection.train_test_split twice. First to split to train, test and then split train again into validation and train. Something like this:

 X_train, X_test, y_train, y_test
    = train_test_split(X, y, test_size=0.2, random_state=1)

 X_train, X_val, y_train, y_val
    = train_test_split(X_train, y_train, test_size=0.25, random_state=1) # 0.25 x 0.8 = 0.2


How to plot the bar charts of precision , recall and f-measure?

Try running the below method which uses a cross validation strategy to evaluate the models' performance across different metrics.
Of course it might be improved by for example changing the plot type to box plot so that you will see not only the mean score for each estimator but also the distribution of it.
from functools import reduce

def _get_model_name(model):
    """
            Returns a string with the name of a sklearn model
                model: Sklearn stimator class
    """
    if isinstance(model, Pipeline):
        estimator = model.steps[-1][1]
        name = "Pipeline_" + str(estimator)[:str(estimator).find("(")]
    else:
        name = str(model)[:str(model).find("(")]
    return name

def plot_cv_score(X, y, models_list, cv = 5, scoring_list = None, refit = True, return_scores = False):
    """
            X: numpy_array/pandas dataframe n_rows, m_features
            y: numpy_array/pandas dataframe n_rows
            Plots min, max and avg kfold crosval_score for a list of models

    """

    names, mean_score = list(), list()
    ldf = list()
    mnames = list()

    for i, model in enumerate(models_list):
        name = _get_model_name(model)

        if refit:
            model.fit(X, y)

        for metric in score_list:

            score = cross_val_score(model, X, y, cv = cv, scoring = metric, n_jobs= -1)
            mean_score.append(np.mean(score))

        tmp = pd.DataFrame({name: mean_score}, index = score_list)

        ldf.append(tmp)

        mean_score = list()

    frame_scores = reduce(lambda x,y: pd.merge(x,y, left_index = True, right_index = True), ldf).T

    fig, ax  = plt.subplots(1,1, figsize = (10,5))

    frame_scores.plot.bar(ax = ax, cmap = 'RdYlBu', edgecolor = "black")
    ax.legend(loc = 'best')
    ax.set_xlabel("Score")
    ax.set_title("Cross validation model benchmark")

    if return_scores:
        return frame_scores

Example:
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import Pipeline

X, y = load_breast_cancer(return_X_y= True)

models_list =[LogisticRegression(random_state= 42),
              SVC(probability= True),
              RandomForestClassifier(random_state = 42),
              GaussianNB()]

score_list = ["roc_auc", "accuracy", "f1", "precision", "recall"]

t = plot_cv_score(X = X, y = y, models_list = models_list, cv = 5, scoring_list = score_list, refit = True)

Outputs:


ways to represent document by its keyword vectors

If you have the vectors for your keywords, you can aggregate those to get the document vectors. The simplest(and probably one of the most effective) way is to average out your keyword vectors to form the document vector. Once you have the vectors for each of the documents, you can use similarity measures like cosine similarity to see how close your documents are.
A typical example of such a method is this :

Edit: Another interesting point would be to evaluate how you are getting your word vectors. Please look at pretrained word embeddings like word2vec, Glove or Fast Text.


RL Sutton book, initial estimate of q*(a) for 10 arm testbed

The description you quote explains how the true values will be set in the test when setting up a test run. This is necessary to fully state how the test works.

Initialisation of your estimates is a different issue. If you know something about the distributions of the true action values, then it would make sense to use that. For instance you could set all action values to the mean expected true value. Which is $0$. However, you may also use $0$ if you have no idea about true values, as it is a simple arbitrary value.

Setting the estimates to something using the same distribution is not unreasonable, as it that itself is an unbiased estimate of the same mean. However, it does not really serve you well here, because it adds variance to the initial estimates (as well as possibilty of them being closer to true values, they are equally likely to be worse) and on average slowing down some agent types slightly.


Minimization algorithm that can consider gradient close to solution

Well, that’s a tricky question. Do the gradients turn large somehow because they are discontinuous? (gradient descent will likely not work then) Is it maybe because the variables are in too different scales (Newton & related would help)? Is there some sort of barrier that would limit the domain (e.g. log(x))? Does the function have some flat uniform area, e.g. __/? Are these saddle points? Is your problem constrained and these are solutions at the boundaries? In some of these cases, switching to something different like subgradients might help you, but this highly depends on the reason why your problem has these optima that you want to avoid.

Generally speaking, gradient-based techniques converge to whatever local optimum they find first, and if you are not happy with that, you’ll have to use other metaheuristics (e.g. add restarts, incorporate penalties at known optima, use derivative-free methods). Perhaps other techniques that carry momentum like ADAM might somehow allow you to dodge these too.


In natural language processing, why each feature requires an extra dimension?

You can think of each feature running along its own axis on a graph. Just because we store all feature e.g. in a single DataFrame – one feature per column – it doesn't mean the data's structure is just 2d (rows and columns). This is not the case only in NLP, but in most contexts involving statistics and modelling.

We can see this with your example data. There are text blocks, which you should match to a category (as far as I can tell).

The initial dataset contains other meta-data, such as a desctiption of the dataset, the names of the target categories and also the location of each sample's file. We don't really care about these for the pure modelling part. So there are only text blocks, called data, and the target categories, called target. Your input is then 1d - the text blocks.

I will show how to put that into a dataframe, being very verbose about dimensions and features:

from sklearn.datasets import fetch_20newsgroups
groups = fetch_20newsgroups()

import pandas as pd                    # needed to use a dataframe

# Get the desired parts from "groups"
desired = ['data', 'target']           # we don't care about the 'filenames' ona so on

# make a new dictionary with only desired key-value pairs
only_data = {k: v for k, v in groups.items() if k in desired}

Now we put this into a dataframe

df = pd.DataFrame.from_dict(only_data)

# Check the shape of the dataframe
df.shape
(11314, 2)

So there are 11314 samples of 1 feature, to 1 target variable. This is therefore 1-dimensional input data (we don't count the target variable).

When we have e.g. 50 features, explaining some target variable, it may be referred to as a 50-dimensional input space. People then may use dimensionality reduction techniques, such as Principal Components Analysis, which will attempt to squeeze the 50 dimensions into a lesser number (you can choose how many to use).

In your data, you will likely pre-process the text samples to create more features. These will just be new columns in the dataframe, whose shape could become e.g. (11314, 40) if you add another 38 features, by doing things like counting words or constructing some word-embeddings.


Where can I find a crowdsourced dataset for multi-label classification with individual participant labels?

I'll think you'll find what you're looking for in this question on Open Data. I've checked the Pew Research Center data, and found this poll about cyber-security that seems to be something like what you look for.


Output of classifier.predict Tensorflow extract probabiltity

At the probabilities key you will find the probabilities of every label. Tensorflow just chooses the one with the highest. So in order to get the probability of the current outcome, you need to do something like this.

results = classifier.predict(input_fn = lambda: mem_input_fn())

for r in results:
    idx = r["classes"][0] # idx is the predicted label
    print idx, r["probabilities"][int(idx)]


What is a good method for detection of rare occurencies of speech in noisy audio data?

Detecting speech specifically is a well known problem, generally called Voice Activity Detection (VAD).
The simplest mechanisms just calculate a time-averaged ratio of energy in the speech frequencies compared to total energy, many implementations on Github of this idea. The audio codes for speech also tend to include a VAD component. A modern example can be found in libvad, based on the VAD from WebRTC project.

More generally you can try Anomaly Detection using algorithms like Isolation Forest or LocalOutlierFactor. What is critical is to use good features as input. A good starting point would be to calculate mel-spectrogram or MFCC. The spectrogram frames should be normalized, typically by subtracting the median/mean and dividing by RMS energy.
Run the anomaly detection on individual frames of 20-50ms and then do some filtering of classifications to reduce false triggering. A simple method is to require N consecutive frames to be classified as an anomaly to consider it a real anomaly. Or to require at least P percent of frames within a larger rolling window (say 200-1000ms).

In Python, you can use librosa for audio feature extraction and scikit-learn for anomaly detection

Note that for evaluating your method you will need to collect and label a test-set.


Probabilistic Machine Learning model to match spatial data

So you have an existing dataset;
$$\mathbf{X} = \{ [id_i, latitude_i, longitude_i, time_i] : i \in \{1,...,n\} \} $$
And you receive a new sample;
$$ \mathbf{x}_* = [latitude, longitude, now] $$
And you want to determine a probability of which datapoints in $\mathbf{X}$ match $\mathbf{x_*}$?

For fear of pointing out the obvious why not just use K-nearest-neighbours, where $K=1$. Obviously you need to establish a distance metric that accounts for time, or at least equate time to longitude and latitude difference, or you could just ignore time altogether.

Distance Metric

The notion of nearest is obviously associated with some measure of distance. Considering a map one can easily associate a point on a map with a coordinate system and the notion of distance is intuitive. Given a sample data point you can just find the nearest datapoint to it on the map.

But what if you have another dimension that is not intuitively obvious. This is often specific to the problem. Suppose you were searching for two criminals in a city, and criminal Adam was last seen at coordinates (0,0) 1 week ago, and criminal Brian was last seen at coordinates (1,1) 1 day ago. Now you have a new sighting at (0.25, 0.25). This is closer in geography to (0,0) than (1,1) but Brian was more recently seen so perhaps this is the more likely lead to allocate resources for a search?

When using K-nearest neighbors you might need some kind of transformation function to convert distances to probabilities. There may be numerous choices, google softmax as one possible starting point.


Getting value of 2 in normalized confusion matrix

Switch the axes:
# Build the table
confusion_table = pd.crosstab(df["prediction"], df["labels"])

# Normalise
confusion_table = (confusion_table / consufion_table.sum(0))

# Verify
print(confusion_table.sum())

Alternatively, you can play with confusion_table.divide and the axis argument: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.divide.html


Splitting into multiple heads -- multihead self attention

In principle, the pseudocode is correct, but it is not how it is implemented. The projection and dot-product attention can be done efficiently using matrix multiplication only for all heads simultaneously.
Instead of looping over the heads, you can use only one dense layer for queries, one for keys and one for values. E.g., for the keys, you would do a dense layer of dimension $h \cdot d_k$ and then reshape it.
Assuming you have batch size $b$, sequence length $l$ and model dimension $d_m$:

The input of the dense layers is of shape $b \times l \times d_m$.

The output of the dense layer is has shape $b \times l \times hd_k$ (or $d_v$ respectively).

Then you can reshape the queries and keys to have shape $b \times l \times h \times d_k$.

Now, if you permute the dimensions, such that the queries have shape $b \times h \times l \times d_k$ and the keys have shape $b \times h \times d_k \times l$, you can do the batch matrix multiplication in the last two dimensions and you end with attention energies of shape $b \times h \times l \times l$. Then, if you do softmax in the last dimension, you get the attention distribution for each head and each query.
By doing the batch matrix multiplication with projected values that now with transposed dimensions to $b \times h \times l \times d_v$, you get the weighted average. So finally, the "concatenation" of all heads is in fact just another tensor reshaping.


how to make a file name as a column in panda?

import pandas as pd
import glob

files = glob.glob('path to dir/*.csv')
df = None

for i, f in enumerate (files):
    if i == 0:
        df = pd.read_csv(f)
        df['fname'] = f
    else:
        tmp = read_csv(f)
        tmp['fname'] = f
        df = df.append(tmp)

df.head()


To learn from traditional software in that domain you could start at https://en.wikipedia.org/wiki/Comparison_of_reference_management_software