AttributeError: 'str' object has no attribute 'keys'

This error is because you are initialising a dictionary writer dict_writer = csv.DictWriter(f, fieldnames=fieldnames) but then you are passing a normal row containing only the values r=rows.values() -> for d in r: ->dict_writer.writerow(d), when the writer expects a dictionary including the keys. That is what the error means.

I don't know how your JSON looks but you can try something like this:

import json
import csv

data = '{"dateTimeValues": [1,3,4], "timeSeries": [0,0,0]}'
rows = json.loads(data)

r=zip(*rows.values())

fieldnames = ['dateTimeValues', 'timeSeries']
with open('c:/Users/jsz.P3-GROUP/Desktop/shit.csv', 'w') as f:
    dict_writer = csv.writer(f)
    dict_writer.writerow(fieldnames)
    for d in r:
        dict_writer.writerow(d)

This initialises a simple csv writer dict_writer = csv.writer(f)(not dictionary), then inputs the fieldnames as a normal row for the header dict_writer.writerow(fieldnames) and finally inserts only the values as in your example.

Note that for my json string I had to transpose the values first as in r=zip(*rows.values()).

Hope this helps.


The best way to calculate variations between 2 datasets?

What you are attempting to do is calculate the distance between two discrete probability density functions. The standard way to calculate this distance is via the Kullback-Leibler Divergence. It is defined as such:

$$D(P||Q) = \sum_iP(i)*log(\frac{P(i)}{Q(i)})$$

To be technically correct, this is not a metric in a mathematical sense because it is not symmetric and does not abide by the triangle inequality. But never the less it is a "distance" that is used a lot to compare distributions. From an information theoretical perspective, the KL divergence (given that you're using base 2 for the log) represents the number of bits it would take to encode distribution P as Q.

In your case, you would have to normalize each distribution to ensure that the sum is 1, and then calculate distance via the KL divergence. If you want a real metric you can use the Jensen-Shannon Divergence which is defined as:
$${{\rm {JSD}}}(P\parallel Q)={\frac  {1}{2}}D(P\parallel M)+{\frac  {1}{2}}D(Q\parallel M)$$$$
{\displaystyle M={\frac {1}{2}}(P+Q)} M={\frac  {1}{2}}(P+Q)$$

Hope this helps.


High validation loss, high validation accuracy

In binary classification we have either 0 or 1. You must be using softmax function at the end layer to get the output . It ranges from (0,1)
Loss: Generally speaking , it is the difference between predicted and actual value.
Suppose
CASE 1:predicted value is 0.6 real answer is 1.
CASE 2:predicted value is 0.9 real answer is 1.
Although they both have the correct answers(because they both round off to become 1) , loss in case 2 is less than case 1.


Strange behavior with Adam optimizer when training for too long

This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them.

One thing Adam does is maintain a rolling geometric mean of recent gradients and squares of the gradients. The squares of the gradients are used to divide (another rolling mean of) the current gradient to decide the current step. However, when your gradient becomes and stays very close to zero, this will make the squares of the gradient become so low that they either have large rounding errors or are effectively zero, which can introduce instability (for instance a long-term stable gradient in one dimension makes a relatively small step from $10^{-10}$ to $10^{-5}$ due to changes in other params), and the step size will start to jump around, before settling again.

This actually makes Adam less stable and worse for your problem than more basic gradient descent, assuming you want to get as numerically close to zero loss as calculations allow for your problem.

In practice on deep learning problems, you don't get this close to convergence (and for some regularisation techniques such as early stopping, you don't want to anyway), so it is usually not a practical concern on the types of problem that Adam was designed for.

You can actually see this occurring for RMSProp in a comparison of different optimisers (RMSProp is the black line - watch the very last steps just as it reaches the target):

You can make Adam more stable and able to get closer to true convergence by reducing the learning rate. E.g.

optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

It will take longer to optimise. Using lr=1e-5 you need to train for 20,000+ iterations before you see the instability and the instability is less dramatic, values hover around $10^{-7}$.


Your result is really a bit strange (it’s the R2, right, so the score makes no sense as R2 should be in a range between 0,1). When you do 10fold cv, each bit of your data will be used in one of the folds. So when 9/10 runs are okay, but one of the 10 scores in a 10th run is very bad, it could be coincidental clustering of your outliers in this one case.

For me this raises the question of robustness. So I would run „a lot“ of 10fold cv over my data to see if the problem occurs again (and to what extent). This is cheap in your case but gives you good arguments.

Also this gives you a good idea of how sensitive your model actually is to potential „strange“ outliers which might be part of real world problems when I understand you correctly.

If true, I would also check where outliers are particularly problematic if possible. I don‘t know the scale of your model, but having a look at predicted values vs actual y, may give you an idea.


I have come across such a dataset in CS 20SI GitHub repo: it's a collection of abstracts from 7200 research papers.

If you need even more, you can always write a simple crawler of arXiv web-site. The abstracts are outlined right on the webpage in a <blockquote> tag, no need to download and parse a pdf.


What are the practical applications of inverse reinforcement learning?

I think that the field has moved on from that paper. There is a trend to use the data from the expert to either "precondition" the agent or extracting a policy directly from the data. You can search for imitation learning or behavioral cloning. Some of these algorithms: Generative Adversarial Learning, DAGGER and Deep Q-learning from Demonstrations . AlphaGo used Supervised Learning as well to get to a good policy before getting trained in a RL setting. So instead of trying to recover a complicated reward function from the data you can use the above methods to get a good policy or initialize the parameters of your agent to more promising directions. Hope this helps!


Why are Convolutional Networks not using cross-correlation

That's a very interesting question.
I started looking into it recently while trying to understand better convolutional neural networks.
In short: yes! They are called convolutional while in actual practical terms using the cross-correlation operator. So this is a case of a misnomer.
I think the important thing to understand is that correlation and convolution differ only because of a flip that is present in the convolution.
So they differ only because of a sign.
Correlation:

Convolution:

The main difference in practical terms is that the convolution is associative.
This means that

This is not true for the correlation operator.
When training a neural network though, the difference in the sign in front of the i, does not matter because you will adjust your weights in order to optimise your objective function.
This is also explained in another post:
Convolution and Cross Correlation in CNN
In this post, it is also explained that what is actually used for CNN is the cross-correlation operator and not the convolution one.
I hope this helps.
PS
Also, see the notes on convolution from the David Jacobs CS course.


Which Amazon EC2 instance for Deep Learning tasks?

I think the differences and use cases are well pointed here. As far the workload, there are features which help you optimise it. According to the official documentation, you can try:

For persistency,

  sudo nvidia-smi -pm 1

Disabling the autoboost feature

  sudo nvidia-smi --auto-boost-default=0

Set all GPU clock speeds to their maximum frequency.

  sudo nvidia-smi -ac 2505,875


The cross-validation error is calculated using the training set only. Choosing the model that has the lowest cross-validation error is the most likely to be the best model for data that it hasn't yet seen. But it's not necessarily actually the best, so in that sense, strictly speaking, you can say the phrase is wrong.

Even if your sample is a properly randomly drawn from the data generating process, it's still a finite sample. Imagine if the data is one-dimensional: Each example is just a number. The average of the training set could, by sheer bad luck, deviate a lot from the average of the data generating process. It's a simplistic example, but the best model on this data is not necessarily the best on any sample from this process.


Estimating the value of $\pi$ with a Monte Carlo dartboard: $<$ or $\leq$?

Short answer: Both formulations lead to the same answer.

Mathematical explanation:

In order to understand that let us look at two similar problems. Imagine we want to integrate a function $f(x)=x^2$ on two intervals $I_1=[0,1]$ (including both bounds) and $I_2=(0,1)$ (excluding both bounds).

For $I_1$ we have

$$\int_0^1 x^2~dx=1/3.$$

For the second interval, we need to introduce a positive dummy parameter $\varepsilon$ then we can calculate the integral as

$$\lim_{\varepsilon\to 0}\int_{0+\varepsilon}^{1-\varepsilon}x^2~dx=1/3.$$

So the line of separation between both intervals does not contribute to the integral (area) because it has an infinitesimal width. The same argument can be applied to the circle area. But including the line of the circle will make the calculations easier because we do not need to introduce a dummy variable.

Numerical explanation:

Because of the numerical precision of your computer, it will be very unlikely that the generated random numbers will lead to points that are really on the line of the circle. Numerically it will be impossible to obtain any value in which at least one of the coordinates is irrational. Only if both values are rational and if $x^2+y^2=1$ they can lie on the circle. But this case is also very unlikely.


Difference between Gensim word2vec and keras Embedding layer

Yep, you're right!  As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation.  This process is called word embedding, and finding word embeddings is the task of the keras Embedding layer.

Ideally, word embeddings will be semantically meaningful, so that relationships between words are preserved in the embedding space.  Word2Vec is a particular "brand" of word embedding algorithm that seeks to embed words such that words often found in similar context are located near one another in the embedding space.  The technical details are described in this paper.

The generic keras Embedding layer also creates word embeddings, but the mechanism is a bit different than Word2Vec.  Like any other layer, it is parameterized by a set of weights.  The weights are randomly-initialized, then updated during training using the back-propagation algorithm.  So, the resultant word embeddings are guided by your loss function.

To summarize, both Word2Vec and keras Embedding convert words (or word indices) to a hopefully meaningful numeric representation.  Word2Vec is an unsupervised method that seeks to place words with similar context close together in the embedding space.  Keras Embedding is a supervised method that finds custom embeddings while training your model.


If you are only interested in the most likely class, during inference you can skip the softmax. This is fairly common even, and the reason why TensorFlow has specific logit functionality. However, for optimizing your network you need a loss function that makes sense. You have normal labels, what would your loss function be if you only compute the linear combinations?


Automatically categorize parts of a piece of writing

As far as I know there's no specific task for this, it's general text classification.  It's also related to text segmentation. There are certainly existing systems for similar tasks, but probably specific to a certain type of data.

In general this would be a supervised process, you would need a sample containing many documents in which the parts are pre-annotated. I think sequence labeling algorithms would be the standard approach.


An r in front of a string makes it a raw string literal. In short it means that any \ in that string will not be seen as an escape character, like when you write \n for newline.

Some more info can be found here:
What exactly do “u” and “r” string flags do, and what are raw string literals?


Similar algorithm to apriori to find unpopular sequential patterns

The algorithms I would recommend in your case are Apriori-Inverse and Apriori-Rare. Disclaimer: I have not found a Python implementation (nor can I provide a reliable one due to time constraints), but there exists an open-source Java library SPMF with implementations of these algorithms (Inverse, Rare, and a whole bunch more) which can be used for reference -- the source code has informative comments and the algorithms are structured well enough to understand what is going on. Of course, this is only an option if you are willing to do some digging and have enough time on your hands. Maybe going through that Java code will provide you with insights on how to modify existing Python implementations to suit your needs, which should require far less effort, since Inverse and Rare differ very little from the classical Apriori. I hope this helps you at least somewhat. Good luck!


Classification problem: estimate the number of people in a household through account transactions

I think regression makes more sense here than classification; you are trying to predict a number. In particular I would recommend Poisson regression since you are modeling count data. Software is readily available for various languages.


How do AI's learn to act when the problem space is too big

That is a good question and many scientists around the world are asking the same. Well, first a game like Age of Empires is not considered to have a really big solution space, there are not so many things you can do. It's the same in games like Mario Bros. The problem of learning in easy games like Atari games was solve by the guys of DeepMind (here the paper), that was acquired by Google. They used an implementation of Reinforcement Learning with Deep Learning.

Going back to your question. A really big problem is how to imitate the amount of decisions a human being takes every day. Wake up, have breakfast, take a shower, leave your house... All these action need a really high level of intelligence and many actions to develop.

There are many people working on this problem, I'm one of them. I don't know the solution but I can tell you in which way I'm looking. I follow the theories of Marvin Minsky, he is one of the fathers of AI. This book, the Emotion Machine, tells a very good view of the problem. He suggested that the way to create a machine that imitates the human behavior is not by constructing an unified compact theory of artificial intelligence. On the contrary, he argues that our brain contains resources that compete
between each other to satisfy different goals at the same moment. They called this Ways to Think.


Theoretically, you could use n-grams to model text sequences. But there are some good reasons why it's not often mentioned in textbooks.

Intuitively, modeling sequences is learning that after a, b, and c comes d. So if your sequence is:

"Hello, my name is Bob, what a lovely day today, how are you?"

It technically doesn't matter if a = "Hello", or "H", or "Hello, my name is Bob". To your model, it will just learn that after a thing, comes another thing etc.

So yes, you could use n-grams. But the problem is that you need to make sure that the way the ngrams are split is meaningful (i.e. "Hello, my" is a probably worse split than "my name is", which probably happens more often)

But don't worry, the whole point of using LSTM is that the memory part of the network will mimic using n-grams, as long as it helps with the overall performance. This means that if it needs to take more words into account to predict the next one, it will do it up to the depth of your network.


Keras - error when adding layers to loaded model

Actually, the number of parameters for the second layer is correct, the number of parameters in a convolutional layer is calculated as follows:

n_parameters = n_filters_l * (n_filters_l-1 * (kernel_size_h * kernel_size_w) + 1)
             = 2048 * (2048 * (3 * 3) + 1) = 37750784

If you want to decrease the number of parameters, you therefore have multiple options:

Decrease the number of filters in the layer before it
Decrease the number of filters in the layer itself
Use a lower kernel height of width


Multiple labels in supervised learning algorithm

For starters, Naive Bayes is probably not appropriate here.  It assumes independence among the inputs (hence the "Naive") and words in a sentence are very dependent.

But, assuming you really want to run with NB as an algorithm to start your experimentation, there are two options I'd consider:

Ungraceful: Lots of NB classifiers

This would be an alternative approach.  Make a corupus of all the words observed as your vector of inputs.  Make a corpus off all the tags that are observed as your vector of outputs.  An NB classifier with multiple outputs is the equivalent of having multiple NB classifiers with one output each (so do whichever is easier to implement in whatever software framework you're using).  Treat each element as a training sample where a given input (a word) is a 1 if that word is present and a 0 if that word isn't.  Use the same binary scheme for the output.

This brute forces the application of the NB Classifier to your data, and leaves you to find meaning by still haivng to mine the huge set of classifiers you'll be left with.

More Graceful: Process your data

This is the approach I'd recommend if you want to run with one multiple-class NB Classifier.

Your goal here is to figure out how to map each set of tags to a single class.  I'm sure there is some sort of clustering scheme or network analysis (perhaps ["celebrity"] linked to ["murder"] could become a the segment ["debauchery"]) that will sensibly map your tags to one single cluster.  If you treat tags as nodes and two given tags together as links, then you'll want to look into community detection algorithms (which is where I'd start).  But, if you just want something working, then some sort of hack on the tags that converts a list of tags to only the tag that's most commonly seen in your dataset would be enough.

This method front-loads the work of cleaning your data and would make the NB Classifier's output easier to understand.


How to determine the complexity of an English sentence?

Yes. There are various metrics, such as the fogg index.
Textacy in python has a nice list and implementations.

>>> ts.flesch_kincaid_grade_level
10.853709110179697
>>> ts.readability_stats
{'automated_readability_index': 12.801546064781363,
 'coleman_liau_index': 9.905629258346586,
 'flesch_kincaid_grade_level': 10.853709110179697,
 'flesch_readability_ease': 62.51222198133965,
 'gulpease_index': 55.10492845786963,
 'gunning_fog_index': 13.69506833036245,
 'lix': 45.76390294037353,
 'smog_index': 11.683781121521076,
 'wiener_sachtextformel': 5.401029023140788}


Training a CNN to convert ellipses into circles

Use your generated images to train an encoder-decoder that attempts to rebuild the input image. We'll call these pieces the input encoder and ellipse decoder. In parallel, train a second decoder that takes the input encoder (i.e. embedding representation) as input and attempts to rebuild the original circle image. The input encoder + circle decoder can then be used together for scoring out-of-sample observations.


CNN image to image translation: multiple image inputs to one image output

GAN is a good choice also have a look at Autoencoder as well at the links below
https://www.tensorflow.org/beta/tutorials/generative/cvae

and

https://towardsdatascience.com/deep-autoencoders-using-tensorflow-c68f075fd1a3

these two are very similar and will solve your problem statement for sure.


How to arrange the dataset/images for CNN+LSTM

I'll expand my answer if there is any interest. I propose building a simple test model using Vgg and two custom fully connected layers that end with a single sigmoid per the document below. If you prefer early fusion I recommend using stacked black and white images.
https://www.cs.cmu.edu/~rahuls/pub/cvpr2014-deepvideo-rahuls.pdf

if you then want to go the route of LSTM, then this graphic from Karpathy will help. Use the many to one architecture. The inputs to the RNNS or LSTMS vary by their implementation. You would most likely want to pass the final convolutional layer though a fully connect layer and then into the RNN/LSTM. https://karpathy.github.io/2015/05/21/rnn-effectiveness/


What is a 'hidden state' in BERT output?

BERT is a transformer.

A transformer is made of several similar layers, stacked on top of each others.
Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n.

The hidden state you mention is simply the output of each layer.

You might want to quickly look into this explanation of the Transformer architecture :
https://jalammar.github.io/illustrated-transformer/

Note that BERT use only Encoders, no Decoders.


Dealing with unseen data/categories in machine learning models for stream data

If you want to make an inference on an order where a categorical variable was not seen in the training data, you could train the model on a hash bucket representation of that variable.
If using tensorflow, you can leverage:
https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_hash_bucket
Or implement yourself.


Performing Binary classification using binary dataset

Yes. Many data sets are one-hot encoded into only 0s and 1s because some ML algorithms work best that way.

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html


Find recurrent dates in a small set (and get rid of non recurrent ones)

If it is a categorization problem then you should look for a classification algorithm, not a regression technique. The simplest classification algorithm is Logistic Regression.

But by the looks of it, seems like you do not have a labelled data-set and if that's the case you should look for Clustering techniques. Clustering is a part of Unsupervised Learning Technique in ML which create clusters or groups of similar data points.


Is there any possibility to apply deep dreaming in data augmentation?

TL;DR Not usually, because Deep Dream ignores the distribution of your input population.
You cannot extract additional informative data using generators, only remix and recombine what you already have. There may be a few situations where that is useful because you can use a generator trained on other data to bring in extra relevant information that was not in your main dataset.

Deep Dream logically alters an input structure to maximise some measure within a neural network based on neuron activations. This might be the activation of a single neuron, or of a layer of neurons.
I have worked with Deep Dream scripts based on Google's Inception v5 network which was trained on ImageNet, just for fun.
Here's a relatively extreme example of output, with a few specific neuron activatioons maximised and a long zoom so that the data is essentially only a super-stimulous for those artifical neurons:

One thing you can say with reasonable certainty is that this image does not look like anything you might draw at random from ImageNet, from any category. It would be a poor choice as augmentation for ImageNet classification tasks, because it is not from the target population.
The same is true in general for all obvious Deep Dream modifications. The modifications do not look natural and produce inputs that would not reasonably be in the input distribution. Once you are clearly outside the input distribution for your use case, then learning how to process those inputs may not give you any benefit related to the goals of your machine learning task.
More subtle changes might be useful, if on inspection it is clear that they should not change the categorisation. This might be similar augmentation to adding noise, and if the alterations were chosen carefully, might help defend against adversarial inputs. In fact generating adversarial inputs is an almost identical process to Deep Dream, usually focusing on the output categories and updates that look like high frequency noise so that humans tend not to see the difference.
There are better data generating models that attempt to create data that could have come from the input population. Two major ones are variational autoencoders (VAE) and generative adversarial networks (GAN). These might be useful for data generation.
However, if you are considering somehow "bootstrapping" a small dataset by training one of these generators on it then using its output to augment your data, it is unlikely that this will work. The quality of the output will usually depend on its training data such that generated data will not truly add information that you did not already have in the original training set. In most cases you will better off looking at simpler logical transformations of data that you are confident will not change the target label (or that change it in predictable ways) - for image-based CNNs that might include changing rotation, crop, brightness/contrast and other basic image manipulations.
There might be a few cases in which working with a generator helps. For instance if you can train it on far more data because you have lots of unlabelled data, or dat afrom a different but similar task. Then the quality of generated inputs could be high enough that you effectively get to make use of unsupervised learning from the same dataset. You might still find it easier to do transfer learning, or train an autoencoder and re-use its first few layers though.


Both Python and R are extensively used in the field of data science.Though both have their advantages and disadvantages, python has many libraries that are awesome to use for data science purpose. For example, scikit-learn, Keras, Theano, XGBoost, Matplotlib,etc. These make lives so easy for data science.
Yes, analyzing and understanding the code of others can give you a lit of information but you cannot learn until you don't get your hands dirty with some python code. In order to learn how to use python in data science, this link provides an excellent tutorial.


When a relational database has better performance than a no relational

It depends on your data and what you're doing with it. For example, if the processing you have to do requires transactions to synchronize across nodes, it will likely be faster to use transactions implemented in an RDBMS rather than implementing it yourself on top of NoSQL databases which don't support it natively.


So long as you want to use those variables to define closeness then yes, you just have to encode them differently. You should use a one-hot-encoding for the discrete variables (e.g., Employee_ID, Store_ID, etc.) and just the Product_Value as is. I answered a similar question about K-means for just categorical variables (like Employee_ID) so I've copied the code below that gives a quick demo for using the Clusters as a feature for predicting something after you use K-means. As I said in my old answer, in general, this framework isn't optimal but it's okay for a simulation.

library(glmnet)
library(Matrix)
n <- 1e5
nclusters <- 5
set.seed(420)
ls <- data.frame(sample(letters, n, replace=TRUE))
xs <- sparse.model.matrix(~.,data=ls)
print(head(xs))
# Now let's run k-means
out <- kmeans(xs, centers=nclusters)
bs <- rep(1, dim(xs)[2])
# Let's run k-means on the different categories
clusterpred <- data.frame(out[[1]])
ys <- xs %*% bs + rnorm(n)
print(table(clusterpred))
# Now let's use a clustered data set to predict some outcome
cxs <- sparse.model.matrix(~.,data=clusterpred)
model <- glmnet(y=ys, x=xs, alpha=0)
cmodel <- glmnet(y=ys, x=cxs, alpha=0)

# Predictions
yhat <- predict(model, xs)
yhatc <- predict(cmodel, cxs)
# Looking at the difference RMSEs
print(sqrt( sum( (ys-yhat)**2 )))
print(sqrt( sum( (ys-yhatc)**2 )))


My best guess here is that your learning rate is way too high for the problem. You also probably have far more neurons in your hidden network than you need, seeing as you're using just one feature.

Recall that learning rate is controlling the "step size" in gradient descent and that for your dataset, it is likely far too high. I made some minor changes to your code and got better results than linear regression. Notice the use of 2 hidden neurons, a 0.001 learning rate, and 20 iterations.

# Now using the sknn regressor
# http://scikit-neuralnetwork.readthedocs.org/en/latest/guide_beginners.html

from sknn.mlp import Regressor, Layer

nn = Regressor(
    layers=[
        Layer("Rectifier", units=2),
        Layer("Linear")],
    learning_rate=0.001,
    n_iter=20)

nn.fit(diabetes_X_train, diabetes_y_train)
print("Results of SKNN Regression....")

# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

SKNN regression:

Results of SKNN Regression....
Coefficients:  [ 938.23786125]
Residual sum of squares: 6123.67
Variance score: 0.50


Spatial Transformer Networks: how is theta differentiable?

In spatial transformer networks, basically, the concept of localisation network is to learn to apply a transformation to find the canonical form of the input. Imagine the output of the network $\theta$ as an activation which is passed to another layer. The point is that the sampling sequence of operations is differentiable. $\theta$ is just an output which specifies how the sampling should be performed. The sampling operation that is usually used is bilinear interpolation which although is not differentiable at all points due to the floor and ceiling functions, it can backpropagate the error and is differentiable in most of its inputs. Consider the $\theta$ just as activation which is passed to the bilinear sampler for changing the input of the next network. bilinear sampling is considered to be differentiable.

To understand it better, consider the following figure which illustrates the process inside a spatial transformer easier than the one in the original paper.

As it is clear, the output of the localisation network which is $\theta$ will be passed to the sampling grid. Sampling gird will be multiplied to the $\theta$ to find appropriate regions in the original image. Consider that you don't multiply $\theta$ to the original image. The reason is that if you multiply by the original image, there will be multiple choices for a single pixel while if you multiply the output of the localisation network by the sampling grid, for each entry there is just a single choice. Next, the sampled grid and the original image will be used in the interpolation to find the transformed image. As it is clear, $\theta$ is like the other activations.


No, no special structure is required for text classification problems in comparison with core Orange.

However — and this holds for core Orange as well as for text addon — if you would like to perform classification one feature must be set as class variable. Otherwise Orange cannot know which feature you would like to predict. When you are reading the data from CSV by default no feature is selected as class variable and hence the message.

There are two possible approaches how one can mark a feature as class variable. The more permanent one, as you already figured, is to modify the data file itself. Second, a bit more user friendly one, is to do this in Orange. To do so you just pass the data to Select Columns widget, drag the desired feature to the Target Variable field and click Send. The data on the output of the widget will have a class variable and can be passed to Test & Score. The following scheme should work without modifying the header of a file:

If you would like to know more about how the Select Columns widget works you can also check its documentation.

So in a nutshell you were missing a step and what happened is expected. If the header does not state that some feature is a class variable — as was the case for your CSV — Orange simply won't mark any feature as class variable. We don't use any heuristics to guess what might be a class variable but solely rely on what is written in the file.


Use chi-square test to check the goodness of fit to a specific distribution
http://courses.wcupa.edu/rbove/Berenson/10th%20ed%20CD-ROM%20topics/section12_5.pdf


It depends on the application and the Data.Could you provide more details about the data of the trips? Do you have GPS-positions for the trips?

There are Libraries such as Barefoot which support you in map-matching, meaning that it maps your positions to the underlying map, in most cases OpenStreetMap. This library uses Hidden Markov Models for online and offline Map Matching. Maybe this could help you as well, when implementing your own model.


very low variance explained after applying pca

It is entirely correct to apply PCA to a dataset like MNIST. Intuitively, corner pixels should almost never contain any information as to what digit is contained in the center of the image. So we should disregard them. You should expect similar results as with other datasets. PCA lowers the dimensionality of your data, thus allowing for a less complex model, however, this is at the cost of some information that is rejected when retaining only $n$ components.

When data is limited a less complex model will result in a lower bias and thus better accuracy when applied to novel data. However, the MNIST dataset is plentiful, thus the benefit of removing some features and losing even minimal information may cause your performance to degrade.


Forcing class imbalance to mirror the target data

Disclaimer: I am assuming that by "source data" you mean the data you are using for training and by "target data" you mean the real world data the model will aim to classify.

SMOTE Is usefull when your data is highly imbalance i.e. 1% A/ 99% B or something around this proportions. In your case I would expect the model to work just fine when classifying a different distribution of data as long as you have enough training examples in each class. If this approach doesn't work you can use cost sensitive learning to fit the imbalance but I would consider this a secondary path


Explaining the similarities between instances in a cluster with KMeans

First things first - You got back 3 clusters from K-Means because you asked it to find the 3 best clusters (and not the other way around). Google up the "elbow-method" to help you determine the best number of clusters for your dataset.
Clustering would make use of the all 10 columns in your dataset.
In order to figure out which fields matter to a cluster - Find the mean and standard deviation of every field within the context of a cluster. The fields with the lowest standard deviation probably tell you why these instances were grouped together in a single cluster.
Another thing you could try out is - use CLusters as Labels - and run your dataset by a classification model such as Random Forests or xgboost. Random Forests for instance gives you a Feature Importance - which will provide you insights on the most impactful features in your dataset.


Unstable accuracy of CNN - When should I stop training?

If accuracy regresses something is wrong in either the network, or (more likely here) the meta-parameters (probably learning rate.)

It can be difficult to tell when a model converges. I'd recommend looking at diagnostic graphs (typically training loss, training/validation accuracy, and ratio of weights:updates) over epochs. Typically convergence is considered when loss and accuracy level out and show diminishing returns beyond some threshold (your tolerance for 1.0e-x% improvements.) So, stop training/validation when it's improving less than what you care about.


Get Logistic regression scores in CNN using Keras

The evaluate method return the loss value & metrics values for the model in test mode.
Instead You should use

y_pred = model.predict(x_test, batch_size=batch_size)

As it generates output predictions for the input samples.
For more information, read Keras official documentation.


Am I supposed to scale binary features alongside other numerical features?

There are two important concepts that you need to understand.

Scaling your features won't affect their "importance" in your neural network

Intuitively, your neural network will itself learn which feature is important or not by learning weights.

Scaling your features will speed up your convergence and will limit the risks of overshooting or being stuck in a local optimum. It also makes numerical sense.

Intuitively, scaling features can make your training faster because your "path to convergence" will probably be shorter if your features are scaled (right picture).

It also makes numerical sense to scale your features because if you have very large and very small values, some of your weights might drop very low and this could cause some numerical issues and hinder the performance of your model.


What is the order of elements in an image in python?

The [64, 64, 3] shape you have found is a common convention to represent a colour image in (x, y, colour_channel) dimensions.

The key word here is convention - there is no inherently preferred way to represent a colour image in terms of fundamental maths or computing needs, and even within Python you will find multiple conventions, varying in the ordering within the dimension - e.g. OpenCV uses (x, y, channel) convention for the shape, but has channels in order BGR - so channel 0 is blue - whilst most other libraries will use RGB ordering (ignoring for now the alternative colour spaces).

  My question is that in numpy if you have a three dimensional array, for accessing rows and columns you have to change the second and third entries in the indexing operator

When you have a 3-dimensional array, what you decide to call "rows" and "columns" is also a convention. It depends partly on what that array represents, and there is no single way to visualise the contents.


Forget Layer in a Recurrent Neural Network (RNN) -

Great question!

tl;dr: The cell state and the hidden state are two different things, but the hidden state is dependent on the cell state and they do indeed have the same size.

Longer explanation

The difference between the two can be seen from the diagram below (part of the same blog):

The cell state is the bold line travelling west to east across the top. The entire green block is called the 'cell'.

The hidden state from the previous time step is treated as part of the input at the current time step.

However, it's a little harder to see the dependence between the two without doing a full walkthrough. I'll do that here, to provide another perspective, but heavily influenced by the blog. My notation will be the same, and I'll use images from the blog in my explanation.

I like to think of the order of operations a little differently from the way they were presented in the blog. Personally, like starting from the input gate. I'll present that point of view below, but please keep in mind that the blog may very well be the best way to set up an LSTM computationally and this explanation is purely conceptual.

Here's what's happening:

The input gate

The input at time $t$ is $x_t$ and $h_{t-1}$. These get concatenated and fed into a nonlinear function (in this case a sigmoid). This sigmoid function is called the 'input gate', because it acts as a stopgap for the input. It decides stochastically which values we're going to update at this timestep, based on the current input.

That is, (following your example), if we have an input vector $x_t = [1, 2, 3]$ and a previous hidden state $h_t = [4, 5, 6]$, then the input gate does the following:

a) Concatenate $x_t$ and $h_{t-1}$ to give us $[1, 2, 3, 4, 5, 6]$

b) Compute $W_i$ times the concatenated vector and add the bias (in math: $W_i \cdot [x_t, h_{t-1}] + b_i$, where $W_i$ is the weight matrix from the input vector to the nonlinearity; $b_i$ is the input bias).

Let's assume we're going from a six-dimensional input (the length of the concatenated input vector) to a three-dimensional decision on what states to update. That means we need a 3x6 weight matrix and a 3x1 bias vector. Let's give those some values:

$W_i = \begin{bmatrix} 1 & 1 & 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 & 3 & 3\end{bmatrix}$

$b_i = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}$

The computation would be:

$\begin{bmatrix} 1 & 1 & 1 & 1 & 1 & 1 \\ 2 & 2 & 2 & 2 & 2 & 2 \\ 3 & 3 & 3 & 3 & 3 & 3\end{bmatrix} \cdot \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \\5 \\6 \end{bmatrix} + \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 22 \\ 42 \\ 62 \end{bmatrix}$

c) Feed that previous computation into a nonlinearity: $i_t = \sigma (W_i \cdot [x_t, h_{t-1}] + b_i)$

$\sigma(x) = \frac{1}{1 + exp(-x)}$ (we apply this elementwise to the values in the vector $x$)

$\sigma(\begin{bmatrix} 22 \\ 42 \\ 62 \end{bmatrix}) = [\frac{1}{1 + exp(-22)}, \frac{1}{1 + exp(-42)}, \frac{1}{1 + exp(-62)}] = [1, 1, 1]$

In English, that means we're going to update all of our states.

The input gate has a second part:

d) $\tilde{C_t} = tanh(W_C[x_t, h_{t-1}] + b_C)$

The point of this part is to compute how we would update the state, if we were to do so. It's the contribution from the new input at this time step to the cell state. The computation follows the same procedure illustrated above, but with a tanh unit instead of a sigmoid unit.

The output $\tilde{C_t}$ is multiplied by that binary vector $i_t$, but we'll cover that when we get to the cell update.

Together, $i_t$ tells us which states we want to update, and $\tilde{C_t}$ tells us how we want to update them. It tells us what new information we want to add to our representation so far.

Then comes the forget gate, which was the crux of your question.

The forget gate

The purpose of the forget gate is to remove previously-learned information that is no longer relevant. The example given in the blog is language-based, but we can also think of a sliding window. If you're modelling a time series that is naturally represented by integers, like counts of infectious individuals in an area during a disease outbreak, then perhaps once the disease has died out in an area, you no longer want to bother considering that area when thinking about how the disease will travel next.

Just like the input layer, the forget layer takes the hidden state from the previous time step and the new input from the current time step and concatenates them. The point is to decide stochastically what to forget and what to remember. In the previous computation, I showed a sigmoid layer output of all 1's, but in reality it was closer to 0.999 and I rounded up.

The computation looks a lot like what we did in the input layer:

$f_t = \sigma(W_f [x_t, h_{t-1}] + b_f)$

This will give us a vector of size 3 with values between 0 and 1. Let's pretend it gave us:

$[0.5, 0.8, 0.9]$

Then we decide stochastically based on these values which of those three parts of information to forget. One way of doing this is to generate a number from a uniform(0, 1) distribution and if that number is less than the probability of the unit 'turning on' (0.5, 0.8, and 0.9 for units 1, 2, and 3 respectively), then we turn that unit on. In this case, that would mean we forget that information.

Quick note: the input layer and the forget layer are independent. If I were a betting person, I'd bet that's a good place for parallelization.

Updating the cell state

Now we have all we need to update the cell state. We take a combination of the information from the input and the forget gates:

$C_t = f_t \circ C_{t-1} + i_t \circ \tilde{C_t}$

Now, this is going to be a little odd. Instead of multiplying like we've done before, here $\circ$ indicates the Hadamard product, which is an entry-wise product.

Aside: Hadamard product

For example, if we had two vectors $x_1 = [1, 2, 3]$ and $x_2 = [3, 2, 1]$ and we wanted to take the Hadamard product, we'd do this:

$x_1 \circ x_2 = [(1 \cdot 3), (2 \cdot 2), (3 \cdot 1)] = [3, 4, 3]$

End Aside.

In this way, we combine what we want to add to the cell state (input) with what we want to take away from the cell state (forget). The result is the new cell state.

The output gate

This will give us the new hidden state. Essentially the point of the output gate is to decide what information we want the next part of the model to take into account when updating the subsequent cell state. The example in the blog is again, language: if the noun is plural, the verb conjugation in the next step will change. In a disease model, if the susceptibility of individuals in a particular area is different than in another area, then the probability of acquiring an infection may change.

The output layer takes the same input again, but then considers the updated cell state:

$o_t = \sigma(W_o [x_t, h_{t-1}] + b_o)$

Again, this gives us a vector of probabilities. Then we compute:

$h_t = o_t \circ tanh(C_t)$

So the current cell state and the output gate must agree on what to output.

That is, if the result of $tanh(C_t)$ is $[0, 1, 1]$ after the stochastic decision has been made as to whether each unit is on or off, and the result of $o_t$ is $[0, 0, 1]$, then when we take the Hadamard product, we're going to get $[0, 0, 1]$, and only the units that were turned on by both the output gate and in the cell state will be part of the final output.

[EDIT: There's a comment on the blog that says the $h_t$ is transformed again to an actual output by $y_t = \sigma(W \cdot h_t)$, meaning that the actual output to the screen (assuming you have some) is the result of another nonlinear transformation.]

The diagram shows that $h_t$ goes to two places: the next cell, and to the 'output' - to the screen. I think that second part is optional.

There are a lot of variants on LSTMs, but that covers the essentials!


Another solution, without however the active learning support that prodigy offers, is doccano. It's open-source (free), easy to install and supports various applications, apart from NER (BIO tagging), also classification and translation.


How to test the influence of a feature on conversion?

Okay, thanks to this post, I was able to test this.
The test we need is a Chi-square test for approx results and Fischer Test for exact variables.

Given, the variables I had were categoric (engaged:yes/no and converted:yes no) in nature, the base assumption was true.

Now, First I created contingency table for each of those engaged and converted with reference from here.

There after, calculated the expected value for each of the cells in table.
Injected the formula and voila, I had the chi-square vale. Further compared with chi-square value table to get the p-value.

This resulted in testing whether the variables are independent or not.

This is what the end results look like:

Reference: https://www.socscistatistics.com/tests/chisquare2/Default2.aspx


Can't interpret the text information and ratings matrix imported to NN

I am the author of the CDL paper.

For the mult.data file, in

63 1:2 1666:1 132:1 901:1 1537:2 8:1 9:1 912:1

63 is the number of words for this documents, 1:2 means word 1 appears twice in the document, 1666:1 means word 1666 appears once in the document, etc.

For the trainuser.dat, in

10 1631 3591 10272 14851 4662 13172 12684 5324 3595 3404

10 is the number of positive samples for this user, the rest is a list of 10 items that are related to (liked by) this user.

You can check the README file in the Datasets collection for more details on the datasets.


What is the difference in plotting ROC curve with probability scores vs binary decisions

How do you go from probabilities to binary decisions? The default choice is to use a threshold of 0.5 but maybe a threshold of 0.3 or 0.7 would have given better results (depending on you metric). The ROC curve gives you more information as it allows to see the results for each probability threshold.
Usually you set some metric to optimize (F1 score for example) and you set the threshold based on this metric. Then you plot the confusion matrix and any other metric that is useful to you


Interpreting a curve val_loss and loss in keras after training a model

The loss curve shows what the model is trying to reduce. The training procedure tried to achieve the lowest loss possible. The loss is calculated using the number of training examples that the models gets right, versus the ones it gets wrong. Or how close it gets to the right answer for regression problems.

The loss curves are going smoothly down, meaning your model improves as it is training, which is good. Your test loss is slightly higher than your training loss, meaning your model is slightly overfitting the training data, but that’s inevitable, it doesn’t seem problematic. All seems ok from this plot.

Now your model is getting an accuracy of 30% or so. Unless you tell us what the model is doing and how you define accuracy, there’s no way of knowing if that’s ok or not.


Clustering: How to cluster multiple CSV files that each represent a Steam user

You're likely going to have to do a little data wrangling to get the data in a better format.

I'm assuming each file has a varying amount of rows, one for each game the user purchased? So, you could create one big matrix with the rows representing users and the columns representing games, and create an indicator matrix for to map purchases to games. Then potentially add additional columns per user including additional insight, like:

the number/percent of games purchased for each category(action, RPG, ...)
total purchases

Having data in that format may lend to more useful application of clustering algorithms.


How to recognise when to stop training based on Overfitting/Underfitting?

Overfitting :

  The model tries to memorize what it has learned. Hence, it could not
  classify unseen samples.

In case of overfitting, the validation accuracy stops increasing and the validation loss also does not decrease.

It means that the model can no more generalise itself to get a validation accuracy above a certain threshold.

Hence, you can stop the training, when the val_acc does not change for a specific number of epochs.

Underfitting :

  Underfitting means that model is not able to classify any of the
  samples even after learning them.

The model should stop its training when the accuracy and loss seem to be constant or they only revolve around a certain value.

In your case :

The loss for the train as well as test  seem to decreasing simultaneously. The test curve flattens a bit earlier. It could be treated if the learning rate is decreased.

Tip: If you are using Keras, try the EarlyStopping callback.


Classifying videos with varying length using ConvLSTM2D in tensorflow

I found a solution that works for me.
Since I wanted to avoid using padding and masking, and didn't fully understand ragged tensors, I decided to continue with using varying input lengths.
My training data consists of a list of image stacks between 6-82 frames. When trying to use this directly with model.fit(x=x, y=y, batch_size=1) where x is a list of tensors, tensorflow will complain that the input dimensions have varying size. I thought this didn't matter since the batch size was 1, but apparently tensorflow tries to change the list of image stacks to a tensor.
A way around this, is to pass a training sample for every step using a generator:
class ArtificialDataset(tf.data.Dataset):
  def _generator(samples):
    for i in samples:
      yield (image_stack, output)

And then making sure that the steps per epoch is the size of the data set
training_data = ArtificialDataset(training_samples)
model.fit(training_data, epochs=epochs, steps_per_epoch=len(training_samples))

This way tensorflow never tries to create a tensor with varying inner dimensions. A drawback of this method is that no batching occurs, so training takes a while. In my case this doesn't matter much since the network size and input data already requires me to load very few samples at a time.
An optimization would be to batch same-length videos (or videos with same number of extracted frames). I might do this at a later time, since the implementation of varying batch sizes is too much work right now.


Quick question on basic Basic concept of experience replay

The 2nd one is the correct one. We update the weights according to the mean squared error between real value (in case of DQN it is a bootstrapped value) and the current value prediction, obtained for every sample ($s,a,R_t$) in the minibatch. In other words your network is going to get "corrected", for every prediction it has made, by the difference between estimation and ground truth values.


What algorithms should I choose for a recommender system and why?

Short answer:

It depends on your data. What do you want to do ?

Longer answer:

Use content based approaches if you have data on your items. Use collaborative approaches is you have data on your users. Use both if you have both. I would say content based approaches are general machine learning problems (how do I extract meaningful information from data) whereas collaborative filtering is really a recommender system specific work (how user's behavior can suggest users/items similarity/connections).
Well you can. Neural nets are just a kind of algorithm, you surely can use them for content based analysis, and it might be possible to use them to enhance your collaborative algorithm.
NN use texts and images as numerical data, so I don't understand your question.

If you want a good insight in today's recommender systems, take a look at this article.


help understanding nested cross validation

Generally, you consider the outer CV as just estimating the performance of the method of fitting.  You don't expect a final model out at this point.  You can then apply the same method of fitting a model (what was happening in the inner CV) to the entire dataset to produce a model, and you say that you expect the performance estimate to apply to this new model.

https://stats.stackexchange.com/a/65158/232706
https://datascience.stackexchange.com/a/16856/55122
https://stats.stackexchange.com/q/11602/232706
https://stats.stackexchange.com/q/52274/232706


Keras/TF: Making sure image training data shape is accurate for Time Distributed CNN+LSTM

As I understand your question correctly, you worry, that change of input shape (adding time-dimension) will affect your output shape. If so, you're wrong, the output is independent of input. The output shape depends only on the structure of a neural network.


Has anyone been able to run Tensorflow with GTX 1070 GPU on Ubuntu 16.04/15.10/15.04?

I was in the same boat, but now that I have figured it out, I have listed the steps for installing tensorflow 0.9 with cuda toolkit 8.0, cudnn 5.1, bazel 0.3 on Ubuntu 16.04 LTS here: http://abhay.harpale.net/blog/machine-learning/deep-learning/getting-tensorflow-to-work-with-gpu-nvidia-gtx-1080-on-ubuntu-16-04-lts/

Here's the gist

Install NVidia Cuda Toolkit
Install NVidia CuDNN
Install Tensorflow dependencies such as swig, python-dev, numpy, python-wheel, zlib1g, g++
Configure and build tensorflow using Bazel


metric accuracy for linear regression and non linear regression

Normally the evaluation measure doesn't depend on the method used, it depends primarily on the task being carried out.

Of course there are standard evaluation measures associated with broad types of tasks, such as classification or regression. There are technical constraints to take into account, for example whether the output is categorical or numerical. It's a common mistake to overlook this question and/or treat it as purely technical, but the choice of an appropriate evaluation setup should be made based on how well it represents the quality of the output of the ML process.

To answer your question: if the task is the same, the evaluation measure should be the same whether the method is linear regression or non-linear regression. Otherwise one would be measuring only some technical aspect specific to the method, not estimating the quality of the output in a comparable way.


I've solved the problem with three changes:

The weights are too small at the beginning, remove the scaling
I think the biases should also be randomly initialized:

w1 = np.random.randn(S1, S0)
b1 = np.random.randn(S1, 1)
w2 = np.random.randn(S2, S1)
b2 = np.random.randn(S2, 1)

Your are using a fixed learning rate of 0.03, change it to use the learning rate, and you can also increase it:

    learning_rate = 0.1 ...

    w1 = w1 - dW1 * learning_rate
    w2 = w2 - dW2 * learning_rate
    b1 = b1 - dB1 * learning_rate
    b2 = b2 - dB2 * learning_rate

The image below shpws what I get for the progression of J:

Just for curiosity, if we change the sigmoid function with the ReLU, take a look to what happens to J (in the code I didn't change the name of the defs...). The learning with ReLU is much faster.

:

    def sigmoid(z):
    return np.maximum(0,z)

    def sigmoid_derivative(x):
    x[x < = 0] = 0
    x[x > 0]  = 1
    return x


How do I calculate for each person in the network how many people agree with their opinion?

As a fast answer, you can represent each student as a vector with $K$ elements (where $K$ is the number of topics) and values $\{+1, 0, -1\}$, denoting positive/non-existent/negative opinion about this topic.

Then, a simple measure of agreement between two students is the element-wise product between two student-vectors. That is the product will be:
$similarity = \sum_{i=1}^{K}st_1[i]*st_2[i]$, where $st_1,st_2$ are the student-vectors. Obviously, only the topics where both students have aligned opinions will boost the total [e.g. $1*1=1$ and $(-1)*(-1)=1]$, while misaligned opinions will decrease the sum. If any of the two students haven't expressed an opinion about a topic, then this topic won't matter in the sum.

In that sense, you can find the most like-minded students to a specific student, as the ones with the highest $similarity$. If what you really need is a number of agreeing students for each unique student, then a threshold on the $similarity$ score can be set. The value of the threshold can be decided empirically from your data.

This is easily implemented and if you are comfortable with coding, I could post a sample script in python. One thing to consider though, is in what format is the bipartite graph (a .csv, a graph file of some kind etc.).

EDIT: MINOR EXAMPLE. Fetch example .csv file used from here.

import pandas as pd
import numpy as np

# Change location of file according to your needs
with open('students_example.csv', 'r') as f:
    df = pd.read_csv(f)
# Print for visualization
print(df.head())
print("~"*25)

# Delete column containing the student_id
del df['Student_ID']
# Parse the pandas DataFrame as matrix
student_vectors = df.as_matrix()
# The number of students at hand, let it be N.
N_students = student_vectors.shape[0]
# Initialize empty matrix of similarity between students
# Its size will be NxN (each student with each other)
similarity_scores = np.zeros((N_students, N_students))
# Iterate over each student vector and calculate the
# similarity with all students
for i, student in enumerate(student_vectors):
    # Reshaping and transposing to get the dot product between each student
    # And all the student vectors
    similarity_scores[i,:] = np.dot(student.reshape(1,-1), student_vectors.T)
# Fill the diagonal (that is the similarity of each student with him/herself)
# with low similarity scores so as not to confuse them with other possibly
# agreeing students
np.fill_diagonal(similarity_scores, -1000)

# Random wanted student for example purposes
wanted_id = 3
# Print Students Opinion
print("Wanted Students Opinion:")
print(df.loc[wanted_id].to_string())
print("~"*25)
print("Most similar:(Student ID = %d)"% np.argsort(similarity_scores[wanted_id,:])[::-1][0])
print df.loc[np.argsort(similarity_scores[wanted_id,:])[::-1][0]].to_string()
print("~"*25)
print("Second most similar:(Student ID = %d)"% np.argsort(similarity_scores[wanted_id,:])[::-1][1])
print df.loc[np.argsort(similarity_scores[wanted_id,:])[::-1][1]].to_string()
print("~"*25)

If you follow the example, the output for the wanted student (with $student_{ID}=3$) with opinions: {Trump -1, Net Neutrality   -1,Vaccination 1, Obamacare -1}

will give you two other students with the same opinions and their ids.

You can modify the script to fit your needs accordingly.

P.S.: Sorry for the messy code, it was written rather hastily. Also, i tried it with Python 2.7.


CNN computing time on good CPU vs cheap GPU

According to this guy, he got a 15x increase from Intel i7 to GeForce 1070.

You also may consider using AWS. You can use a machine 100x as powerful (as a single 1070) and your employer may find it attractive because the upfront sunk cost is zero.


The idea is that the faster you can isolate a sample the higher the chance it is an outlier/anomaly.

Here is one possible scenario for your example:

x1 = [2, 1, 4, 6, 4, 2, 1, 2, 3, 4, 19]

Draw a random value between min=1 and max=19; e.g. 10
Split according to selected value:

x1_left = [2,1,4,6,4,2,1,2,3,4]
x1_right = [19]

Repeat on subsets

We already isolated 19 after one step whereas the other samples "to the left" are not yet isolated and haven't changed much so far. Any threshold sampled greater than 6 would isolate 19 in a single split which has a probability close to 70%.

To make it more obvious, say x1=[1,2,3,4,5,10000000000]. Sampling a value between 1 and 10000000000 will almost surely give a result > 5 and therefore immediately isolate 10000000000.


How can you make use of Json format data?

Here is a tutorial by Corey Schafer on working with JSON data. He explains in detail how to convert JSON data into python objects, but I don't think he explains how to convert them to excel here. But if you are familiar with pandas, you should be able to create dataframes and then convert them to excel files as needed.


PCA method for feature selection - How to solve the raise Exception error ("Data must be 1-dimensional")?

pca.components_ is a 2d array, so you cannot make a Series out of it.


I think basic support vector machine means hard-margin SVM. So, let's review:
What is a Hard-Margin SVM
In short, we want to find a hyperplane with the largest margin which be able to separate all observations correctly in our training sample space.
The optimisation problem in hard-margin SVM
Given the above definition, what is the optimisation problem which we need to solve?

The largest margin hyperplane: We want max(margin)
Be able to separate all observations correctly: We need to optimise margin and also satisfy the constraint: No in-sample errors

What happens when we train a linear SVM on non-linearly separable data?
Back to your question, since you mentioned the training data set is not linearly separable, by using hard-margin SVM without feature transformations, it's impossible to find any hyperplane which satisfies "No in-sample errors".
Normally, we solve SVM optimisation problem by Quadratic Programming, because it can do optimisation tasks with constraints. If you use Gradient Descent or other optimisation algorithms which without satisfying the constraints of hard-margin SVM, you should still get a result, but that's not a hard-margin SVM hyperplane.
By the way, with non-linearly separable data, usually we choose

hard-margin SVM + feature transformations
use soft-margin SVM directly (In practical, soft-margin SVM usually get good results)


Is it good practice to always remove highly correlated variables?

(1) No. For example, if you have computer vision problems, then each pixel of the image is a feature. most neighboring pixels are highly correlated.

For example, see the following image from The HASYv2 dataset:


Dealing with diverse groups in regression

You can't really do that, there may be some factor which binds certain "groups" of data together, but there are many reasons for this. Your relationship may be nonlinear, or the "groups" of data may represent subjects / objects, where a stronger correlation exists. Unless you know for a fact that these points belong to different populations you shouldn't do that, use the data that you have to model these groupings.


How to match features in new records for NLP BOW

If there is new data with labels, you have to retrain the existing model to learn the new relationships.


How to feed key-value features (aggregated data) to LSTM?

Sounds to me like you can reshape the input of a timestep as matrix s.t. $X_t \in \mathbb{R}^{n \times 2}$ (assuming you only have a and b as features) and use Convolutional LSTM layers:
$t=0$:
$X_0 = \begin{bmatrix}a(0,0)&b(0,0)\\a(0,1)&b(0,1)\\\vdots&\vdots\\a(0,n)&b(0,n)\end{bmatrix}$
$t=m$:
$X_m = \begin{bmatrix}a(m,0)&b(m,0)\\\vdots&\vdots\\a(m,n)&b(m,n)\end{bmatrix}$
Basically each sequential keys $y$ is a row in your input matrix. You can think of each matrix as some kind of image you feed to your LSTM network.
Hope that helps you any further.


Selection of base model for transfer learning

There is no specific rule associated with the base model selection for transfer learning. It is generally a trade-off between model precision and resource allocation. As the number of layers increases, the number of parameters increases and the model becomes more and more resource heavy but deeper models tend to have better accuracy over shallower counterparts. Here's a comparison:

Apart from that also refer to this question: Which is the fastest image pre-trained model? if the speed of the model matters.


I see that this threshold always matches the percentage of observations equal to 1 in my original data. Is there any conceptual explanation for this?

Yes, although the fact it always matches exactly is probably a coincidence or maybe due to a small sample.

The training data contains a proportion $p$ of instances labelled 1. From the ROC plot you can see all the possible values for setting the threshold at a certain level and the resulting performance; for every possible level you can calculate the corresponding proportion $q$ of instances predicted as 1:

if $q$ is much lower than $p$, then the system predicts many 0s, so there are many false negative errors and that makes the recall lower. Precision is high in this case.
if $q$ is much higher than $p$, then the system predicts many 1s, so there are many false positive errors and that makes the precision lower. Recall is high in this case.

I assume that you optimize on the F1-score right? The fact that the F1-score is based on the product of the precision and recall means that both values need to be reasonably high, otherwise the F1-score drops. As seen above, having very different values for $p$ and $q$ will cause either the precision or recall to be low. Therefore the optimal F1-score is achieved when $q$ is close to $p$.


How can I improve the recall of a certain class in a multiclass-classification result

Recall is the ability of a search model to find the correctly labeled items amongst all the items for a given query.

One common method to improve specific query results is to create a custom model. Create a model just for "Dermatology". That model can be tuned to increase recall without impacting other queries that can use the generic search model.


Is it possible to (de)activate a specific set of cells in jupyter?

Welcome to DataScience.SE!

This is not currently possible. You could change the cells to Raw.


Matrix properties and machine learning/data mining

The importance of a concept in mathematics depends on the circumstances of its application. Sometimes, its importance relies on the fact that it allows you to carry on with what you are doing.

For example, you usually need column independence (independent variables between predictors) because multiple regression will behave badly with highly correlated variables. Even worst, when some of your columns (or rows) are dependent, your matrix is not invertible. Why? Because matrix inversion A^-1 involves the determinant 1/|A|, which is 0 when columns or rows are linearly dependent.

Eigenvalues is a common occurrence in calculations related to maximization/minimization in machine learning. Let's say you are interested in principal component analysis. A very important idea there is dimensional reduction (you have a dataset with many variables and want to reduce the number of variables without losing too much explanatory power.) One solution is to project your data onto a lower dimensional space (e.g. taking your data with 50 variables and reducing them to 5 variables.) Turns out a good projection to use is one that includes as much variation as possible and maximization of this variation results in the eigenvalue equation S u  = λ u.

In other cases, you explicitly include the eigenvalue equation of some quantity of interest because in doing so, you're changing the coordinate system in which you represent the variables. Take the case of a (multivariate) Gaussian distribution in which the argument of the exponent is given by Δ = (x-μ)^T Σ (x-μ). If you consider the eigenvalue equation of Σ, the exponent can be written as Δ = y_1^2 / λ_1 + y_2^2 / λ_2 (in two dimensions) This is the equation of an ellipse only if λ_1 and λ_2 are positive. Therefore, you obtain the following graphical interpretation (Bishop, PRML, p.81):

Positive semi-definite matrices are used as a matter of convenience. They are well-behaved and well-understood. For instance, their eigenvalues are non-negative, and if you remember the previous paragraph, the argument Δ required positive eigenvalues. By now, you can see why some concepts are very popular: You need them for your calculations or they need each other.

I can recommend a couple of books:

Linear Algebra: A Modern Introduction by David Poole
Understanding Complex Datasets: Data Mining with Matrix Decompositions  by David Skillicorn.

The second recommendation is more specialized and requires a decent understanding of the basics, but it is of great help to understand matrix decompositions.


Does it make sense to combine PCA with an artificial neural network?

Neural networks are actually extremely effective at performing dimensionality reduction. A great example is word2vec, which applies a shallow neural network to reduce inputs on the order of several million features (i.e. unnormalized text) to a 30-150 dimensional vector via a process that is mathematically analogous to matrix factorization (which is the class of techniques PCA belongs to). Autoencoders function very similarly to word2vec: if you're planning on using an autoencoder to learn an embedding of your data, I wouldn't expect you'd gain anything from applying PCA first rather than using an autoencoder (or something similar) to learn a better embedding for your data that isn't constrained by the assumptions of PCA.


Organization of layers in Keras for a NLP problem

In this kind of problems, one needs to be sure what model he/she wants to use. A convolution network would be great with final softmax layer giving 8 outputs. you don't need a bidirectional layer in your network architecture. Try a simple conv network and make sure your data pipeline is working then you can add more layers and make it complex.


Is it a good idea to combine fine tuning and feature extraction techniques?

The concept of multimodal learning is relevant: in this case, combining data from two modalities: 1) image signal using ResNet50 and 2) genomic features extracted from genes. Multimodal learning for extending state-of-the-art performance of pre-trained unimodal models is currently an area of active research in the literature. In the NLP domain, the paper on Multimodal Bert describes combining a pre-trained language model with additional signals from visual and auditory inputs for improved classification performance.
In the specific scenario mentioned here, combining the image and non-image signals into a single deep neural network graph would allow for additional fine-tuning. It is possible to first train the layer combining signals from both modalities, then later fine-tune all weights in the graph (with a lower learning rate) for improved performance. This procedure discussed a bit in the Keras Guide to Transfer Learning.


How to calculate time difference in between rows using loop in panda python

I realize that this has already been answered, but I thought I would propose another solution that takes advantage of vectorization.  It should perform better if there are a lot of records by avoiding iteration.

First, I'll create a test data set:

import pandas as pd
import datetime
from tabulate import tabulate
import numpy as np

start_date = datetime.datetime(2018, 1, 1, 00, 0, 0)
end_date = datetime.datetime(2018, 12, 31, 00, 0, 0)

duration = (end_date - start_date).total_seconds()
custom_index = range(0, 20)

duration_df = pd.DataFrame(columns=['Random Duration'], index=custom_index)
duration_df['Random Duration'] = duration

duration_df['Random Duration'] = duration_df['Random Duration'].apply(lambda x: np.random.randint(0, x))
duration_df['Random Date'] = duration_df['Random Duration'].apply(lambda x: start_date + datetime.timedelta(seconds=x))
duration_df = duration_df.drop(columns='Random Duration')
duration_df = duration_df.sort_values('Random Date')
duration_df = duration_df.reset_index(drop=True)

Now we can create the new column with this one liner:

duration_df['Hours Since Previous'] = (duration_df['Random Date'] -
                                       duration_df['Random Date'].shift(1)).astype('timedelta64[h]')

This will give you the following results:

    Random Date            Hours Since Previous
--  -------------------  ----------------------
 0  2018-01-16 21:56:47                     nan
 1  2018-01-24 17:54:12                     187
 2  2018-02-13 19:30:24                     481
 3  2018-02-21 04:18:23                     176
 4  2018-03-05 02:07:50                     285
 5  2018-03-13 20:18:30                     210
 6  2018-04-16 09:05:13                     804
 7  2018-04-17 20:30:46                      35
 8  2018-05-17 02:39:36                     702
 9  2018-05-20 16:43:20                      86

How this works

duration_df['Random Date'].shift(1) creates a new series of dates that are offset by one row.

(duration_df['Random Date'] - duration_df['Random Date'].shift(1)) creates a pandas.Timedelta() object and .astype('timedelta64[h]') converts the resulting Timedelta to hours.


Which tool is good to collect tweets on 50 keywords over the last 5 years and then analyze them with the LDA algorithm or sentiment analysis?

As for Algorithmia and monkeylearn, I don't know if you will find much help for those here due the fact that Python and R/RStudio are the predominant 'data science' open source (read: free) platforms. Otherwise, if Algorithmia or monkeylearn has tutorials go for it.
The project you are interested in doing is not trivial (with no experience) but completely within the grasp of a novice given enough time. May I suggest starting with R/RStudio. This is simply my opinion based on my learning path. "Your mileage may vary!" Then find some tutorials such as Towards Data Science and see how it goes.


Why Deep Reinforcement Learning fails to learn how to play Asteroids?

I suspect a bug, or some subtle implementation detail.
In many ways, Asteroids is as near an ideal environment as one can get without custom design:

Unambiguous, simple reward system (get points, don't die)
limited options (move or shoot a single weapon type)
all enemies are very similar
The player has perfect information.

Compare this to DOTA 2, which has not been mastered, but with a moderate reduction in complexity (1v1 instead of 5v5), OpenAI was able to achieve some impressive results, despite being magnitudes of order more complex than asteroids.
There are certain compromises made in the 2015 DQN paper, for example:

"Following previous approaches to playing Atari2600 games,we also use
a simple frame-skipping technique (15). More precisely, the agent sees
and selects actions on every kth frame instead of every frame, and its
last action is repeated on skipped frames. Because running the
emulator forward for one step requires much less computation than
having the agent select an action, this technique allows the agent to
play roughly k times more games without significantly increasing the
runtime.
We use k - 4 for all games"

As a counter-example, top level human players often make single frame level decisions / actions in modern fighting games (rendered both visually and in game logic at 60 FPS), so we know this approach will not work well with all video games.
I suspect that even though these were effective generalized approaches, one of them severely failed with asteroids.
This is a good question, as diagnosing why asteroids is causing difficulty could give some strong insight into testing procedures or algorithm design, as clearly something is happening. I think to get an accurate answer would require fixing the problem.
Sources


Finally I found the answer. In elastic net the cost function is written as
$J(\theta)=MSE + r\alpha\sum_{i=1}^n \vert\theta_i \vert + \frac {1-r}{2}\alpha\sum_{1=1}^n \theta_i ^2 $. Here $r$ is the ratio, $\alpha$ is the hyperparameter and $n$ is the number of features. The ratio slider controls $r$ and $\alpha$ slider controls the value of hyperparameter.
If the ratio $r=0$, then Elastic net regression is equal to ridge regression. For $r=1$, the Elastic Net Regression is equal to Lasso regression. I hope this perfectly answers the question I posed. I would like to have comments from followers of this post.


Word list as a baseline for measuring a classifier's performance?

Features of your naive bayes classifier is most likely a bag of words where each feature represents a single word.
The naive bayes classifier also assumes that each feature (or word) contributes independently to the final prediction.
So if your sentence is composed of a single word - you get an idea of how the model reacts to that word.


Gradient Boosting Tree: "the more variable the better"?

My logic is that because these noise variables do NOT give maximum gain split at all, then they would never be selected thus they do not influence the tree growth.

This is only perfectly correct for very large, near infinite data sets, where the number of samples in your training set gives good coverage of all variations. In practice, with enough dimensions you end up with a lot of sampling noise, because your coverage of possible examples is weaker the more dimensions your data has.

Noise on weak variables that ends up correlating by chance with the target variable can limit the effectiveness of boosting algorithms, and this can more easily happen on deeper splits in the decision tree, where the data being assessed has already been grouped into a small subset.

The more variables you add, the more likely it is that you will get weakly correlated variables that just happen to look good to the split selection algorithm for some specific combination, which then creates trees that learn this noise instead of the intended signal, and ultimately generalise badly.

In practise, I have found XGBoost quite robust to noise on a small scale. However, I have also found that it will sometimes select poor quality engineered variables, in preference to better-correlated data, for similar reasons. So it is not an algorithm where "the more variables the better for XGBoost" and you do need to care about possible low-quality features.


Is logistic regression actually a regression algorithm?

Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a model, I'm going to start with the model in order to show how this is truly a regression problem.

In logistic regression, we are modeling the log odds, or logit, that an event occurs, which is a continuous quantity. If the probability that event $A$ occurs is $P(A)$, the odds are:

$$\frac{P(A)}{1 - P(A)}$$

The log odds, then, are:

$$\log \left( \frac{P(A)}{1 - P(A)}\right)$$

As in linear regression, we model this with a linear combination of coefficients and predictors:

$$\operatorname{logit} = b_0 + b_1x_1 + b_2x_2 + \cdots$$

Imagine we are given a model of whether a person has gray hair. Our model uses age as the only predictor. Here, our event A = a person has gray hair:

log odds of gray hair = -10 + 0.25 * age

...Regression! Here is some Python code and a plot:

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

x = np.linspace(0, 100, 100)

def log_odds(x):
    return -10 + .25 * x

plt.plot(x, log_odds(x))
plt.xlabel("age")
plt.ylabel("log odds of gray hair")

Now, let's make it a classifier. First, we need to transform the log odds to get out our probability $P(A)$. We can use the sigmoid function:

$$P(A) = \frac1{1 + \exp(-\text{log odds}))}$$

Here's the code:

plt.plot(x, 1 / (1 + np.exp(-log_odds(x))))
plt.xlabel("age")
plt.ylabel("probability of gray hair")

The last thing we need to make this a classifier is to add a decision rule. One very common rule is to classify a success whenever $P(A) > 0.5$. We will adopt that rule, which implies that our classifier will predict gray hair whenever a person is older than 40 and will predict non-gray hair whenever a person is under 40.

Logistic regression works great as a classifier in more realistic examples too, but before it can be a classifier, it must be a regression technique!


Putting a predictive model into production

Depending on what exactly you mean by framework, I would argue that there is. Using a REST interface to serve a production model at inference time is pretty close to a general serving framework at this point.

However, there are a potentially infinite number of use cases that would diverge from this standard even if one categorically existed. Some mobile apps using image recognition models on personal data that the user doesn't want transmitted to the cloud are pushing the envelope of ways to compress formerly huge neural network architectures into ones that easily fit into mobile-sized memory, for example.

If you're talking software framework, it shouldn't be surprising that a field of programming practice (or, rather, plethora of fields) that find needs for a variety of languages and even ecosystems within languages would have difficulty forming a consensus around a single aporoach.

It's likely that deployment frameworks have yet to build in enough benefit to justify a reduction in general flexibility to justify their use.


Implementation of Image Recognition techniques
There are lots of open source libraries available for the image recognition and classification.
You can make use of the "TensorFlow" library for image recognition and can be integrated with your android application.


Am I overfitting my random forest model (more information in description)?

This is more like a comment than an answer but my newbie status doesn't allow me to comment yet.

You might be interested in reading this article:
Random forest

4 samples appears to me being low a number of samples,  I would recommend that you try using more samples and compare your results.

Besides, there are some good implementation of the algorithm out there, are you trying to re-implement it for academic purposes?


Does the add_regressor method on Facebook Prophet also work with categorical variables?

From my understanding Prophet is just a linear regression library that helps to analyze time series with some nice features (like holidays, Fourier transformation, etc.). So from the mathematical standpoint, the regressor must be an ordinal scaled value. The docstring also implicitly says something about it. See the keywords 'additive' and 'multiplicative'. Categorical data is neither of the two. A category is a nominal scale, which can only be counted and if it's ranked, it can be sorted.
Have a nice day.


Gumbel-Softmax trick vs Softmax with temperature

Let's say you have two states, $X_1$ and $X_2$, and you have a model, $M$, that produces a score $M(X_i)$ for each state (i.e, the logits). Next you can use the logits to compute some distribution

$$P = softmax(\{M(X_1), M(X_2)\})$$

and take the state with the highest probability

$$X=argmax_{X_i}(P)$$

But what if you actually want to sample from $P$ instead of just taking the argmax - and you want the sample operation to be differentiable! This is where the Gumbel Trick comes in - instead of softmax, you compute

\begin{equation}
\ X = argmax_{X_i}(\{M(X_i)+Z_i\})
\end{equation}

Where $Zi$ are i.i.d Gumbel(0,1). It turns out that $X$ will be equal to $X_1$ about $P(X_1)$ of the times and to $X_2$ about $P(X_2)$ of the times. In other words, the equation above samples from $P$.

But it still not differentiable because of the argmax operation. So instead of doing that we'll compute the Gumbel-Softmax distribution. Now if the temperature is low enough then the Gumbel-Softmax will produce something very close to one hot vector, where the probability of the predicted label will be 1 and other labels will have a probability of zero. So for example, if the Gumbel-Softmax gave the highest probability to $X_1$, you can do:

\begin{equation}
\ X = \sum_{x_i} P_g(X_i)*X_i = 1*X_1 + 0*X_2 = X_1
\end{equation}

Where $P_g$ is the Gumbel-Softmax operation. No argmax is needed! So with this cool trick we can sample from a discrete distribution in a differentiable way.


Interpretation of accuracy score on subset of data points

Usually when working with classification problems, one tries to have 3 subsets of data:

A training set: this subset is usually the biggest one and can take up to ~80% of the available data. It is used to train the chosen algorithm, using the known labels of each data sample.
A validation set: this subset is much smaller. It will typically be ~5-10% of the available data. It is used to evaluate the performance of the algorithm trained on the training set. Typically, one will tune the parameters of the algorithm in order to reach the best performance on the validation set.
A testing set: this subset is of the same size order as the validation set or bigger. Very important: it should NEVER be used for training purpose! Once the model is trained and tuned using the training and validation sets, the testing set allows one to get the accuracy (or any other performance metric) on unseen data. If the model generalizes well, the score will be close to the ones seen on the validation set, often times a tiny bit worse.

In order for this to work properly, it is important that all the subsets are representative of the available data. For example, that the proportion of each class is approximately the same across the subsets.

In the light of such broadly used process, we can see that most algorithms are tuned and tested on a fraction of the available data. As long as the used test set is balanced in a way that is similar to the training and validation sets and that it has not been used at all during the training/tuning of the model, there is no reason why the performance scores would not generalize well to N > n samples used in the test set.


Is it common practice when working with text?

No, you could split dataset as you wish, in general in real-world problem you should use cross validation.

  Does it have anything to do with using a "pre-trained" TensorFlow Hub layer?

No, it doesn't.


Non-mutal exclusive classification task examples

For example, due to the complexity of the images in the ImageNet database. Algorithms will often use hundreds or thousands of output nodes to be capable of classifying a large array of different things. Researchers also relax the cost function and allow the $k$ highest outputs to be considered. If one of these is correct then the example is considered to have been correctly classified. Furthermore with the existence of such complex data, often times a certain object might be a subset of another object.

For example, consider the picture of a human being. If the algorithm detects a human with the highest activation at the output node, but also has high activations for hands, eyes, face. This is not wrong! The image does in fact contain these objects as well and them being detected by the model is encouraging.


combining trained neural nets in tensorflow

It seems merging two neural networks does not make any sense. You may instead train one deep R-CNN or with just 2 different NN trained, You may classify as for a test case,if outputs of both NN agree, then output the common output, else the NN which outputs 1, its precision and NN which outputs -1, ratio of its correct negatives to total cases which it predicted negative(calculated in training), whichever is greater, output that NN's output.
Hope these articles will help you.
Build Your First Deep Learning Classifier using TensorFlow: Dog Breed Example
Deep Learning Part 3: Combining Deep Convolutional Neural Network with Recurrent Neural Network


How to make my Neural Netwok run on GPU instead of CPU

First some stupid sanity-check questions: do you have a GPU in your local machine? (you didn't mention that explicitly). I ask because it will not work e.g. on an integrated Intel graphics card found in some laptops.

Second, you installed Keras and Tensorflow, but did you install the GPU version of Tensorflow? Using Anaconda, this would be done with the command:

conda install -c anaconda tensorflow-gpu

Other useful things to know:

what operating system are you using? (I assume Linux e.g. Ubuntu)
what GPU do you expect to be shown as available?
Can you run the command: nvidia-smi in a terminal and update your question with the output?

If you have installed the correct package (the above method is one of a few possible ways of doing it), and if you have an Nvidia-GPU available, Tensorflow would usually by default reserve all available memory of the GPU as soon as it starts building the static graph.

If you were not already, it is probably a good idea to use a conda environment, which keeps the requirements of your model separate from whatever your system might already have installed. Have a look at this nice little walkthrough on getting started - this will likely be a good test to see if your system is able to run models on a GPU as it removes all other possible problems created components unrelated to your script. In short, create and activate a new envrinment that includes the GPU version of Tensowflow like this:

conda create --name gpu_test tensorflow-gpu    # creates the env and installs tf
conda activate gpu_test                        # activate the env
python test_gpu_script.py                      # run the script given below

UPDATE

I would suggest running a small script to execute a few operations in Tensorflow on a CPU and on a GPU. This will rule out the problem that you might not have enough memory for the RNN you're trying to train.

I made a script to do that, so it tests the same operations on a CPU and GPU and prints a summary. You should be able to just copy-paste the code and run it:

import numpy as np
import tensorflow as tf
from datetime import datetime

# Choose which device you want to test on: either 'cpu' or 'gpu'
devices = ['cpu', 'gpu']

# Choose size of the matrix to be used.
# Make it bigger to see bigger benefits of parallel computation
shapes = [(50, 50), (100, 100), (500, 500), (1000, 1000)]

def compute_operations(device, shape):
    """Run a simple set of operations on a matrix of given shape on given device

    Parameters
    ----------
    device : the type of device to use, either 'cpu' or 'gpu'
    shape : a tuple for the shape of a 2d tensor, e.g. (10, 10)

    Returns
    -------
    out : results of the operations as the time taken
    """

    # Define operations to be computed on selected device
    with tf.device(device):
        random_matrix = tf.random_uniform(shape=shape, minval=0, maxval=1)
        dot_operation = tf.matmul(random_matrix, tf.transpose(random_matrix))
        sum_operation = tf.reduce_sum(dot_operation)

    # Time the actual runtime of the operations
    start_time = datetime.now()
    with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as session:
            result = session.run(sum_operation)
    elapsed_time = datetime.now() - start_time

    return result, elapsed_time

if __name__ == '__main__':

    # Run the computations and print summary of each run
    for device in devices:
        print("--" * 20)

        for shape in shapes:
            _, time_taken = compute_operations(device, shape)

            # Print the result and also the time taken on the selected device
            print("Input shape:", shape, "using Device:", device, "took: {:.2f}".format(time_taken.seconds + time_taken.microseconds/1e6))
            #print("Computation on shape:", shape, "using Device:", device, "took:")

    print("--" * 20)

Results from running on CPU:

Computation on shape: (50, 50),       using Device: 'cpu' took: 0.04s
Computation on shape: (500, 500),     using Device: 'cpu' took: 0.05s
Computation on shape: (1000, 1000),   using Device: 'cpu' took: 0.09s
Computation on shape: (10000, 10000), using Device: 'cpu' took: 32.81s

Results from running on GPU:

Computation on shape: (50, 50),       using Device: 'gpu' took: 0.03s
Computation on shape: (500, 500),     using Device: 'gpu' took: 0.04s
Computation on shape: (1000, 1000),   using Device: 'gpu' took: 0.04s
Computation on shape: (10000, 10000), using Device: 'gpu' took: 0.05s


How do you apply SMOTE on text classification?

Actually NLP is one of the most common areas in which resampling of data is needed as there are many text classification tasks dealing with imbalanced problem (think of spam filtering, insulting comment detection, article classification, etc.). But SMOTE seem to be problematic here for some reasons:

SMOTE works in feature space. It means that the output of SMOTE is not a synthetic data which is a real representative of a text inside its feature space.
On one side SMOTE works with KNN and on the other hand, feature spaces for NLP problem are dramatically huge. KNN will easily fail in those huge dimensions.

So I can propose you two approaches:

Do not care about the real text representation of new synthetic samples which I assume should be fine. You need to balance the distribution for your classifier not for a reader of text data. So apply SMOTE as traditional (however I usually use the solution 2 bellow so I do not gaurantee the result!) with some Dimensionality Reduction step.

  1) Lets assume you want to make your data samples from minor class double using 3-NN. Ignore the major class(es) and keep only minor class samples.

  2) For each sample point in feature space choose 5 nearest neighbors. Then choose 3 of them randomly (isn't it unnecessary complicated? if I didn't want to explain the original algorithm I would say just choose 3 neighbors!)

  3) For each dimension calculate the distance between sample and neighbors and multiply it in a random number between 0-1 and add it to the original value of sample in that dimension. (this complicated paragraph simply means for each dimension choose a random value between the original sample and that neighbor!)

But I usually do another oversampling which is on the text (so more intuitive) and is kind of SMOTE.

  1) Ignore the major class. Get a length distribution of all documents in minor class so that we generate new samples according the the true document length (number of words/phrases). We assume we want to make the size of class triple (so producing $k=2$ synthetic documents per original document)

  2) Generate a sequence of $n$ random integers according to that distribution. It's used for determining the length of new synthetic document.

  3) For each document: Choose one integer from random length sequence and $m$ random document whose length is close to the integer. Put tokens of all $m$ documents in a set and randomly choose $n$ tokens $k$ times. these are your $k$ new documents.


Autoencoder and Neural Network Overfitting in terms of parameter number?

Additional drop out layers do not reduce parameters. Drop out is applied after each sample independently. This is done to Limit overfitting. So you still have the same parameters.

From over 6000 dimensions to only 2 seems a bit too much. But it is hard to see. You could run a PCA Analysis and use that as a starting point for the number of hidden neurons, e.g. get the dimensionality that Accounts for 75% of the Variation in your data and go from there. Also try a second layer and add some form of regularization.