Assign a unique cluster based on a dataframe column with KMeans Algorithm

Very interesting question! I try my best:

It depends a bit on the number of clusters and number of restaurant but in general I explain a bit.

If the number of restaurants and clusters are the same, then, theoretically, your question has just one trivial answer: "each restaurant is a cluster". You even don't need any algorithm. I go a bit deeper on it.

Most of ML algorithms solve an optimization problem to find the answer. Sometimes optimization problems are subject to some constraints.

Example:

Cluster restaurants such that all similar restaurants are necessarily assigned to the same cluster.
Cluster restaurants such that the density of same restaurants in same clusters is maximum.

The first one has the trivial answer I gave before but second one can be solved. You run several clustering methods (or just k-means but with several initial conditions) and accept the one in which higher number of similar restaurants are in identical clusters. For this you need to convert "density of same restaurants in same clusters" to a mathematical formulation and use it as criterion of choice. If you need help on it just drop a comment so I update the answer.

In any case, you change the output of the clustering and you don't let it "naturally" find the clusters as you push a criterion which is not normally considered in the algorithm. But don't worry! The good thing is that at least you have a criterion for "goodness" of your clustering which does not exist normally in clustering problem.

UPDATE

Let's try $\chi^2$ test first. It is pretty simplified but try it and if it didn't work we can think of something else. For know how, I prepared it for you in a simple way so you don't get confused with different tutorials on the net.

Imagine you have 4 restaurants and you want 4 clusters. You will end up with such a frequency table which says how many restaurants of which type fall in which cluster:

Then in Python, you simply calculate $\chi^2$ statistic which tells you if clusters and restaurants "are correlated or not".

from scipy.stats import chi2_contingency
obs = np.array([[10,1,2,1], [1,11,0,1], [1,2,8,1], [0,2,2,12]])
chi, p, _,_ = chi2_contingency(obs)
print('The chi-square statistic of {} with p-value of {}'.format(chi,p))

P-value, as you know, tells you if the statistic is significant. There theoretical consideration in this solution but I am not going to confuse you with that. I apologize as I did not go through your proposal in the comment. Will answer accordingly as soon as I find time to have a look at that.

Good Luck!


Interpretation for test score , training score and validation score in machine learning?

Interpretation for test score , training score and validation score ?
what they actually tell us?

We usually divide our data-set in 3 parts. Training-data, validation-data and test-data. Then we analyze the score:
Training Score: How the model generalized or fitted in the training data. If the model fits so well in a data with lots of variance then this causes over-fitting. This causes poor result on Test Score. Because the model curved a lot to fit the training data and generalized very poorly. So, generalization is the goal.
Validation Score This is still a experimental part. We keep exploring our model with this data-set. Our model is yet to call the final model in this phase. We keep changing our model until we are satisfied with the validation score we get.
Test Score This is when our model is ready. Before this step we have not touched this data-set. So, this represents real life scenario. Higher the score, better the model generalized.

What's an acceptable difference between cross test score , validation
score and test score?

I think the "cross test score, Validation score" there are no difference. The right naming is "cross validation score". Cross validation is used by interchanging the training and validation data in different ways multiple times. A image that describes (Copyright reserved by towardsdatascience.com):

So your question should be "What's an acceptable difference between cross validation score and test score?" There's no straightforward answer to this question. Here's a long discussion you might want to consider: Cross validation vs Test set.

If difference between test score and training score is small mean it
is a good model/fit?

Yes!! This is what we strive for. And often times to achieve this, We require various engineering techniques, calculations and parameter tuning.

overfitting and under fitting on the basis of test score , training
score and validation score?

Usually, high training score and low test score is over-fitting. Very low training score and low test score is under-fitting. First example here, in technical term is called low bias and high variance which is over-fitting. The latter example, high variance and high bias called under-fitting. In other words,  A model that is underfit will have high training and high testing error while an overfit model will have extremely low training error but a high testing error.

whether any of these score or difference among them tells us , if we
need more data(observations)

This is a very good question. you can refer here. And the best explanation of this I got from Andrew NG. This is the link. These will give you a solid understanding. but in simple words I can give example for intuition (won't be accurate): suppose I have 500 rows of data. I will first take 100 of my data. Then I will analyse my bias and variance. Next I will take 100+100=200 data from those 500. The again I will record my bias and variance. I will keep doing this by taking 300, 400 and 500 data. If I observe that my bias and variance is improving (if model is improving) it means adding more data might actually help. This is a very simplified example. But hope gives an intuition.


The error message is pretty clear... you feed a vector of length 24 to your model, but your model is outputting a vector of length 1.

Change :

dense3 = Dense(1 ,activation = 'softmax')(dense2)

to :

dense3 = Dense(24 ,activation = 'softmax')(dense2)


The Israeli MOH databases on Covid patient numbers have a lower cutoff value on published numbers of patients. How does this protect privacy?

It means any number lower than 15 is sufficiently small over a sufficiently large feature space so that identifiable data can be inferred by brute force reverse engineering.


How to use the same scale with new data? - scikit learn - scikit learn

IMO, you don't need to use scaling if your classifiers are based on decision trees. Also, in your final piece of code, the variable scaler is never used, so I am not sure at all why it is defined. Nevertheless, if you insist on using a scaler, you should Pipeline it so it automatically applies to the test data the same scaling it has learned by fitting to training data:

pipeline = Pipeline([('scaler', StandardScaler()), ('classifier', DecisionTreeClassifier())])
pipeline.fit(X_train, y_train)
predictions = pipeline.predict(X_test)

Note that you'd obviously need to import the necessary modules (I was too lazy to write them here!).


In k-means there is no similarity or order of axes. They are all assumed to be independent. Hence, these three series are all equally dissimilar. You got what you asked for - but the question you asked (minimize the squared errors in each component) was not what you wanted.

You'll need to choose a different algorithm. One that knows about time, and allows for some tolerance in time, or one where you can use a distance function such as DTW to measure similarity.

Don't treat clustering as a black box. That will not work, you'll get results that you don't like. Instead, understand the algorithms first, then guide them carefully to solve your problem.


How to reduce the Root mean square error

The problem is that you are using a classifier. You should use RandomForestRegressor or other type regressors.

Those three airports seems like leverage points. Check this wiki topic. You can log transform your target values or simply remove those 3 airports from your training set.

But still you have very less observations. You may want to add new data to your set. After train test split 35 observations for train 15 for test seems not so satisfactory.


Correct me if I'm wrong: As far as I understood, you have files of 1000 entries with 19 numerical sensor values and a type of 22 movements in each line.
You would like to predict the type of movement based on the sensor values.

Are the entries being recorded in a fixed interval?
If so, you could use an RNN.

Either way, it seems to be a classification problem. You can find a tutorial for Tensorflow here.

You can read data from csv files with pandas.read_csv.

As for the encoding of your data: If the types of movement cannot be sorted and are not related to each other, you should use one-hot encoding to present the type of movement to the NN. You can keep numerical sensor data as is, just make sure to normalize it.


The algorithm works by adding or subtracting the feature vector to/from the weight vector. If you only add/subtract parts of the feature vector your a not guaranteed to always nudge the weights in the right direction, which could mess with the convergence of the procedure.

The idea is that in the weight space every input vector is a hyperplane. You need to find a weight vector that is on the correct side of all the hyperplanes of your data inputs. The correct weight therefore is in a convex cone. If you observe a misclassification, that means your weight vector is on the wrong side of the hyperplane and therefore outside the convex cone of possible solutions.

Now, by adding/subtracting the input vector to the weight vector you make sure that this data input vector is now correctly classified. You also make sure, that you decrease the distance of your weight vector by a sufficient margin (at least the length of the input vector) towards the cone of all possible solutions.

If you only add parts of the vector (i.e. not update all weights) at each iteration, you cannot be sure that you make sufficient progress or even move in the right direction of the solution space.


How can I know if my conv1D model is overfitted or underfitted from loss curve?

From the curves you are showing yes. Over-fitting would mean that your validation accuracy would be lower than your training accuracy, which is not the case here. Since you say that your training and validation sets are completely independent (i.e. no training samples are present in the validation set) you can consider the results reliable.

However accuracy might not be the best indicator of the model's performance. Make sure your dataset is balanced (i.e. the number of samples in both classes are equal to one another). If not try another metric that better represents the performance of your model.


Getting Error in RStudio while loading a package- 'tm'

This is a dependency error. You're just missing one of the packages that tm assumes you have.

One solution is to simply directly install that package:

install.packages("slam")

another is to use a package manager that takes care of this for you:

install.packages("pacman")
pacman::p_load(tm)


Train and predict two labels in a single process

This would not be possible since the two variables you are trying to predict are of a different type. You are first predicting the default label, which would be yes/no, so this is a classification problem. The second variable you are trying to predict is the prepayment percentage, which is a continuous variable, this is therefore a regression problem. You are not able to combine the two (i.e. a regressor and a classifier) into one model using RandomForestClassifier. You might be able to create a single model yourself that combines the two using the BaseForest base class.


Why Decision trees performs better than logistic regression

Have you considered using a neural network for this? The challenge with the two approaches you're using is that both models are not great at handling a large number of features and you, essentially, have 10,000 features. So, my guess is that even with the decision tree,  you're probably just picking up some background pixels that are inflating your accuracy rate. There's probably something there that the decision tree is picking up and then "locking" you into that branch of the model, which is less than ideal. Using a neural network can help you create a model that can do more nuanced choices


Depending which tree algorithm you want to use you could manually construct the two first levels of the tree. You can just follow the pseudo code explained for example here for the C4.5 tree. Once you have done this you can remove the two features from the data set and create trees for the remaining part of the tree. If you want to create a rpart object you would be required to take some parts of the source and this may be a bit more demanding. Depending on what tree algorithm you use you will just have a binary split at both levels so you will only need to build 4 separate trees and not 264. Note that you may not have the optimal decision tree since after stepping through the first two levels, the country and product type may still be variables that cause a split. But without seeing the data is impossible to tell.

Side note, it may be valuable to explain the business that country and product type are not the most sensible variables to have in the top of the decision tree. Sometimes it is better to educate the end users than to force machine learning to do something inaccurate. In my experience end users prefer to have a correct solution than a solution that works because people have a gut feeling that it should be in a certain way.


Scalling and unscalling of data for SVR prediction

First , I am not sure why you want to scale the "Y" value at all? Inverse transformation is to scale back features of X mainly.
https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler.inverse_transform

Second, You are passing only 1 column of X now for prediction and your sc_y remembers the shape you passed it earlier. Your error is telling you the difference also. One has(24,) and another(1,).


How can I combine two single-column datasets into a single Pandas data frame?

I would import the datasets in pandas separately, mold them as you please, and then you can use the pd.concat function. This will assume that the instances are aligned by the automatically assigned index in pandas. If there is more data in one list than the other, the missing values will be NaN.

df1 = pd.DataFrame(data=[1,2,3])
df2 = pd.DataFrame(data=['a','b','c','d'])
dfs = pd.concat([df1, df2], axis=1)

If you have an index to link the text to the labels, then you can use the pd.merge function.

df1 = pd.DataFrame(data=[1,2,3])
df2 = pd.DataFrame(data=['a','b','c','d'])
dfs = df1.merge(df2, on='index')


Is Python a viable language to do statistical analysis in?

Python is more "general purpose" while R has a clear(er) focus on statistics. However, most (if not all) things you can do in R can be done in Python as well. The difference is that you need to use additional packages in Python for some things you can do in base R.
Some examples:

Data frames are base R while you need to use Pandas in Python.
Linear models (lm) are base R while you need to use statsmodels or scikit in Python. There are important conceptional differences to be considered.
For some rather basic mathematical operations you would need to use numpy.

Overall this leads to some additional effort (and knowledge) needed to work fluently in Python. I personally often feel more comfortable working with base R since I feel like being "closer to the data" in (base) R.
However, in other cases, e.g. when I use boosting or neural nets, Python seems to have an advantage over R. Many algorithms are developed in C++ (e.g. Keras, LightGBM) and adapted to Python and (often later to) R. At least when you work with Windows, this often works better with Python. You can use things like Tensorflow/Keras, LightGBM, Catboost in R, but it sometimes can be daunting to get the additional package running in R (especially with GPU support).
Many packages (or methods) are available for R and Python, such as GLMnet (for R / for Python). You can also see based on the Labs of "Introduction to Statistical Learning" - which are available for R and for Python as well - that there is not so much of a difference between the two languages in terms of what you can do. The difference is more like how things are done.
Finally, since Python is more "general purpose" than R (at least in my view), there are interesting and funny things you can do with Python (beyond statistics) which you cannot do with R (at least it is harder).


Keras loaded model output is different from the training model output

The problem was due to the output format. Instead of outputting one value per input sequence, in order to produce a complete and unique sequence as output, it was outputting sequences. Though instead of having a unique sequence as result it got a sequence of sequences. The solution was to fix the network output shape and data shape as well.


Can the learning rate be considered both a parameter AND a hyper-parameter?

My understanding is that $\eta$ is set before the training starts to a large value but then, as the training progresses and the function gets closer and closer to a local minimum, the learning rate is decreased. In that case, doesn't the learning parameter satisfy both the definitions of a parameter and of a hyper-parameter?

No it does not, because you are at all times controlling the learning rate without reference to the data. Typically both the learning rate and the learning rate schedule are considered hyper-parameters, so by adding a learning rate schedule you have not transformed learning rate into a parameter, but instead added a new hyper-parameter that may need tuning!

The learning rate is also is not part of the model as it would be used in production, so it is a training hyper-parameter. Although that is not a strict separation, this also applies to things that are neither hyper-parameters nor parameters, such as the current momentum values, which are used to improve training rate, and are usually stored in a some data structures alongside the model being trained. Likewise most adaptive learning rate structures (used in e.g. RMSProp, Adagrad etc) are in this category - neither model parameter nor hyper-parameter.

It might be possible to argue that for a continuous online model, the learning rate could be a model parameter, as it is part of the production model being used to make predictions - provided it was still being used, but adaptive, depending on data seen to date, and/or current learning progress. Then if you had to feed in an initial learning rate you could maybe say that the learning rate was both a parameter and hyper-parameter of the ANN model. Although you would likely have to explain the scenario in full to be understood, it would be more usual to just say you were using an adaptive learning rate.


Fast Python implementation of the gradient descent

Parallel gradient descent has been implemented in this repository in Python. It should have a familiar interface, since it's being developed for implementation as a scikit-learn feature.


Why does Siamese neural networks use tied weights and how do they work?

Have a look at this post: https://datascience.stackexchange.com/a/43820/98414
which should provide the intuition behind it.


Approximation of long sequence of layers by one layer

The teacher/student model approach (at least as I understand it) could be used.  It is normally used to replace all layers, but there is nothing stopping you applying it to a subset of layers.
First you create training data, by running your training data through the (trained) network, and recording the inputs to layer i, as your new training data (x) and the outputs from layer j as your desired answer (y).
You now simply (!) want to design a model that takes x and produces y. Ideally using fewer weights than layers i to j currently use.
If those layers were a fully-connected NN, then this is the idea behind autoencoders. If a Transformer then try with one attention + FCNN layer, and see how low the error will go. If it flatlines at an unacceptable error rate, maybe you will need to try two layers. If a CNN, then I am less sure it will work; my hunch would be would have better success keeping the same number of layers and convolution sizes, and just using fewer features in each layer.
Anyway, when you have trained this student model, you can plug it in to your original model to replace layers i to j. (Then maybe run some fine-tuning training.)
Note: if your layers have residual connections, you will have to think carefully about how they handled. Taking the i-1th output and feeding it to the j+1th layer, when it previously expected the output of the jth layer, may go badly. Especially if the dimension of the data is different.


Generate predictions that are orthogonal (uncorrelated) to a given variable

This requirement can be satisfied by adding sufficient noise to predictions $\hat{y}$ to decorrelate them from orthogonal values $v$. Ideally, if $\hat{y}$ is already decorrelated from $v$, no noise would be added to $\hat{y}$, thus  $\hat{y}$ would be maximally correlated with $y$.

Mathematically, we want to create $\hat{y}'=\hat{y}+\epsilon$ from $\epsilon \sim \mathcal{N}(0, \sigma_{\epsilon})$, to satisfy $$r_{\hat{y}'v} = \frac{\sigma_{\hat{y}'v}}{\sigma_{\hat{y}'}\sigma_{v}}  < \delta$$ for arbitrary threshold $\delta$. Now, lets expand this inequality to find a lower-bound for std of noise $\epsilon$, i.e. $\sigma_{\epsilon}$.
$$\begin{align*}
\sigma_{\hat{y}'}^2&=\sigma_{\hat{y}}^2 + \sigma_{\epsilon}^2,\\
\sigma_{\hat{y}'v}&={\Bbb E}\left[(\hat{y}+\epsilon - \mu_{\hat{y}} - \overbrace{\mu_{\epsilon}}^{=0})(v-\mu_{v})\right]\\
&={\Bbb E}\left[(\hat{y} - \mu_{\hat{y}})(v-\mu_{v})\right]+\overbrace{{\Bbb E}\left[\epsilon(v-\mu_{v})\right]}^{=0}&\\
&=\sigma_{\hat{y}v},\\
r_{\hat{y}'v} &= \frac{\sigma_{\hat{y}'v}}{\sigma_{\hat{y}'}\sigma_{v}} =\frac{\sigma_{\hat{y}v}}{\sigma_{v} \sqrt{\sigma_{\hat{y}}^2+\sigma_{\epsilon}^2}} < \delta\\
&\Rightarrow \sigma_{\hat{y}}\sqrt{\left(\frac{r_{\hat{y}v}}{\delta}\right)^2 - 1} < \sigma_{\epsilon}
\end{align*}$$

Since all the variables in the left side of inequality can be calculated, we can sample noises from $\mathcal{N}(0, \sigma_{\epsilon})$ and add them to $\hat{y}$ to satisfy the original inequality.

Here is a code that does the exact same thing:

import numpy as np
import pandas as pd
from sklearn.datasets import make_regression
from xgboost import XGBRegressor

ORTHO_VAR = 'ortho_var'
IND_VARNM = 'indep_var'
TARGET = 'target'
CORRECTED_VARNM = 'indep_var_fixed'

seed = 245
# Create regression dataset with two correlated targets
X, y = make_regression(n_samples=10000, n_features=20, random_state=seed, n_targets=2)
indep_vars = ['var{}'.format(i) for i in range(X.shape[1])]

# Pull into dataframe
df = pd.DataFrame(X, columns=indep_vars)
df[TARGET] = y[:, 0]
df[ORTHO_VAR] = y[:, 1]

# Fit a model to predict TARGET
xgb = XGBRegressor(n_estimators=10)
xgb.fit(df[indep_vars], df[TARGET])
df['yhat'] = xgb.predict(df[indep_vars])

delta = 0.01

# std of noise required to be added to y_hat to bring the correlation
# of y_hat with ORTHO_VAR below delta
std_y_hat = np.std(df['yhat'], ddof=1)
corr_y_hat_ortho_var = np.corrcoef(df['yhat'], df[ORTHO_VAR])[1, 0]
corr_y_hat_target = np.corrcoef(df['yhat'], df[TARGET])[1, 0]
std_noise_lower_bound = std_y_hat * np.sqrt((corr_y_hat_ortho_var / delta)**2 - 1.0)
std_noise = max(0, std_noise_lower_bound) + 1
print('delta: ', delta)
print('std_y_hat: ', std_y_hat)
print('corr_y_hat_target: ', corr_y_hat_target)
print('corr_y_hat_ortho_var: ', corr_y_hat_ortho_var)
print('std_noise_lower_bound: ', std_noise_lower_bound)
print('std_noise: ', std_noise)

# add noise
np.random.seed(seed)
noises = np.random.normal(0, std_noise, len(df['yhat']))
noises -= np.mean(noises)  # remove slight deviations from zero mean
print('noise_samples: mean:', np.mean(noises), ', std: ', np.std(noises))
df['yhat'] = df['yhat'] + noises

# measure new correlation
corr_y_hat_ortho_var = np.corrcoef(df['yhat'], df[ORTHO_VAR])[1, 0]
corr_y_hat_target = np.corrcoef(df['yhat'], df[TARGET])[1, 0]
print('new corr_y_hat_target: ', corr_y_hat_target)
print('new corr_y_hat_ortho_var: ', corr_y_hat_ortho_var)
# Correlation should be low or preferably zero
assert corr_y_hat_ortho_var < delta, corr_y_hat_ortho_var
assert -delta < corr_y_hat_ortho_var, corr_y_hat_ortho_var

which outputs:

delta:  0.01
std_y_hat:  69.48568725585938
corr_y_hat_target:  0.8207672834857673
corr_y_hat_ortho_var:  0.7663936356880843
std_noise_lower_bound:  5324.885500165032
std_noise:  5325.885500165032
noise_samples: mean: 1.1059455573558807e-13 , std:  5373.914830034988
new corr_y_hat_target:  -0.004125016071865934
new corr_y_hat_ortho_var:  -0.000541131379457552

You can experiment with other deltas. By comparing std_y_hat with std_noise_lower_bound, you can see that a huge noise must be added to $\hat{y}$ to decorrelate it from $v$ bellow $0.01$, which dramatically decolerates $\hat{y}$ from $y$ too.

Note: Assertion might fail for too small thresholds $\delta$ due to insufficient sample count.


Yes there are so many approaches to handle missing data or missing values depending on the task and the property of the data itself. For example in time series you can think about forward filling or even backward filling, max, mean or median over a time lapse. There are also 'smarter' ways like training a model over the available data and try to predict the missing ones. The latest requires much data so can give efficient results. You can refer to these two links for more information.

How to Handle Missing Data with Python
Machine learning algorithms to handle missing data


High model accuracy vs very low validation accuarcy

When a machine learning model has high training accuracy and very low validation then this case is probably known as over-fitting. The reasons for this can be as follows:

The hypothesis function you are using is too complex that your model perfectly fits the training data but fails to do on test/validation data.
The number of learning parameters in your model is way too big that instead of generalizing the examples , your model learns those examples and hence the model performs badly on test/validation data.

To solve the above problems a number of solutions can be tried depending on your dataset:

Use a simple cost and loss function.
Use regulation which helps in reducing over-fitting i.e Dropout.
Reduce the number of learning parameters in your model.

These are the 3 solutions that are most likely to improve the validation accuracy of your model and still if these don't work check your inputs whether they have the right shapes and sizes.


Difference between MDS and other manifold learning algorithms

The images in the link you provide, of the severed sphere and its lower-dimensional representations, go some way towards explaining the difference.

The severed sphere is a set of points in a three-dimensional space, but we want a two-dimensional representation of it. The objective of manifold-learning is (shockingly) to find a manifold: a subset of that three-dimensional space which (a) closely fits all the points that make up the severed sphere, and (b) can be described with a two-dimensional coordinate system.

If you look at some of the other lower-dimensional representations of the severed sphere, it's like they're taking it and flattening it out into a rectangle so it'll fit in two dimensions. It's taking the severed sphere and figuring out a new coordinate system that maps as closely as possible onto all the points that make up the severed sphere.

The MDS lower-dimensional representation, though, is more like a shadow that the severed sphere casts on a wall. Rather than finding a new coordinate system that closely fits the sphere, it's just "forgetting" whichever of the dimensions it thinks it can most afford to lose while maintaining the same distance to and from all the points.

A good analogy would be maps of the earth. A good map of the earth makes a new coordinate system that fits a sphere onto a 2D surface. To do this it has to distort the relative distances between places, but you end up with effective 2D coordinates that relate well to places on the globe.

Instead of doing this, you could just take two photos of the earth from above the north and south pole and glue them back to back. You'd still have a 2D representation of the earth, but it doesn't work so well as a coordinate system.

This isn't to say that MDS is "bad". It's just doing something different. You probably wouldn't use MDS for dimensionality reduction prior to carrying out some sort of statistical procedure, but if you're trying to produce a graphic that gives some idea of how close multidimensional points are to one another, it might be a good choice.


how to iterate over certain rows of a certain column an update their value. by applying condition

You can use df.loc to make the selection and then set the values. Take a look at this example and modify according to your dataset.

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'a':[1,2,3], 'b':[0.1,0.3,0.9], 'c':[4,5,6]})

In [3]: df
Out[3]:
   a    b  c
0  1  0.1  4
1  2  0.3  5
2  3  0.9  6

In [4]: df.loc[df['b'] < 0.5, ['a', 'c']] += 20

In [5]: df
Out[5]:
    a    b   c
0  21  0.1  24
1  22  0.3  25
2   3  0.9   6

Also, in future try to provide a sample from your dataset as a table of values and not an image so that it becomes easier to reproduce an example.


why we sample when predicting with Recurent Neural Network

It seems to me you have two questions:

Why use sampling to generate text from a trained RNN language model?
How does this particular sampling function from Keras work?

Why use sampling to generate text from a trained RNN language model

After training a language model, very often you would like to use the model to generate new text. For a word-level RNN language model, text is generated one word at a time. In each step, the model outputs a probability distribution over the entire vocabulary. With a toy vocabulary such as:

$$
vocabulary=\hspace{0.2cm} \begin{bmatrix}
           cat \\
           dog \\
           frog
         \end{bmatrix}
$$

the distribution output by the model at each generation step could look like

$$
predictions=\begin{bmatrix}
           0.7 \\
           0.1 \\
           0.2
         \end{bmatrix}
$$

each entry in the vector corresponds to one entry in our vocabulary.

After that, it is up to the user how to select a word from this distribution as the next word in the generated text. Two very different methods to pick a word from the distribution are:

Greedy search: take the word with the highest probability (will pick "cat")
Sampling: sample from the distribution, take into account probabilities (will pick a random word, but "cat" has the highest chance of being picked)

So, searching will return a very probable string (and the same one every time we try), while sampling will return somewhat likely, but more varied text. Often, this means that sampling leads to more "interesting" text.

How does this particular sampling function work?

As far as I can tell, the input for this sampling function is a vector of normalized probabilities after softmax, such as

np.array([0.7, 0.1, 0.2])

Here are some explanations about each line:

preds = np.asarray(preds).astype('float64')

This just makes sure that other array-like objects, such as Python lists, work as input, but are converted to the correct numpy array type.

preds = np.log(preds) / temperature

preds is a probability distribution that was created by applying the softmax function to the final output of the model network (those final values are called logits). Logits are assumed to be unnormalized log probabilities, applying softmax normalizes them.

This is a problem for softmax with temperature, because it is the unnormalized logits that are supposed to be divided by the temperature:

$$
q_i = \frac{exp(z_i/T)}{\sum_j{exp(z_j/T)}}
$$

So, np.log(preds) just moves the values back into logspace. Without this, we would not be scaling the actual network outputs, but exponentiated values.

exp_preds = np.exp(preds)
preds = exp_preds / np.sum(exp_preds)

Those two lines just apply the softmax function, regardless of what happens before or after.

probas = np.random.multinomial(1, preds, 1)

This function with those particular arguments is sampling a one-hot vector such as

$$
predictions=\begin{bmatrix}
       1 \\
       0 \\
       0
     \end{bmatrix}
$$

where the probability of each index being the 1 is dictated by preds.

return np.argmax(probas)

Finally, this returns the index of the largest element in the vector. This index will correspond to one word in our vocabulary.

I hope this helps you understand the matter, or perhaps express more clearly what you would need additional explanations for.


PRML Bishop - Doubt in Entropy Formula (Section 1.6, equation 1.94)

You are not missing anything. You simply found another derivation of the same relationship.

We know that

$$^{n}C_{k} = \frac{n!}{k!(n-k)!} $$

If we expand the terms in your product,

$$^{N}C_{1} * ^{N-1}C_{2} * ^{N-3}C_{3}* ^{N-6}C_{4} \cdot \cdot $$

, we get

$$\frac{n!}{1!(n-1)!} \cdot \frac{(n-1)!}{2!(n-3)!} \cdot \frac{(n-3)!}{3!(n-6)!} \cdot \frac{(n-6)!}{4!(n-10)!} \dots $$

If we simplify the factorials,

$$\frac{n}{1!} \cdot \frac{(n-1)(n-2)}{2!} \cdot \frac{(n-3)(n-4)(n-5)}{3!} \cdot \frac{(n-6)(n-7)(n-8)(n-9)}{4!} \dots $$

And now it's obvious that this is

$$ \frac{n!}{\prod_i i!} $$


What is the accuracy majority class classifier?

I suspect you are right that there is a missing "of the," and that the "majority class classifier" is the classifier that predicts the majority class for every input. Such a classifier is useful as a baseline model, and is particularly important when using accuracy as your metric. This matches what your notebook comments in the next bullet, so that's likely what was intended.


How many ways are there to check model overfitting?

The direct way to check your model for overfitting is to compare its performance on a training set with its performance on a testing set; overfitting is when your train score is significantly above your cv score. According to your comments, your r2 score is 0.97 on the training set, and 0.86 on your testing set (or similarly, 0.88 cv score, mean across 10 folds). That's somewhat overfitting, but not extremely so; think if 0.88 is "good enough" for your requirements

The r2 score is 1 - MSE of errors / variance of true values. In the example you showed, all of the three true values were the same; i.e. their variance is zero. The r2 score should've been a negative infinite, but apparently sklearn corrects this to 0; you can verify that changing y_true to [0.9, 0.9, 0.90001] changes your r2 score to a very large negative number (around -2*10**9). This is why checking r2 against a small sample is not a good idea; the mean of the small sample contains too much important information.

You added that you want to know which parameters to tune in order to prevent over-fitting. In your edit to your question, you said you're using grid-search over n_estimators (3 options), min_samples_split (2 options) and min_sample_leaf (2 options).
There are other parameters you can try, and in my experience max_depth is important to tune. This question on Stack Overflow and this question on Cross Validated deal with overfitting, and there are good options there.  I'd add that if you're trying many options, then maybe you'd be better off doing using Bayesian Optimization (there's a package that functions well with SKLearn: https://scikit-optimize.github.io/stable/auto_examples/sklearn-gridsearchcv-replacement.html).


Selecting the right algorithm for match probability prediction

It seems a variant of Movie Recommendation problem. Take Netflix for example, they have  user ratings for different type of movies; in your case users are telling what attributes they like. When a new user joins and rates the things he/she likes, you will find the people with similar interests, again with Netflix analogy it means recommending movies based on people with same tastes as the new user.

Read on Collaborative filtering, which is used for building recommendation systems. The reason I drew the Netflix/Movie Recommendation analogy is because you can find a lot of literature on it.


How to reduce position changes after dimensionality reduction?

You can initialize a UMAP embedding with a custom set of initial positions, so potentially you can initialise step 2 with the embedding from step 1 (with random positions for the new points).


Recurrent Neural Network (RNN) are quite efficient at dealing with sequential information such as time-series.
LSTM (Long Short Term Memory) is a type of RNN that can learn long-term dependencies and overcome some problems that vanilla RNNs have.
If you want to try it on your problem, I'd suggest this tutorial on LSTM for time-series prediction.


Training a neural network with TWO possible correct outputs for one input

With Keras, you could use the functional API, to estimate a model with two outputs („multioutput“). Simply train the model on two outputs like:
# Outputs
out1 = Dense(1)(x)
out2 = Dense(1)(x)

# Compile/fit the model
model = Model(inputs=Input_1, outputs=[out1,out2])
model.compile(optimizer = "rmsprop", loss = 'mse')

# Add actual data here in the fit statement
model.fit(train_data, [train_targets,train_targets2], epochs=500, batch_size=4, verbose=0, validation_split=0.2)

Find the full model here: https://github.com/Bixi81/Python-ml/blob/master/keras_multitarget_functionalAPI.py


XGBoost vs ARIMA for Time Series analysis

Usually, ARIMA regressions are used in classical statistical approaches, when the goalis not just prediction, but also understanding on how different explanatory variables relate with the dependent variable and with each other. ARIMA are thought specifically for time series data.

On the contrary, XGBoost models are used in pure Machine Learning approaches, where we exclusively care about quality of prediction.
XGBoost regressors can be used for time series forecast (an example is this Kaggle kernel), even though they are not specifically meant for long term forecasts. But they can work.

In conclusion, I don't know your data, but if I had to bet I'd go for ARIMA. However both could work great, and I suggest you to try both and pick the best for your particular need. They are comparatively quick to implement, it shouldn't take too much time.


Have a look at the rnn package (full disclosure, I am the author). It implements a multilayer RNN, GRU, and LSTM directly in R, i.e. not an underlying C++ library, so you should also be able to read the code and understand what is going on.

install.packages('rnn')

The CRAN version is quite up to date, but the GitHub version is bleeding edge and can be installed using:

if (!require('devtools')) install.packages('devtools')
devtools::install_github('bquast/rnn')


What is the standard model for CNN to compare against other classifier technique?

If your model is simple and you don't have a lot of training data, then you need a model with few parameters to compare to, or else you won't be able to train it. Using standard CNN architectures may not be a good option in this case, because even if the point of using a CNN rather than a full-connected network is to reduce the parameters, they still have many millions of parameters. Depending on the details of the problem, I think you have to design something yourself. Have a look at this Kaggle guide on choosing a CNN architecture.

If you have a lot of input data and a complex model that you're comparing to, I would recommend using a standard architecture that people are familiar with, such as VGG-16 or VGG-19, so that a performance comparison means something to the reader.

You can get these models with pretrained weights, so you don't need to do any training necessarily. However, they are set up for multiclass classification and typically trained on ImageNet, so you need to adapt the classification part. The simplest way is to replace the last (output) layer with a single output node for your binary classification.

To make a fair comparison you then need to train the entire model on the same training dataset used for your model, and evaluate on the same test set. But I would start by retraining the final FC layers first, to get a quick preview of how it's going. Then retrain the full model for the fair comparison.


Looking for a causality to effect dataset

I doubt anything so general exists. The closest I can think of would be to exploit resources which exist in the medical domain, in particular https://semrep.nlm.nih.gov/

Also try asking on https://opendata.stackexchange.com/


How does ResNet bottleneck architecture's input size is possible to change from 56x56x64 to 56x56x356?

I do not know the answer for sure but I assume that the "256-d" refers to the shortcut connection and not the input. Since the output dimension of the 3 conv. layers in your right hand picture (fig. 5) is 256 the shortcut is transformed from depth 64 to depth 256.

How this is being done is explained in the following paragraph on p. 4:

  Residual Network. Based on the above plain network, we
  insert shortcut connections (Fig. 3, right) which turn the
  network into its counterpart residual version. The identity
  shortcuts (Eqn.(1)) can be directly used when the input and
  output are of the same dimensions (solid line shortcuts in
  Fig. 3). When the dimensions increase (dotted line shortcuts
  in Fig. 3), we consider two options: (A) The shortcut still
  performs identity mapping, with extra zero entries padded
  for increasing dimensions. This option introduces no extra
  parameter; (B) The projection shortcut in Eqn.(2) is used to
  match dimensions (done by 1×1 convolutions). For both
  options, when the shortcuts go across feature maps of two
  sizes, they are performed with a stride of 2.

For ResNet-34 this not required for the first building block which is shown in fig. 5 but for later blocks too. This is shown by the dotted lines in figure 3:


Saving LSTM hidden states while training and predicting for multi-class time series classification

There didn't seem to be any way to build a working trainable model while also returning the hidden states with return_sequences=True.

The fix I found was to build a predictor model and train it, and save the weights. Then I built a new model which ended with my LSTM layer, and fed it the trained weights. So, using return_sequences=True, I was able to predict on new data and get the data's representations at each hidden state.


Augmenting the validation set in Ensemble Model

Have you considered freezing the ensemble models themselves and allowing only the weights in final layer to change? Then the overfitting shouldn't be so pronounced...
You might also want to consider cross validation procedure.


train.py code:

builder = tf.saved_model.builder.SavedModelBuilder("/home/datam/cnn-text-classification-tf/model/20180423")
builder.add_meta_graph_and_variables(
    sess,
    [tf.saved_model.tag_constants.SERVING],
    clear_devices=True
)
builder.save()

Java Coding:

SavedModelBundle model = SavedModelBundle.load(SAVED_MODEL_PATH+"/20180423", "serve");
Session s = model.session();
Graph g = model.graph();


Is there any difference between feature extraction and feature learning?

Yes I think so. Just by looking at Feature Learning and Feature extraction you can see it's a different problem.

Feature extraction is just transforming your raw data into a sequence of feature vectors (e.g. a dataframe) that you can work on.

In feature learning, you don't know what feature you can extract from your data. In fact, you will probably apply machine learning techniques just to  discover what are good features to extract from your dataset. Then you can extract them them apply machine learning to the extracted features. Deep learning techniques are one example of this.

In the word2vec toolkit, for instance, you extract vectors from documents which can't be easily interpreted by a human, you can't look at it and tell what features have been extracted at all. It's just a mass of vectors which, for some reason, give good empirical results.


Does data anonymization conflict with GDPR rules?

Formally speaking, this is clarified in the GDPR Recital 26, unofficially titled Not Applicable to Anonymous Data:

The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable.
This Regulation does not therefore concern the processing of such anonymous information, including for statistical or research purposes.

Informally speaking, the claim that data anonymization would violate the data subject's rights to data erasure and portability, hence we should seek to use reversible anonymization techniques, sounds awkward, and against the very spirit of GDPR; and the official interpretation regarding anonymization is very clear:

Effective data anonymization is made up of two parts:

It is irreversible.
It is done in such a way that it is impossible (or extremely impractical) to identify the data subject.

In other words:

if a subject's data have been effectively anonymized, they are no longer personal data, hence they are no longer governed by GDPR; consequently, Articles 17 & 20 are not applicable, and this does not constitute any conflict
if any personal data used as a source for the anonymized ones remain, they are subject to GDPR; data subjects can exercise their right to erasure and portability for these non-anonymized data
if the personal data used as a source for the anonymized ones are already erased (possibly in compliance to GDPR), then neither the right to erasure nor the right to portability are applicable any more, and this does not constitute any kind of conflict either.

Notice that the fact that ensuring effective anonymization is not as clear-cut as it may sound has also been legally recognized in the Opinion 05/2014 on Anonymisation Techniques of the Data Protection Working Party:

Thus, anonymisation should not be regarded as a one-off exercise and the attending risks should be reassessed regularly by data controllers.


How to use pca results for linear regression

In general, I would suggest to use a regularization technique for reducing the dimensionality ofa data set in linear regression cases. Please refer to L1 regularization.

If you want to decrease the number variables using PCA, you should look at the lambda values that describe the variations in the principle components, then, select the a few components with the largest corresponding lambda values (eg the first four).

Note:

Do a scaling if necessary.
Sometimes the very first component is not very relevant and can be eliminated.


Is there a name for a scale which mixes ordinal and nominal?

One option is create new features. One feature that is completely nominal, and another feature that is completely ordinal.
Otherwise, the scale types are "rounded down". There is no danger in treating ordinal data as nominal. However, treating truly nominal data as ordinal can cause analysis problems.
There is not much literature on mixing scale types because that could easily lead to misinterpretation of results.


What is the output polytree after aplying the Ramex algorithm to this graph?

Here's my interpretation of the algorithm, not sure it is completely right but that could help you:

Initialize S: null (a is added after that)
Enter in first loop on vertices (assume we start with a)
Select the edge with highest weight connected to a: a -> c, and add it
Same with updated S (a or c) c -> d, and add it
Same with updated S: d -> e
Same with updated S: e -> h
Same with updated S: e -> g
Same with updated S: this is where forward and forward & backward algorithms diverge. Forward algorithm only looks at weights for edges starting from S (this is a -> b), while forward and backward algorithm also screens backward links, so the selected link is b -> d which has a higher weight (396) than a -> b (335)
Last update of S with e -> f

This is what seems right to explain the figure, and works to build a polytree. I am a bit confused because it does not exactly correspond to what is written in Algorithm 3, but it is not so rare that pseudo-code algorithm make no actual sense, and would yield wrong results if written directly in any programming language.


How to solve ValueError: Negative dimension size caused by subtracting 3 from 1 for '{{node model/Conv1/Conv2D}} = Conv2D... in mobilenet_v2

I found out my mistake!
I shouldn't use a condition before resizing the frame:
face = cv.resize(face, (224, 224)) #without if (face.shape) > (224, 224):

I've used the condition because I had to do that for SSD Resnet in the past, but I just realized it's not necessary for mobilenet_v2


How to select one record from multiple record for a subject during analysis?

I will try and be as concise as possible. First, let's redefine the way you think about your data points. There can ever only be two types of visits in terms of time. Periodic and Non-Periodic. Let's call each visit an event. Some events could be related to chronic conditions where periodic visits are quite common. Some events could be related to flu, head injury, etc., These are non-periodic visits. You need to think about what you are trying to predict. Are you predicting an inpatient visit based on periodic events or non-periodic events?

1) I would not recommend picking a single record in multiple visits as it is insufficient. For example, if I come in for a visit for a Blood Pressure Check-Up and you pick that event, then that means you only get that information but where as I would be visiting for a follow up check up or something like that or it could be pre-surgery visits. Some of these events could be both inpatient and outpatient depending on the nature of the event. It is also possible that some of the outpatient visits will lead to an inpatient visit. Jack's earlier outpatient visits could be a sign of him coming in as an inpatient. All of John's visits are outpatient but will he ever be an in-patient? Is that what are you trying to predict?

2) You can definitely have multiple records per person and treat them as a sequence of visits over a period of time. This basically means that you are treating these visits as sequences. Most of the health care related risk prediction stuff is based on sequences. It is just the way you treat sentences. Each patient is described by a sentence of visits. This meaning, you can leave John's visits and Jack's visits as they are and use padding techniques to make them appropriate for a deep learning solution to classify which one would be inpatient.

You can refer to this paper.

This is definitely not concise but the number of permutations and combinations when it comes to healthcare stuff is high. Please let me know if you've got any questions.


Understanding predict_proba from MultiOutputClassifier

Assuming your target is (0,1), then the classifier would output a probability matrix of dimension (N,2).
The first index refers to the probability that the data belong to class 0, and the second  refers to the probability that the data belong to class 1.

These two would sum to 1.

You can then output the result by:

probability_class_1 = model.predict_proba(X)[:, 1]

If you have k classes, the output would be (N,k), you would have to specify the probability of which class you want.


Which correlation test can be used on low count contigency table?

I don't think it's worth doing any kind of significance test for such a small sample.

option 1: pure categorical values

You could just count the number of identical ratings between the two ratings and divide by the total number of players

option 2: take the order into account: weak < moderate < strong

Define a similarity measure such as:

sim(X, X) = 1
sim(weak,moderate) = 0.5
sim(moderate,strong) = 0.5
sim(weak,strong) = 0

Then use the same process as option 1 except that for every player you add sim(ratingA, ratingB), and divide by the total. This will also give you a normalized score between 0 and 1 which is a bit more precise (the value will be higher or equal than in option 1).


What is Continuous Ranked Probability Score (CRPS)?

CRPS is in a sense just the mean square error (MSE) of your predicted cumulative density function (CDF) and the true CDF.

The CRPS generalizes the MAE (Mean Absolute Error) to the case of probabilistic forecasts. The CPRS is one of the most widely used accuracy metrics where probabilistic forecasts are involved.

The CRPS is frequently used in order to assess the respective accuracy of two probabilistic forecasting models. In particular, this metric can be combined with a backtesting process in order to stabilize the accuracy assessment by leveraging multiple measurements over the same dataset.

This metric notably differs from simpler metrics such as MAE because of its asymmetric expression: while the forecasts are probabilistic, the observations are deterministic. Unlike the pinball loss function, the CPRS does not focus on any specific point of the probability distribution but considers the distribution of the forecasts as a whole.

Find More Info

Paper

Five ways to calculate CRPS

Using python library properscoring 0.1

pip install properscoring

This library focuses on the closely related Continuous Ranked Probability Score (CRPS) and Brier Score. We like these scores because they are both interpretable (e.g., CRPS is a generalization of mean absolute error) and easily calculated from a finite number of samples of a probability distribution.

We will illustrate how to calculate CRPS against a forecast given by a Gaussian random variable. To begin, import properscoring:

import numpy as np
import properscoring as ps
from scipy.stats import norm

Exact calculation using crps_gaussian (this is the fastest method):

ps.crps_gaussian(0, mu=0, sig=1)

Numerical integration with crps_quadrature:

ps.crps_quadrature(0, norm)

From a finite sample with crps_ensemble:

ensemble = np.random.RandomState(0).randn(1000)
ps.crps_ensemble(0, ensemble)

Weighted by PDF values with crps_ensemble:

x = np.linspace(-5, 5, num=1000)
ps.crps_ensemble(0, x, weights=norm.pdf(x))

Based on the threshold decomposition of CRPS with threshold_brier_score:

threshold_scores = ps.threshold_brier_score(0, ensemble, threshold=x)
(x[1] - x[0]) * threshold_scores.sum(axis=-1)

More at


I found a dataset with EXIF data! It's the MIRFLICKR dataset. I was looking for more amateur-style images, but this is a great start.

I'll post if I make any more findings. Other datasets that I've checked have some images with EXIF data, but they've ranged from 1 in 100 to less than 1 in 1000 images with useful EXIF data.


I need to force my random forest model to learn one crucial relationship between features. How can I achieve that?

Machine learning algorithms are not to know, they predict. A tree-based ML algorithm will give you probabilities of being in each class, not whether it belongs to a particular class. By preparing your labels carefully; if your data has enough examples of which “feature 1 and 3 are less and equal to 100” and the result is 0, it is highly probable that your algorithm will predict its class with a strong accuracy. Obviously,  the strength of the result depends on what your ML task is, what is your data, how it is structured and what your hyperparameter settings are. I do not know if there is another way to bend the forest in such a manner.

I know that is not the kind of an answer you expected, but I’ll be glad if I could help.


Should I divide my dataset in multiple prediction models?

Yes, you can divide your dataset and use different prediction models for different part of your dataset. Take advantage of your domain knowledge to build a good model.

After all, you can actually count the following as a single model
$$f(x) = \begin{cases} f_1(x) & ,x \in A \\ f_2(x) & ,x \in A^c \end{cases}$$


How to use precomputed distance matrix and min_sample for DBSCAN clustering method?

DBSCAN does not guarantee a minimum cluster size. There are known situations, c.f. Wikipedia, where a cluster can have fewer than "minPts" points. Furthermore, it has the concept of noise: points that do not have enough neighbors.

For epsilon, also see the Wikipedia article. As you don't specify the number of clusters, this parameter is what mostly controls how many clusters you get. Set it to 0, and everything will be noise. Set it to the maximum distance, and everything will be in one cluster.

Really read the article. It's about density, not about cluster sizes.


Recommending college degrees based on high school subjects

This problem is perfectly suited for a neural network. Your model will have 40 input nodes (this is fine), then you will have some arbitrary hidden layers, you need to tune this, and 20 outputs. After the training process you can even get a probability for each of them. This can be used to rank suggestions for potential students!

How to do this

Load the data to memory

Depending on the means by which your data is stored this step will be different. However, the goal is to go from the raw file source to either Python Numpy array or a Pandas DataFrame. I will assume your data is structured as follows

and is saved as a .csv file.

Let's load our data into an X and Y matrix. We will be encoding the labels for 'degree' as values. Make sure these are all well spelled otherwise a new label will be created for the mispelled ones.

import pandas as pd
import numpy as np

df = pd.read_csv('test1.csv')
df['Degree'] = df['Degree'].astype('category')
df["Degree_encoding"] = df["Degree"].cat.codes

X = np.asarray(df.loc[:, df.columns != 'Degree'])
Y = df['Degree_encoding']

print(X.shape)
print(Y.shape)

  (39, 7) (39,)

Applying a neural network to this dataset

First we will split the data randomly into a training and testing set. This is used to evaluate our model while maintaining that we are not overfitting. Then we identify the number of output classes. Then we reshape the input matrices such that they have a channel in their last dimension, this is how data flows through the model. Then we will categorize our outputs as one-hot encoded vectors.

from sklearn.model_selection import train_test_split
import keras

# Split the data
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

# The known number of output classes.
num_classes = len(set(df["Degree_encoding"]))
# Input dimensions
shape = X.shape[1::]

# We need to add a channels dimension to our data
# Channels go last for TensorFlow backend in Keras
x_train_reshaped = x_train.reshape((x_train.shape[0],) + shape)
x_test_reshaped = x_test.reshape((x_test.shape[0],) + shape)
input_shape = shape
print(input_shape)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

We then design our model

model = Sequential()
model.add(Dense(32,
                 activation='relu',
                 input_shape=input_shape))
model.add(Dense(128, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

You can use model.summary() to get a description of these layers. Then we are ready to train our model!

epochs = 100
batch_size = 128
model.fit(x_train_reshaped, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test))


Disclaimer:

This is a question that is probably going to be flagged since it is too broad and answers will be mainly based on opinion.

Is this effort worth it?

That is subjective, what commercial/social use does this have? Is this kind of detection really relevant? What can you do with those detections and classifications? That is something only an entomologist could answer properly, you will get better answers for this one with them.

If you can find this kind of motivations for your work, you probably get a good dataset with good use. But if you don't even know what kind of applications you're aiming to you should start doing more research on motivation.

Would these pictures significantly increase the model's accuracy?

It is highly probable that it will, but you can't be sure until you try. Also, train your models with recently acquired images during the acquisition process to make sure that those images are having an impact on your models.

Is there any opportunity for publication?

Surely there is, if it will be cited multiple times or if the paper will be well received by Machine Learning community? Probably. If not, that probably would get you a top publication on entomology.

Is there any paper that uses studio photos as well as field photos for object detection?

Well, there are many. These controlled datasets are pretty easy to come up with and have been used extensively see BioID for example, until today many publications use it for benchmarking on Eye Pupil Detection and Facial Landmark Detection. In the wild datasets (usually collected from the web) are usually more challenging and also more fitted to real-world application, but it is not always possible to acquire them easily.

Last year CEFET-RJ and UFRJ published papers in IEEE with created datasets for Aedes Aegypti breeding sites detection. This dataset was created using drones and manually annotated by CEFET-RJ and UFRJ undergrad students.


LinearRegression with fixed slope parameter

You can just compute:  $\hat{b}_0 = \operatorname{mean}(y-cx)$


How to Replace an object in Pandas array using replace with dictionary from excel file?

Assuming you have a pair of csv files: "replace.csv" representing the first table and "table.csv" representing the second table i.e. "dict". Use the following code:

import pandas as pd

frame = pd.read_csv('replace.csv')
table = pd.read_csv('table.csv', header=None)

mapping = dict([(k, v) for k, v in table.values])

frame.replace(to_replace={"name": mapping, "gender": mapping}, inplace=True)
print frame

The line mapping = dict([(k, v) for k, v in table.values]) is the one that transforms your dataframe into a dict. For more on transforming a dataframe into a dictionary see the documentation, also this question provides different ways of transforming a dataframe into a dictionary. For reading excel files, instead of csv files, see this.


Percentile as a threshold for Anomaly Detection?

You have a distribution of data (multidimensional or not it does not matter). And anomalies will be scattered towards corners of this dataset. Hence low percentile will indicate that certain points are anomalies in the data---cornered in. Threshold is, in this context, just a cutt-off value when you claim something is anomaly.


You should fit preprocessing transformers, i.e. imputation, scalers, encoders, resampling, only to train set and apply them to both train and test respectively. Your dataset is imbalanced and you may expect some improvement using resampling techniques, though you should always confirm it conducting cross validation tests.


Neural Network returning training data ratio instead of probability

In case of any neural network if you are trying to solve the classification problem then it should generate probabilities.

If you are trying to solve 2 class classification problem then one can opt for sigmoid or softmax. In the case of sigmoid, your neural network only contains 1 output neuron which generates value probability and if probability greater than equal to 50 then belongs to one class and less than 50 then belongs to another class. e.g.(0.333 0.333 0.333) = (0.80) that means belongs class 1 and (0.666 0.333 0.999) = (0.40) that means belongs class 0
In the case of softmax, the neural network should have an equal number of output neurons to the number of categories. e.g.(0.333 0.333 0.333) = (0.80, 0.12, 0.08) that means belongs class 1 and e.g.(0.333 0.333 0.333) = (0.08, 0.20, 0.72) that means belongs class 3


How prevalent is `C/C++` in machine learning development?

As you already understand, the vast majority of the data science work is made with rather high level languages such as Python and R. So it's not a matter of prevalence, it's a matter of which part in the big world of data science you want/can do with your skills and your tools.

Imho inventing new models requires:

strong theoretical background in maths and statistics, in-depth knowledge of existing estimation/inference methods
good understanding of computational complexity and (preferably) algorithmic optimization methods.

If additionally you implement your models yourself (that's not necessarily the case), that's where you probably need to know low level languages such as C/C++, because computational efficiency is crucial when people are going to use the model with massive datasets which require a lot of computation.


Most cloud systems support the command line mv command to move files or folders.

Taking a command line approach allows for more control and visibility into tasks.


How to improve results in classification problems (SVM, Logistic Regression and MultiNaive Bayes)?

If you have a lot of data - down-sample your negative class to achieve 50/50 split on your fake news/real news classification. If you don't have much data - you can use techniques like SMOTE to up-sample the lesser class.
You seem to have better accuracy than randomly choosing fake/real which is a good sign. Your probability of a negative class based on your data split is 71.6% - and you are able to achieve 85.4% accuracy with LogReg. Don't get too down on that (especially if you are new to ML).
I would recommend checking out Gradient Boosting or Bagging algos if this is an NLP problem - these usually yield the best results for me when I'm encountered with sparse text data in classification.
As for the punctuation and stop words this is a common first step - however it's not good general advice for any problem. Do you think the presence of exclamation points might weed out some fake news in your data? If so I would include punctuation. If not - you're probably already on the right track. Removing stop words and punc only makes sense if the context of your specific problem calls for it.
More generally - your desire to reach 92% accuracy might not be possible given the difficulty of your problem. This is not to say it's not possible but keep in mind that the tutorials you may follow online are pre-determined to show that you can get good results. Some projects are simply harder than others (and some are not even possible given the context).
Good luck!


What is Hellinger Distance and when to use it?

Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance.

Given two probability distributions, $P$ and $Q$, Hellinger distance is defined as:

$$h(P,Q) = \frac1{\sqrt2}\cdot \|\sqrt{P}-\sqrt{Q}\|_2$$

It is useful when quantifying the difference between two probability distributions. For example, if you estimate a distribution for users and non-users of a service. If the Hellinger distance is small between those groups for some features, then those features are not statistically useful for segmentation.


How to handle missing data for machine learning

There are three main approaches to handling missing data.

Impute - use some method to fill in the missing values with reasonable guesses. You could interpolate between two time points, take the average value over all time points, or use a variety of other techniques leveraging co-occurrence of other variables to get a reasonable estimate.
Ignore - some methods can just ignore missing data, and not use it in the model at all
Utilize - for cases where data is not missing-at-random, missingness itself can be an informative feature. You could include missing values as another data point to model your output.


What does 'lm' mean in the lmplot() in seaborn?

"lm" = "linear model". As reported in the docs:

  Plot data and regression model fits across a FacetGrid. [...] It is intended as a convenient interface to fit regression models across conditional subsets of a dataset.


What do positive and negative gradient values mean for Convolutional Neural Network?

As you know, the purpose of gradient maps like these is to show which pixels are contributing most to the prediction.

  The specific question . . . is why the positive values of gradient maps mean that object was detected there?

It's not positive/negative values that matter, but rather the magnitude of the gradient.  If the gradients associated with a pixel are small in magnitude, then that pixel is likely to be unimportant to the prediction (since small changes in the pixel's value translate to small changes in network outputs).  By contrast, if the magnitudes of the gradients are large, then it signifies that the pixel is important to the prediction.

  Why here the highlighted gradients maps have parts of dog/bird/spider highlighted?

It looks like the authors of the paper are post-processing the gradient map with a function they call $\psi$.  Notice the following passages from section 4.2 (highlighting was added by me)

and

The authors are taking the absolute value of the gradient.

So the gradient maps from this paper only show the magnitude of the gradient, not the sign.  When you see a part of the image highlighted, it doesn't mean that the gradients were positive.  Rather it means that the magnitude of the gradients were large.  Blueish regions correspond to near-zero gradients (low importance), while the orange-red regions correspond to large gradients (high importance).

Hope that helps!


How to analyse move kindness in python (Logistic regression, neural network, etc.)?

Okay, so from what I understand, you have a regression problem taking into account a variety of physical features. The reason I say that this is a regression problem, verses a classification problem is because the scale you are trying to predict is an ordinal scale.
There are a couple approaches to this. If your features are discriminative and linear enough, a simple least squares linear regression might work. If you believe the problem you have is to complicated for linear regressions, a simple vanilla neural network with one single output. I would recommend using the scikit-learn library in python for all models that are not neural networks. Here is a link to the generalized linear regression page.
That link has code samples and mathematical explanations. If you decide to use neural networks, and you don't have a great amount of samples or a need to use the GPU, the pyBrain library is great.
I wouldn't recommend using a logistic regression (since you mentioned it in your question), simply because a logistic regression is a classification problem, and I believe you would be better off approaching this from a regression standpoint.


Is is possible to see the input labels for a randomforest model?

It is impossible to retrieve column names from a trained Random forest classifier from my experience, there is also a previous answer for an identical question.


How to use prediction model after onehot encoding?

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestRegressor

df.head()

    Service    Tasks Difficulty     Hours
0   ABC         24     1           0.833333
1   CDE         77     1           1.750000
2   SDE         90     3           3.166667
3   QWE         47     1           1.083333
4   ASD         26     3           1.000000

df.shape
(998,4)

X = df.drop(["Hours"],axis = 1)
y = df.Hours

ct = ColumnTransformer([("cat", OneHotEncoder(handle_unknown = "ignore"),[0])], remainder="passthrough")

rf_model = RandomForestRegressor(random_state = 1)

model = Pipeline([("preprocessing",ct),("model",rf_model)]).fit(X,y)

x_test = pd.DataFrame({"Service":"SDE", "Tasks":90, "Difficulty":3}, index = [0])

# Ideally you split your data into train and test,in this case you need to pass x_test that is a pandas dataframe with the values you want to predict
model.predict(x_test)


Can accuracy become worse on the training set with more epochs?

Yes, accuracy measured on the training set over epochs is not monotonically decreasing.

Possible reasons:

Software bug
Nan due to division by zero / log (0) / overflow
Too big learning rate


Suggestions on how to explain 'models' & 'algorithms'

I think your definitions are mostly right....I would add that model is a specific implementation of algorithms designed to handle a specific issue or group of issues
Regarding pre-trained model, my understanding is that it can be used to build other models indeed, but the main use case is simply using it to predict or classify on new data, having already been trained previously


How do you overcome results oriented thinking from a client?

The scenario you described with one horse is basically anecdotal evidence. To objectively evaluate what you're doing, you need to agree with the client to make this an experiment.

  I understand you don't trust my horse predictions and that's totally
  understandable. It is human nature to worry about the ones that got
  away. Let's handle this objectively and run an experiment. I'll give
  you buy/sell ratings on 50 horses. We then track their performance for
  six months. If on average my ratings outperform a coin flip, we'll
  know we're onto something at a macro level even if a few got away. If
  my models end up being worse than a coinflip, we'll know the model
  needs to be improved or you need to fire me.


Dataset recommendation, ordinal target variable

Not sure what you are after, but Kaggle has many datasets you can use. You can use this search to get started:
https://www.kaggle.com/search?q=ordinal+datasetSize%3Asmall+datasetSize%3Amedium
another source:
https://www.gagolewski.com/resources/data/ordinal-regression/


Why is the memory of common GPUs so constrained?

Basically, the reason is because the memory of the GPU is much more expensive. This is due to the fact that GPUs using a very high memory bandwidth. A normal GTX 480 has 384 bit wide bus, which allows a maximum transfer rate of ~180 GB/sec. However, an Intel i7 CPU using a 192 bit wide bus  has a maximum of about ~30 GB/sec depending of what type of RAM you use. This means that the transfer of data in the GPU is about 5x faster compared to a CPU. Therefore, memory for GPUs is much more expensive but you can get more transfer ratio.

Additionally increasing more chips, the subsequent increase in capacity could end up messing with the timing of transfer of data.


What is the origin of YOLO/darknet coordinates

(0, 0)  is top left.

Here an a helpful blog that goes through all the features in the output vector.

This is common in image processing. There are a few reasons that this is the convention in computer vision. Check here for a list, the accepted answer states:

  This is caused in the history. Early computers had Cathode Ray Tubes (CRTs) which "draw" the image with a cathode ray from the upper left corner to the lower right.


How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)?

You need to discriminate between two types of neural networks. If your output variable is continous you can use linear, ReLU, tanh, logistic-sigmoid,... as activation functions, because these functions map continous inputs to continous outputs. If your output is discrete / categorical you can use the signum (binary) or softmax activation (multiclass) function as activation function for the output layer.

The cost function is often a function that is comparing the real outputs $y_n$ and the predicted outputs $\hat{y}(x_n)$ for the input $x_n$ for all $n=1,...,N$. Let us introduce the comparison function $D(y_n,\hat{y}(x_n))$. The comparison function has a low value if the predicted output is almost equal to the real output and high if the outputs are not similar. Assuming all the observations are equivalently important, we could sum the values of comparison function applied on all observations and obtain the integrated loss

$$J=\sum_{n=1}^ND(y_n,\hat{y}(x_n))$$

for the whole data set.

In order to see the influence of the activation function $g$ in the last layer we summarize the transfer function from the input to the last layer as $f(x_n)$.  Then the predicted output $\hat{y}(x_n)$ can be written as

$$\hat{y}(x_n)=g(f(x_n)).$$

Hence, the activation function at the output has an effect on the integrated loss $J$. For example if you choose the $\tanh$ as output activation you will bound your outputs in the intervall $(-1,1)$ which will be a bad choice if your outputs can be from $\mathbb{R}$ and your cost function will probably have a very high value while training. A better choice would be a linear activation function at the output layer.


Maximum Entropy modelling - likelihood equation

I'd merely add a comment if I could ...

What is your interpretation of p(y|x)? The usual expression is "the probability of y given x", i.e. the dependent variable corresponding to the independent variable in the probability distribution. This is the function you want to maximize, not the inverse function.

As a side note, there are two ways that the entropy form of the equation on the right can be used. This can be added to a cost function as a regularization kernel, with a modulating coefficient; or it can be the primary term in the cost function -- the entropy to be maximized -- with constraints from the physical system introduced with Lagrange multipliers.


How to group identical values and count their frequency in Python?

From this sample set I would expect a histogram of receipt that shows
  two occurrences of receipt 102857 (since that person bought two items
  in one transaction) and one occurrence respectively of receipt 102856
  and of receipt 102858.

Then you want:

  df.groupby('receipt').receipt.count()

receipt
102856    1
102857    2
102858    1
Name: receipt, dtype: int64


Data normalization before or after train-test split?

Normalization across instances should be done after splitting the data between training and test set, using only the data from the training set.

This is because the test set plays the role of fresh unseen data, so it's not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a potential bias in the evaluation of the performance.


How to move forward on Regression problem

You have little features (as in none hehe). Your data reminds me a lot of stock market predictions, where you know nothing about what you're studying except for price. Furthermore, price behaves chaotically, because if it was easily predictable, then everybody would be predicting it, thus removing its predictability.

Therefore, if you are serious about studying that data, you may want to look up literature on stock market prediction.

I suggested in a comment that one approach could be to see this as a classification problem rather than a regression problem. This is in fact what many people do in the stock market. Instead of predicting actual price, people study whether the price will go up or down (and therefore whether they should buy or sell).

Here is an example of what I mean, implemented in sklearn:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.cross_validation import cross_val_score
from sklearn.linear_model import LogisticRegression

# random signal
n = 100
t = np.arange(n)
x = np.random.rand(n)
plt.plot(t, x)
plt.show()

# model ups and downs: convert to 0s (down) and 1s (up)
t = t[1:]
y = ((np.sign(x[1:] - x[:-1])+1)/2).astype(int)
plt.plot(t, y)
plt.show()

# one look-ahead model using the four previous observations
X = np.c_[y[:-4], y[1:-3], y[2:-2], y[3:-1]]
y = y[4:]
scores = cross_val_score(LogisticRegression(), X, y, cv=10, n_jobs=-1)
print('Acc: %.2f (%.2f)' % (np.mean(scores), np.std(scores)))

My results:

Acc: 0.75 (0.17)

One advantage of this approach is that it gives you probabilities of going up or down. Remember, a logistic regression is a probabilistic model $P(y|x)$, and here we are taking $P(y|x)<0.5$ to mean down and $P(y|x)>0.5$ to mean up. But if the probability is close to 0.5, you could have your model say "I don't know". This way you only take action when the model reports down or up with high confidence (close to 0 (down) or 1 (up)). A colleague of mine wrote his thesis on this kind of thing. Are you trying to model sports in order to place bets? If so, this is a pretty simple but effective model of choosing when it is worthy to place bets.

But if the idea is to learn about data mining, then I would try another dataset.

sklearn comes with a bunch of nice datasets
kaggle has a lot of 101 training competitions rich in tutorials


Which recommender system: Content based or Collaborative filtering?

You are right. Content-Based Recommender System is the best approach to tackle such problems using historic information about the users.


What is the significance of model merging in Keras?

It is used for several reasons, basically it's used to join multiple networks together. A good example would be where you have two types of input, for example tags and an image. You could build a network that for example has:

IMAGE -> Conv -> Max Pooling -> Conv -> Max Pooling -> Dense

TAG -> Embedding -> Dense layer

To combine these networks into one prediction and train them together you could merge these Dense layers before the final classification.

Networks where you have multiple inputs are the most 'obvious' use of them, here is a picture that combines words with images inside a RNN, the Multimodal part is where the two inputs are merged:

Another example is Google's Inception layer where you have different convolutions that are added back together before getting to the next layer.

To feed multiple inputs to Keras you can pass a list of arrays. In the word/image example you would have two lists:

x_input_image = [image1, image2, image3]
x_input_word = ['Feline', 'Dog', 'TV']
y_output = [1, 0, 0]

Then you can fit as follows:

model.fit(x=[x_input_image, x_input_word], y=y_output]


Filling missing values for important features

I would first build some models without those predictors. If 2/3 of your training set is missing so you can't have your basic model based on them. If your accuracy is too low I would try to use a smaller data set and not use the missing values. Those kind of missing values are not somthing that you can easily define (like a default job) unless you have a prior on your data (e.g most people in the data are data scientists). If you do have any kind of such prior you can use missing data methods more efficiently.


Would a neural network trained on extracted features have the same accuracy as a full network with frozen layers?

When layers are "frozen" it generally means that their weights are not updated when backpropagation happens. So, technically, there is no difference between:

Using a "frozen" VGG16 and training some fully connected layers and
Using the VGG16 embeddings and training some fully connected layers

In practice, if you did both and compared the results, you may see differences if you haven't accounted for the fully connected layers' random initialisation.


There are various approaches for dealing with missing values. Suppose we've got 4 instances in a dataset:

x1 = [1 2 3]
x2 = [1 ? 3]
x3 = [2 4 2]
x4 = [1 3 3.5]

one simple approach (specially popular in medical datasets) is finding values with regard to most similar instance; in the above case, missing value of x2 would be then: 2 (since x1 is the most similar)
a more sophisticated approach is weighted averaging through k most similar instances (of course only applicable if the missing value is numeric or at least ordinal) in this case, you should calculate: x2(2) = (2*(1-(0/3)) + 3*(1-(0.5/3)) / (2-(0/3)-(0.5/3)) (x4 has also been counted as a similar case
another approach, is voting among k most similar instances (applicable both for categorical and ordinal values).

cases 1 and 3 are what you have implemented (case1 = 1 Nearest Neighbor / case2 = k-nearest neighbors)

there are other approaches for handling a missing value, but it depends on what you're going to do with your dataset. For example in very large datasets, sometimes it is efficient to simply ignore every instance that contains at least one missing value, or ignore only the missing value (not the entire instance) in further processes (e.g. in VFI algorithm)


1)Random search if often better than grid
https://www.analyticsindiamag.com/why-is-random-search-better-than-grid-search-for-machine-learning/
Try:

#clf = RandomizedSearchCV(clf ,param_distributions = params, cv=kfold, scoring="accuracy", n_jobs= 10, verbose = 1)

2) For unbalanced data set:

Resampling: undersampling or oversampling
create new data points

https://towardsdatascience.com/methods-for-dealing-with-imbalanced-data-5b761be45a18


Bad results for training set and good results for test set

Seems quite strange. I bet you've got a bug or test set size is small enough. Like if there're only 2 objects: 1 positive and 1 negative, you could have classified them correctly even with a random classifier. I assume your test set is not that small, which means there's a bug somewhere. Also, in theory, if both of your sets are large enough and are drawn from the same data distribution, you can usually have one of 2 types of behavior:

about the same result both on training and testing sets (when your model underfits or fits well)
better results on the training set then on the testing (when your model if overfit)


You deal with a highly skewed censored distribution which makes it really hard to get good estimates. The key question is what information is available to model the skewed income distribution (you don't say anything on that).
There is quite some literature on the issue of imputing top incomes. E.g. "Measuring inequality using censored data: a multiple‐imputation approach to estimation and inference" by Stephen P. Jenkins  Richard V. Burkhauser  Shuaizhang Feng  Jeff Larrimore. The abstract kind of nails the problem:

To measure income inequality with right‐censored (top‐coded) data, we
propose multiple‐imputation methods for estimation and inference.
Censored observations are multiply imputed using draws from a flexible
parametric model fitted to the censored distribution, yielding a
partially synthetic data set from which point and variance estimates
can be derived using complete‐data methods and appropriate combination
formulae. The methods are illustrated using US Current Population
Survey data and the generalized beta of the second kind distribution
as the imputation model. With Current Population Survey internal data,
we find few statistically significant differences in income inequality
for pairs of years between 1995 and 2004. We also show that using
Current Population Survey public use data with cell mean imputations
may lead to incorrect inferences. Multiply‐imputed public use data
provide an intermediate solution.

Also see "Recent Trends in Top Income Shares in the United States: Reconciling Estimates from March CPS and IRS Tax Return Data" to get an idea how to model a censored right-skewed income distribution in a proper way.


Transform a skewed distribution into a Gaussian distribution

You can do a log transformation on your data with the help of numpy log functionality as shown below :

log_data = np.log(data)

This will transform the data into a normal distribution. Moreover, you can also try Box-Cox transformation which calculates the best power transformation of the data that reduces skewness although a simpler approach which can work in most cases would be applying the natural logarithm. More details about Box-Cox transformation can be found here and here


How to combine PCA and MCA on mixed data?

You may want to use Factor analysis of mixed data.

It allows you to do dimension reduction on a complete data set.

A R implementation could be found in the FactoMineR package. But this function struggle when you have a high number of data/columns.

I am not aware of the existence of the equivalent in python.


XGBOOST : model.predict_proba() and model.predict() conflicting behaviour

If you have two classes i.e you're doing binary classification, you should specify the target labels as 0 or 1 (neg/pos). The predict_proba() method returns estimated probabilities that a sample is in class 1.

"In the binary classification case, '1' is used to indicate positive samples, and '0' is used to indicate negative samples. We also support probability values in [0,1] as label, to indicate the probability of the instance being positive." - https://github.com/dmlc/xgboost/tree/master/demo/binary_classification


Does convergence equal learning in Deep Q-learning?

Policies found by Deep Q-Learning, even after convergence, are not guaranteed to be optimal. The reason is that the neural networks that approximate the Q function in DQN inherently come with a statistical error (bias and variance), a pointer can be found here.
Furthermore, convergence to the optimal policy for tabular Q-learning is only guaranteed when every action is sampled infinitely often in every state. This may make using any 'converged and optimal policy' in experiments very hard, even when disregarding additional complexities from function approximation in DQN.
Please be aware, additionally, that generalization in supervised learning requires that the training and test data are sampled from the same distribution. Similarly for RL, the training and test environments are assumed to be sampled from the same 'distribution'. Generalization to environments not reflected in the training set is referred to as transfer learning in the RL literature.
This is an important and very interesting line of research though, so please be not be discouraged by this comment.


Some reasons the network may not be getting better:

Learning rate is too high (jumps back and forth forever)
Learning rate is too low (when divided by training set size, goes to zero)
You're replacing the error signal with the weight gradient signal.  The reuse of error (based on the indentation) seems to be right on the same line as the inner loop.  I'm not sure if that's correct.  First, backprop your errors, THEN calculate the gradient based on the activation of the hidden neurons and delta sigmoid(error).


In normal RNNs, you can train using either back-propagation through time (BPTT) or teacher forcing. With BPTT the network receives as input for each time step the output of the previous time step. In teacher forcing, the network receives the gold data tokens directly; this induces exposure bias on the trained network.

With textual GANs, you normally don't feed the gold data to the generator, because the generator is generating the sequence so there is no gold data to use.

In the article you refer, they are taking subsequences from real data, feeding them to the generator and having it only generate the final token. Then the discriminator receives the concatenation of the real sequence prefix and the generated final token. This is what they call "teacher helping".


Python giving memory error while fitting the text into vector using various vectorizer

to answer that question you need to get and give us more information, no chance to advise without knowing where the OOM is thrown and how the memory profile looks like. You may tune the memory usage of your program also in other parts than the part where the error came from, to raise the limit.

Just one hint for a cause that can be resolved without getting more physical memory:
Sometimes you can "lazy load" large data sources. Get it record by record or chunk by chunk, then process it, and write results to the sink, before you get the next record into memory. This is done with python generators, watch out for the keyword "yield" that is used instead of return.
https://stackoverflow.com/questions/1756096/understanding-generators-in-python

further you may be better off using low level libraries for certain tasks, see i.e. http://www.nltk.org/book/ch06.html  2.4


Determining which entries in input vector have the greatest effect on output vector

I'm not very experienced in the interpretation of neural networks but I have two suggestions.
1: Alter the values of the input, one feature at a time, and watch how the output varies.
2: Use another model with more interpretability. You could do this just for interpretation.


Error in using fit() on RandomForest Classifier where X was a pandas.DataFRame object

You can't pass a categorical variable as it is to one of sklearn's classifiers. One approach for dealing with this is to dummy-encode the column in question.

I realise that it's not a strict requirement to post minimal, complete, verifiable code on this site, but it will help if you provide code that can be run by others so that they can go straight to the solution without having to guess what's happening from your error message.

Here's some code which reproduces your error:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier

df = pd.DataFrame({'sex': ['male', 'female', 'female', 'male', 'female'], 'survived': [0, 1, 1, 0, 1]})
rf = RandomForestClassifier()
rf.fit(df.drop('survived', axis=1), df['survived'])

We can fix the error by using the get_dummies function from pandas. The following code gives no errors:

df_dummies = pd.get_dummies(df)
rf.fit(df_dummies.drop('survived', axis=1), df_dummies['survived'])


Using keras with sklearn: apply class_weight with cross_val_score

The sklearn MLPClassifier does not implement any option for class weights at the moment. There are at least two paths for you to follow. You could go with writing a custom loss function, which allows you to stay in the sklearn framework without reaching out to keras. Another option is to implement the cross-validation yourself, which is not difficult to do, and run your keras model for each fold. An example can be found here.


Training an AI to play Starcraft 2 with superhuman level of performance?

You pose an interesting question. The problem that I see is that even if you develop all the items you mentioned (and I'm not sure you need all of them) you wouldn't have the computing power needed for this.

You see, most of these items are based on reinforcement learning. Which means that the models are given a relatively small set of data (roughly speaking, the rules of the game) and then are set off to play millions & millions of games. Read the whitepaper that was written by Google on beating Go. They basically admit that there whole point was to write something very, very simple that could learn on it's own and then just set it free on Google's massive power. It's nothing for them to say, "let's have this robot play 10 million games". So the level of sophistication from your initial model is pretty low, you just need to make it an exceptional learner.

So you make an exceptional learner of Starcraft - then what? How are you going to have the power to make your algorithm to play millions of games?


UPDATE: the landscape has changed quite a bit since I answered this question in July '14, and some new players have entered the space.  In particular, I would recommend checking out:

TensorFlow
Blocks
Lasagne
Keras
Deepy
Nolearn
NeuPy

They each have their strengths and weaknesses, so give them all a go and see which best suits your use case.  Although I would have recommended using PyLearn2 a year ago, the community is no longer active so I would recommend looking elsewhere.  My original response to the answer is included below but is largely irrelevant at this point.

PyLearn2 is generally considered the library of choice for neural networks and deep learning in python.  It's designed for easy scientific experimentation rather than ease of use, so the learning curve is rather steep, but if you take your time and follow the tutorials I think you'll be happy with the functionality it provides.  Everything from standard Multilayer Perceptrons to Restricted Boltzmann Machines to Convolutional Nets to Autoencoders is provided.  There's great GPU support and everything is built on top of Theano, so performance is typically quite good.  The source for PyLearn2 is available on github.

Be aware that PyLearn2 has the opposite problem of PyBrain at the moment -- rather than being abandoned, PyLearn2 is under active development and is subject to frequent changes.


data analysis EDA issues, indent or type

Try something like this (the function also needs to know what cols is - perhaps pass that in as another parameter?):

def read_auto_data(auto):
    for column in cols:
        auto[column] = pd.to_numeric(auto[column])
    return auto

auto = read_auto_data()


Analyze additions and subtractions to/from a data set

This response is based on your domain. For,e.g. sales, a product may not be sold every day and so there would be no record of it.

According to your dataset in which each day you would expect to see an occurrence of the product in your data set, you could consider the following approaches.

Mathematically:

A = the set of all possibilities (i.e. product occurrence for each date - you could generate this and use )

B = sample data set provided would be provided

C = A - B
  = days that a product was missing.

DPART1 = You could then continue by retrieving the min(date) and max(date) for each product in the dataset to represent the introduction of the new product and possible cease of an existing product.

D= You could then filter the dataset (C) to remove dates less than the min(date) and greater than the maxdate).

In terms of sql:
A - Cartesion/Cross product of all your products and dates
B = Your current sample data set
C = SELECT * FROM A MINUS SELECT * FROM B
DPART1 = SELECT PRODUCT, MIN(date) as INTRODUCED_DATE, MAX(date) as CEASED_DATE from YourSampleDataSET GROUP BY PRODUCT
D = SELECT C.PRODUCT, C.Date FROM C LEFT JOIN DPART1 ON C.PRODUCT = DPART1.PRODUCT
    WHERE DPART1.PRODUCT IS NULL OR (
       C.Date BETWEEN DPART1.INTRODUCED_DATE AND DPART1.CEASED_DATE
    )

Nb. "DPART1.PRODUCT IS NULL" ensures that you do not filter products that may not be in your sample subset.

Operational considerations, I would recommend indexes in your database to assist with the queries, partitioning where possibly.