Why mini batch size is better than one single "batch" with all training data?

The key advantage of using minibatch as opposed to the full dataset goes back to the fundamental idea of stochastic gradient descent1.
In batch gradient descent, you compute the gradient over the entire dataset, averaging over potentially a vast amount of information. It takes lots of memory to do that. But the real handicap is the batch gradient trajectory land you in a bad spot (saddle point).
In pure SGD, on the other hand, you update your parameters by adding (minus sign) the gradient computed on a single instance of the dataset. Since it's based on one random data point, it's very noisy and may go off in a direction far from the batch gradient. However, the noisiness is exactly what you want in non-convex optimization, because it helps you escape from saddle points or local minima(Theorem 6 in [2]). The disadvantage is it's terribly inefficient and you need to loop over the entire dataset many times to find a good solution.
The minibatch methodology is a compromise that injects enough noise to each gradient update, while achieving a relative speedy convergence.
1 Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 (pp. 177-186). Physica-Verlag HD.
[2] Ge, R., Huang, F., Jin, C., & Yuan, Y. (2015, June). Escaping From Saddle Points-Online Stochastic Gradient for Tensor Decomposition. In COLT (pp. 797-842).
EDIT :
I just saw this comment on Yann LeCun's facebook, which gives a fresh perspective on this question (sorry don't know how to link to fb.)

Training with large minibatches is bad for your health.
More importantly, it's bad for your test error.
Friends dont let friends use minibatches larger than 32.
Let's face it: the only people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. That's a terrible reason. It just means our hardware sucks.

He cited this paper which has just been posted on arXiv few days ago (Apr 2018), which is worth reading,
Dominic Masters, Carlo Luschi, Revisiting Small Batch Training for Deep Neural Networks, arXiv:1804.07612v1
From the abstract,

While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance ...
The best performance has been consistently obtained for mini-batch sizes between m=2 and m=32, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.


Handling Numerical Categorical Column in ML models in Python

If you don't encode numerical categories with dummy variables, some models will end up being trained to use an ordering of numbers (e.g. 1 < 2 < 3 < 4 < 5 < ...) in their predictions. Whether or not this is desirable or useful depends on the context, and in particular on the meaning of the numerical categories and the model and implementation being used.

In your example, if the numerical labels for the Pclass categories represent "poverty class", with lower numbers corresponding to less income and higher numbers corresponding to higher income, then the ordering of the integers is really related to a meaningful ordering of the categories ( "1 < 2 < 3" and correspondingly "poverty class 1 is less wealthy than poverty class 2 is less wealthy than poverty class 3"). In this case, using the numerical encoding as it is doesn't appear to do any harm, and may even help some models. For example, a decision tree could learn rules like

if (Pclass >= 2):
    prediction = "survive"

whereas if the categories were encoded with dummy variables, this rule would instead look like:

if ((Pclass_2 == True) or (Pclass_3 == True)):
    prediction = "survive"

For some implementations/models, the first rule may be more likely to occur than the second, be more cost effective (just storing dummy variable data is more costly, let alone the effect it might have on model performance), generalize to new data better, etc.

On the other hand, if the numerical labels for the Pclass category represented, say, location of the room on the boat in which a person was staying (with 1 representing right next to the iceberg hit, 2 representing the super fancy ballroom with it's own getaway boat, 3 representing near-ish to the iceberg hit, 4 representing mid-tier, etc.), then the ordering of the integers appears to have no meaningful relationship to an ordering of the actual classes (that I can see anyway!). If one were to use the encoding for the numerical categories as they are, many model implementations could learn patterns in a "bad" way. For example, a decision tree could learn that the people near the ice berg hit had poor survival rates and end up producing the rule:

if Pclass <= 3:
   prediction = "does not survive"

This is "bad" both because it assumes being near the iceberg hit has something to do with the numerical encoding being less than 3 (it may not, for example future data might include many new category labels greater than 3 all of which encode rooms near the iceberg hit), and because the rule seems likely to produce many incorrect predictions for those with Pclass=2. If the categories were encoded with dummy variables, the rule learned might instead be:

if ((Pclass_1 == True) or (Pclass_3 == True)):
    prediction = "does not survive"

This rule is "better" in this context, as it does not share the two issues the previous rule did.

In general, I suggest the following rule of thumb:

  Use dummy variables to encode numerically labeled categories if the natural ordering on the numerical labels has no meaningful interpretation in terms of the categories.

All else being equal, I tend to prefer using dummy variables by default when I have the space in memory, and then think on whether it makes sense to use integral encodings. In your situation, what does the Pclass represent, and is the ordering of the numerical categories actually meaningful to you? If not, I would recommend using dummy variables.


Neural Network cannot learn nonlinear function

You should try using a sigmoid activation function, as leakyRelu is very close to a linear function, and remove one hidden layer. Just one hidden layer should be engough as you have only one input, with a very small length, maybe 3 or 5 neurons.


Best strategy for extracting specific structured data from unstructured sentences

The task is a specific case of  NER (technically NER is a sequence labeling task, a special case of classification).
I think you would have two main options:

Apply a pre-trained NER model: most deal with time entities but not always very accurately, and it wouldn't be specifically adapted to your data so you wouldn't obtain the distinction between the three types of values. Advantage: no need for training data.
Train your own NER model: that's the ideal scenario in terms of performance, assuming you have (or can have) a good amount of annotated data for training.


Agglomerative Clustering Stopping Criteria

Clearly, 2 standard deviations beyond Omega is not the same as twice the mean.

Apparently, their process is this:

compute the distance matrix
compute the mean
compute the standard deviation
compute hierarchical clustering with maximum linkage
cut the tree at mu+2*sigma

Because complete linkage is in O(n^3), this approach will not scale to longer videos or higher frame rates.


It largely depends what kind of paper you are writing — you haven't provided too much information in that regard. You could discuss a selection of models that have scored the highest on ImageNet. Most of these networks have historical significance, and brought a new aspect to the table.


Is it reasonable that a problem "solved" by a traditional ANN can also be solved by a CNN?

In theory yes in practice maybe.

What do I mean by that. There is universal approximation theorem for smooth functions regarding ANN, there is also theory backing this up for CNN variant. Another thing that backs this claim up is that you can replicate any ANN with CNN architecture, hence effectively backtracking to the original ANN.

Practice, you say this data can also be processed as images it can be processed but does it have the same amount of informativness as other data representation? Could be but it does not have to, think about doing regression with images of time series data, sure you can catch some general trends but to really catch all of it it will get expensive, really fast.


(Crowdsourced) Dataset with label/annotation metadata like duration/quality

I did a project with the CheXpert dataset which includes metadata and labels for uncertainty. I don't know if it's the right kind of metadata you are looking for. In general, medical datasets usually contain metadata. A similar one would be from the NIH.
Otherwise, this is a decent index of high quality datasets.


Is there a package for using SQL to manipulate Pandas dataframes in Python?

I found a package called pandasql, which is based on sqldf for R. It seems quite a bit slower than doing the transformations with the pandas package, but it gets the job done. Just put the SQL query into a string like this:

query_string = """
    select * from df
"""

Then use the string in the pandasql.sqldf package, as follows:

new_dataframe = pandasql.sqldf(query_string, globals())

Choose globals() or locals(), depending on the scope you want for your variables.

As I mentioned, it seems a bit slow, but I couldn't find anything else. I may use this from time to time until I become better at Pandas.

Sean


How does Sigmoid activation work in multi-class classification problems

If your task is a kind of classification that the labels are mutually exclusive, each input just has one label, you have to use Softmax. If the inputs of your classification task have multiple labels for an input, your classes are not mutually exclusive and you can use Sigmoid for each output. For the former case, you should choose the output entry with the maximum value as the output. For the latter case, for each class, you have an activation value which belongs to the last sigmoid. If each activation is more than 0.5 you can say that entry exists in the input.


A multivariate linear regression for explaining impacts of the predictors

To handle autocorrelation, you should try to "subtract it" from the series. In other words, using differentiation, de-seasonalization, and transformations you should try to subtract the impact of time on your time series data. Once you end up with a distribution that looks like white noise, then you can run your regression.

You can do it with an ARIMA model, simply add all the AR(), I(), and MA() components in the regression equation, and then ignore them when you evaluate the impact of other coefficients.


Image classification: Predict all objects' classes without bounding boxes

Object detection doesn't mean bounding boxes. It is as the name suggests to detect objects in an image being single or multiple which is what you actually need in your problem.

The display of bounding boxes is your choice if you need the class along with its position or not. As you already get bounding boxes in your solution, you might be getting their classes as well. Simply, don't draw the bounding boxes using OpenCV or any other. Just comment out those lines.


XG Boost result interpretation for unbalanced datasets (Accuracy & AUCROC)

With Success already being the larger class, you probably shouldn't be using a scale_pos_weight larger than one: you want to scale the positive class's contribution to the loss function down rather than up.
I suspect that's what's happening in the first case.  With scale_pos_weight=75, the model ends up basically only caring about the positive class, predicts everyone is in the positive class, and so your accuracy is just a little better than the 77% baseline you'd expect with that strategy.  With that motivation, it's not too surprising the AUC is poor, although I wouldn't have expected a drop all the way to the 50% baseline...
If you don't use scale_pos_weight (you said "if I did not tune", but does that mean you left it at the default 1?), then the model performs better in rank-ordering (AUC=70%), but not so well in the hard classification.  You might want to tweak the prediction threshold here; there's probably a different threshold that will perform better for accuracy score.  You could also try scale_pos_weight=0.25 or so; that should make the default cutoff better, hopefully with little effect on AUC?


Orange3 doesn't calculate the OOB. This must be implemented a separate code. More information about how to calculate the OOB error in scikit-learn.


Multi-class classification with mostly zero valued data

Welcome! Sounds like your data set is highly imbalanced. For techniques on handling an imbalanced data set, please have a look at this. https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

EDIT: Looks like I misunderstood the question, sorry! Also: I think the term "multi-class regression" does not exist. It sounds more like you're doing multi-class classification. If your input features are consistently 0, then it sounds like your actual value of each feature and the mean of the feature are identical, in which case you have very poor features. The features will not do a good job of differentiating between the classes. You should try (or hopefully use a neural network to learn) a new set of features.


Processing data in the right manner in data science

The goals of all these methodological guidelines is to avoid data leakage.

Example: let's imagine we want to classify short messages (e.g. tweets). When inspecting the data we find various kinds of smileys: :-), :|, :-/... At preprocessing stage we replace all smileys found in the data with a special token like <smiley> (or something more specific).

If the detection/replacement is done on the whole data, every occurrence of a smiley in the test set is replaced with <smiley>.
If the detection/replacement is done on the training set, even after preprocessing there might be a few smileys left in the test, because some uncommon ones didn't appear in the training set.

In the first case there is data leakage: we fixed some issues in the test set even though this wouldn't have been possible with actual fresh data (here the variants of smiley that were not seen in the training set). In the second case the test set is "imperfect", i.e. it's exactly as if it was made of "fresh" unseen data, therefore the evaluation will be more realistic.

This example shows why it's always safer to separate the data first, design the preprocessing steps on the training data, then apply exactly the exact same preprocessing steps to the test data.

In practice there can be cases where it's more convenient to apply some general preprocessing to the whole data. The decision depends on the task and the data: sometimes the risk of data leakage is so small that it can be neglected. However it's crucial to keep in mind that even the design of the preprocessing can be a source of data leakage.


Identify objects (bus) on the map based on coordinates (lat, lon)

You might build a logic of distance. This will raise an issue with buses passing each other. For curiosity,  can't you identify source of signal?

To be honest, me thinking of same problem but from moving objects identifying (computer vision), so you can consider approaches out there.

Good luck.


Question about reshaping array size for KNN Classifiers

Seems you are making two mistakes -
Reshaping dim. is mismatched with the size i.e. 10 != 3X3
Dim in train and test are different. Trained of 5 features and testing on 4 features.
What I will suggest -
 You would be aware of your features count - let's say N
 then reshape using reshape(-1,N). It will automatically decide the first dim.
x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
N=2
x.reshape(-1,N)


It can be made in many ways. My starting approach would be (1) to apply the tf-idf as in your code snippet, then (2) reducing the output matrix to a lower dimensionality (say 2D for seeing it in a scatterplot) with a decomposition method (or with topic modelling approach), and finally (3) apply a clustering algorithm to visualize the samples/documents of each cluster.

However, this may have many flaws, because I don't know if the decomposition results would make any sense and I'm not aware of how many clusters are good enough, but anyway it would be useful as an exploratory analysis.

The logic would be something like that: (didn't tested)

# 1
tfidf_vect = TfidfVectorizer(tokenizer= sent_tokenization)
tfidf = tfidf_vect.fit_transform(df['Sentences'])

# 2
decomp_method = NMF(n_components=2)
tf_2d = decomp_method.fit_transform(tfidf) # output shape: [N, 2]

# 3
kmeans = KMeans(n_clusters=2)
tf_kmeans_labels = kmeans.fit(tf_2d).labels_ # array

# visualization
data = pd.DataFrame(tf_2d, columns=['c1', 'c2'])
data['kmeans_labels'] = tf_kmeans_labels
scatterplot(x='c1', y='c2', hue='kmeans_labels',data=dat)
```


Is applying pre-trained model on a different type of corpus called transfer learning?

The basic concept of transfer learning is: Storing knowledge gained from solving one problem and applying it to different but related problem
I guess to be precise this is called Transductive Transfer Learning. In this we learn from the already observed training dataset and then predict the labels of the testing dataset. Even though we do not know the labels of the testing datasets, we can make use of the patterns and additional information present in this data during the learning process.
Refer: Ruder


Openai Spaces for a modified environment

In your case it seems you simply can return a 2D vector and extract out the components for that. You can take a look at the MountainCar example under classic control envs for a fully working case. But something like:

vec_1_low = 0
vec_1_high = 366
vec_2_low = 0
vec_2_high = 1

space_low = np.array([vec_1_low, vec_2_low])

space_high = np.array([vec_1_high, vec_2_high])

obs = spaces.Box(space_low, space_high, dtype=np.float32)

Then:

vec1_obs, vec2_obs = obs.sample()


Use output of K-Mean for Logistics regression

It is strange to use k-means in addition to logistic regression. Usually k-means is reserved for unsupervised learning problems, this is when you do not have labelled data. Unsupervised learning algorithms are not as powerful and it seems here you have labelled data, thus you should stick to supervised learning techniques.

If you have additional features for your instances then you should include them in your logistic regression.


Yes you are right. But I think the authors only want to emphasize that if $||1|| > ||2||$, the decision boundary will be closer to $2$ and vice versa.


What is a lower bound on the vocabulary size for generating word/sentence embedding using word2vec or skip thought vectors?

It's not a straightforward question to answer as it is hard to compare the quality of two word2vec models with a meaningful metric. You could, of course, use the loss function, but that won't give much.

Another approach is  more heuristic: take for example the frequency of each word, and remove those words that are repeated less than N times, where you can set N to for example 10 or 20. This is common practice as you need a certain number of repetitions of the same word to have some meaningful results.


Inputting (a lot of )data into a dataframe one row at a time

I'm getting timeit results of about 1/4 of the time using:

flatX = X.apply(lambda x: x.flatten())
pd.DataFrame(item for item in flatX)

See also
https://stackoverflow.com/questions/45901018/convert-pandas-series-of-lists-to-dataframe
for some possibly better options for the second line.

(Regarding my earlier comment, I don't get any real savings by just dataframe-ing at the end.)


Using Amazon Personalize to build a Recommendation System

As far as I have researched, Amazon Personalize is not useful for this project.
This is because Amazon Personalize requires User-Item Interaction data to train the model.
To use Amazon Personalize, you need a dataset with historical data that contains at least the fields:

User
Item
Date

where the Date field represent the time when the user interacted with the Item.


What are common ways to identify subject and object from a question?

Google SyntaxNe has pre-trained models for French.  It is open source and can be customized for a specific use-case.

https://github.com/tensorflow/models/blob/master/research/syntaxnet/g3doc/universal.md

With this, you can create a dependency tree for questions and evaluate fitness for this application.


seaborn heatmap not displaying correctly

Current version of matplotlib broke heatmaps. Downgrade the package to 3.1.0
pip install matplotlib==3.1.0
matplotlib/seaborn: first and last row cut in half of heatmap plot


Using TSNE to Visualize Clusters in Python

As far as i see, you have your temporal columns involved. They are sequential variables and create such curves in visualization.

Try to exclude them from your data and visualize again. If you still the curves, update the post with new plot and we discuss further.


Interpret User Interfaces with Machine Learning

I would try experimenting with recurrent neural networks: http://karpathy.github.io/2015/05/21/rnn-effectiveness/. Recurrent neural networks can output sequences of variable length given inputs of variable length. In your case a recurrent neural network might output a sequence like the following when given a user interface: click a button, select a field, type some text, hit enter. For another interface, the network might output only: click one button, click another button, and that's it. This would be useful for you because the sequence and length of actions from interface to interface might change a lot.

You could also experiment with reinforcement learning and build an algorithm that has an objective (reach some final page in as few actions as possible). The algorithm would start by doing random things (like clicking the same button a bunch of times), and then gradually learn over time to take appropriate actions. If you go that route you could use deep learning and Monte Carlo Tree Search (MCTS) like what Alpha Go did.

In either case you're going to need a framework that can train an algorithm quickly because you're likely to have to go through a lot of iterations. TensorFlow (https://www.tensorflow.org/) is one option (I've started using it recently, and I like it a lot because of its easy of use). TensorFlow is capable of building both recurrent neural nets and deep neural nets.


Searching for a 3D Dataset, segmented by 2 or more Experts

You can look into the MICCAI dataset. This contains 35 3D MRI scans of brains. The brains have been parcelated into 134 distinct brain regions by professionals.


According to me, for model training we should always use the actual data, so that your prediction is always close to realty.

  But in case if I have the prediction data as well,

  I will trained another model with the prediction data.
  Analysis the outcome of the two models.
  Identify the deviation between two outcomes.
  Will use as threshold(variation) in future prediction.


Difference between RFE and SelectFromModel in Scikit-Learn

They effectively try to achieve the same result but the methodology used by each technique varies a little.

RFE removes least significant features over iterations. So basically it first removes a few features which are not important and then fits and removes again and fits. It repeats this iteration until it reaches a suitable number of features.

SelectFromModel is a little less robust as it just removes less important features based on a threshold given as a parameter. There is no iteration involved.


Is it possible to integrate Keras and TensorFlow code?

A naive approach to integrate Keras and Tensorflow:

input_img = tf.placeholder(tf.float32, (1,200,300,3), name='input_img')
vgg19 = tf.keras.applications.VGG19(weights='imagenet', include_top=False)
output = vgg19(input_img)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    output_val = sess.run(output, {input_img:
                          np.expand_dims(img,0)})

output_val.shape, output_val.mean()


If you are training a classifier on only one angle, you will overfit to that particular position - it won't be able to recognise the same car from a different angle. It may learn something, I guess, and you could test generalisation by testing on the unseen images from the other two angles. So train on each set individually and test on all 3 sets individually with the usual caveats on training vs testing.

A better approach, however, would be to train on all your data. The network will automatically learn what is the most efficient and you can then look at the results on each of the 3 test sets to see which gives the best results. I am guessing that this is an example problem, so more generally it makes sense to do a statistical analysis on why the classifier fails. If you can identify which features make classification hard, you can work harder to create a training strategy that will improve the model. The main goal would be to make a model that generalises. I suspect that training using extra information from other angles will only improve the effectiveness for all cases.


Library to calculate Distance Metrics on GPU

If you are using python, use CUDA tool-kit with cudamat library. You will be able to perform matrix operations on GPU using cudamat.


If you remember the theory, CNN kernels basically work on the principle of edge detection. So now that your picture is purely blue or purely red there are no edges. I am guessing that is the main problem here, since I think ML libraries initialise kernels which are suitable for edge detection.
To be more comprehensive:

Since kernels are randomised small numbers, sum of red kernel is almost equal to sum of  blue kernel. The next layer will mainly have (sum of red kernel * 255) and (sum of blue kernel * 255) in all the channels for the next layer, which will be roughly equal. Since you are using an inappropriate loss function, the CNN becomes insensitive to small changes due to the difference in summation of kernels.

I would suggest:

Changing the padding to same might work since then edges will be introduced in the picture (black-red, black - blue).
Try by adding significant white edges to the side of the picture.
Squared error does not work that good for sigmoid activations, you need to use the cross entropy loss.
One must i believe experimenting with smaller learning rates, since CNN have the problem of exploding gradients.


Multi-class classification v.s. Binary classification

The greater the number of output nodes the higher complexity you will add to your model. This means that given a fixed amount of data, a greater number of output nodes will lead to poorer results. I would use a ABCD vs. others strategy.

Instead of conditioning your model to learn the distributions of the class A, B, C and D separately you will combine them. This means that is A and B are different in some way, but this difference is irrespective of the classification with "others" then there is no need to learn that distinction.

For example: if you want to detect dog, cat, human with features such as weight, height and number of legs. The number of legs feature will have relatively low importance, because cats and dogs will likely all have 4 legs. However if I want to classify cat/dog vs humans, then the number of legs will be the most important feature. It might be the only feature you need.

One caveat may be severe class imbalance. By combining your classes in this way you may end up with an over representation of ABCD. You can use techniques such as anomaly detection to train a model on your ABCD data and then detect whether a novel instance falls within this distribution, or is an outlier in which case you would label it as "others."


normalization/denormalization for linear regression problem

I think it is ok, as long as your training and test data have the same maximum values for every feature, approximately. The idea is that the scaling has to be done with the training set (remember that using the test set for anything that is not testing is illegal, not even for scaling).

So, you actually fit $y'$ as a function of $X'$, and you have a model that maps properly $y' = f(X')$. When you get your test data, you just obtain the predictions by doing $f(X_{test}')$. As the paragraph before states, if you have that $scale \approx scale_{test}$, then you can just recover $y_{test}$ by doing $y_{test} = scale \cdot f(X_{test}')$.

Edit: Don't worry about nonlinearities

Even if the function $f$ is highly nonlinear, it is a function capable of mapping $X'$ to $y'$. If you trust this function and trust the fact that $y = y' \cdot scale$, then there is no need to worry about the way $f$ acts, as function composition makes sense for all kind of functions, both linear and nonlinear.


Why is the variance going down so much in this weight initialization problem(using pytorch)?

Basically, because the variance is quadratic in w.

You can consider your problem as computing the variance over scalar products $z = \vec{x}\vec{w}$ by drawing 500000 samples of $\vec{w}$ from $p(\vec{w})$. Let's compute the mean first

$$ E_{p(\vec{w})} [ \vec{w}^T\vec{x} ] =  \vec{\mu}_w^T \vec{x} = 0$$

because you draw the $\vec{w}$ from a zero-centered Gaussian. With the expectation value as zero, the variance is given by

$$ E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\
= E_{p(\vec{w})} [ \vec{x}^T\vec{w} \vec{w}^T \vec{x} ] \\
= \vec{x}^T E_{p(\vec{w})} [ \vec{w} \vec{w}^T ] \vec{x}
$$

$E_{p(\vec{w})} [ \vec{w} \vec{w}^T ]$ is the covariance matrix of the $\vec{w}$ and since they stem from independent standard normal distributions is diagonal with $\sigma_w^2 = 1$ on the diagonal. So that gives

$$
E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] = \vec{x}^T \vec{x} \approx 512
$$

as you are approximating the sum of variances of 512 independent samples of standard Gaussian. If you now scale the $\vec{w}$ with a constant $k$, you can do this calculation again and it will give you

$$
E_{p(\vec{w})} [ (\frac{1}{k}\vec{w}^T\vec{x})^2 ] \\
= \frac{1}{k^2} E_{p(\vec{w})} [ (\vec{w}^T\vec{x})^2 ] \\
\approx \frac{1}{k^2} 512
$$

So in order to end up with 1 in the very end, you need to scale with $k=\sqrt{512}$.

I hope this answers your question.


Keras binary classification model is not learning anything. How to improve my model?

I wrote few notes on your implementation:

Classification on imbalanced data can be a problem. How imbalanced is your dataset? You can try to compensate by training your model by feeding balanced mini batches of data. You artificially build each batch so that classes are more or less equally represented, so that your classifier can learn from all of them.
I would change the last layers. I suggest you to use softmax activation at the output layer, and a crossentropy loss function that is specific for this task. The number of output nodes must be equal to the number of classes.
Adam optimizer is commonly referred to as the best algorithm for gradient descent.
MAE is a metric for regression tasks, it doesn't work for classification. You can use 'accuracy' instead.


This is just the comment from Emre expanded, but yes you should look into recurrent neural networks for generating text in the style of a given corpus. RNNs and LSTM work really quite well for this.
This writeup is widely cited, and to your question, shows how it's pretty easy to generate something like this, given the text of Shakespeare's plays:

PANDARUS: Alas, I think he shall be come approached and the day When
little srain would be attain'd into being never fed, And who is but a
chain and subjects of his death, I should not sleep.
Second Senator: They are away this miseries, produced upon my soul,
Breaking and strongly should be buried, when I perish The earth and
thoughts of many states.

If you follow this you can easily run this on your local GPU too to generate text from whatever input poetry you like. I have had pretty good results with 0.1-0.5 dropout, 2 layers, layers of size 512-1024.


Recommendations for statistical models given my dataset

My guess is that either your Failure/Normal class is a lot lesser than the other. As such, for a certain (i.e. nth) fold, there only exists instances of one class. You can try doing oversampling the under-represented class to prevent this, or try doing a stratified K-Fold so that each fold will have occurrences of both classes.


Tensorflow - How to set gradient of an external process (py_func)?

Here https://www.tensorflow.org/versions/r0.9/api_docs/python/framework.html (search gradient_override_map) is an example on gradient_override_map:

@tf.RegisterGradient("CustomSquare")
def _custom_square_grad(op, grad):
  # ...

with tf.Graph().as_default() as g:
  c = tf.constant(5.0)
  s_1 = tf.square(c)  # Uses the default gradient for tf.square.
  with g.gradient_override_map({"Square": "CustomSquare"}):
    s_2 = tf.square(s_2)  # Uses _custom_square_grad to compute the
                          # gradient of s_2.

So, a possible solution could be:

@tf.RegisterGradient("ExternalGradient")
def _custom_external_grad(unused_op, grad):
    # I don't know yet how to compute a gradient
    # From Tensorflow documentation:
    return grad, tf.neg(grad)

def external_process (myvar):
    subprocess.call("process.sh", myvar)
    with open('result.json', 'r') as f:
        result = json.load(data, f)
    return np.array(result["result"])

myvar = tf.Variable(1.0, dtype = 'float32', trainable = True)

g = tf.get_default_graph()
with g.gradient_override_map({"PyFunc": "ExternalGradient"}):
    external_data =  tf.py_func(external_process, [myvar], [tf.float32])[0]

loss =  tf.reduce_sum(external_data)

optimizer = tf.train.AdamOptimizer(0.05)
train_step = optimizer.minimize(loss)
sess.run(train_step)


Model selection metric for validation phase in deep learning

It doesn't have to be the same function and usually is not.

The point of the validation set is to measure how well our model is actually doing. This is only useful if we measure it on a metric that has some value to us.
For instance, in classification, no one is concerned about which model achieves the least cross-entropy, rather which one achieves the highest accuracy, which makes more sense from a business standpoint.

One clarification I'd like to make on your initial statement that doesn't have to do with the question itself.

  I have been taught that for each epoch in training, we perform a training phase, and then a validation phase where we decide whether the new set of parameters is better than the current best. This selection of parameters is done using the same loss function that we use for training, but applied to validation data instead of training data.

This is incorrect! An epoch is technically the point at which the model has seen the training dataset once. After the end of the epoch, we usually do a pass on the validation set to measure how well our model is actually doing. However, there is no hyperparameter tuning performed at this step, as the model hasn't been fully trained yet. A model typically takes multiple epochs to train, after which we can change the hyperparameters and start again. The only change we might make to the hyperparameters at the end of an epoch is something like a scheduled learning rate reduction.


How to retrain a K-Modes model based on daily data?

There's dedicated pypi package for incremental/online learning. It's called Creme and here's there repo. It contains KMeans implementation. Though under the hood, there's no incremental stuff going on, as you need all data in a pass (read more about Lloyd's algoritm here).


Take a look at https://goo.gl/yCZvSb, a view of the Gapminder health data sets. Of the 519, you may find one suitable.

For purposes of your thesis, I'd encourage you to think of big data in terms of high-dimensional data with n >> the typical clinical trial. I'd be very surprised if you found health-related observational data other than vital statistics, such as number of births/deaths that involved even 10^9 scale data.


In sklearn Pipeline, why are all parameters fit_transform(), but the last one can be just fit()?

To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation.

$$z' = \frac{x-\mu}{\sigma}$$

or

$$z = ln(x)$$

You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before forecast. But you have to use the same two parameters $\mu$ and $\sigma$ (values) that you used for centering the training set.

Hence, every sklearn's transform's fit() just calculates the parameters (e.g. $\mu$ and $\sigma$ in case of StandardScaler) and saves them as an internal objects state. Afterwards, you can call its transform() method to apply the transformation to a particular set of examples.

fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set $x$, but it also returns a transformed $x'$. Internally, it just calls first fit() and then transform() on the same data.


Can anyone explain me the difference between Factor Anaysis and PCA?

Principal components don't have to make sense, they are just artificial numbers, summarized data if you will, used e.g. for visualizations

With factor analysis you try to find a latent variable (an underlying mechanism) that explains the data. Factor analysis is often used in psychology, eg the personality trait extraversion (the factor) explains certain behaviours (eg certain behaviours reported in a questionnaire)


Splitting a sentence to meaningful parts

You are looking to create a  parse tree to find multi-token clauses.

Here is code to generate a parse tree:

import spacy
from nltk import Tree

nlp = spacy.load('en')

def to_nltk_tree(node):
    if node.n_lefts + node.n_rights > 0:
        return Tree(node.orth_, [to_nltk_tree(child) for child in node.children])
    else:
        return node.orth_

query = u'tell me about people in konoha who have wind style chakra and are above jonin level'
doc = nlp(query)
[to_nltk_tree(sent.root).pretty_print() for sent in doc.sents]

Output:

    tell
  ____|______
 |         about
 |           |
 |         people
 |     ______|_____
 |    |           have
 |    |       _____|____________
 |    |      |     |     |     are
 |    |      |     |     |      |
 |    |      |     |   chakra above
 |    |      |     |     |      |
 |    in     |     |   style  level
 |    |      |     |     |      |
 me konoha  who   and   wind  jonin

From that parse tree, you can then select the phrases.


Try the intrinsic pandas conversion:

p.to_dict()


How to change the threshold value in orange?

I don't think you can do it for classifiers or any other widgets (for example confusion matrix).

For classifiers, workaround would be exporting the data along with the precise classification values (via "Predictions" and "Save data" widgets) and using external tool to process them.


Lower training accuracy than testing accuracy (MLP/Dropout)

Yes, your understanding is correct: dropout can be the cause. Nevertheless, you can only be sure if you evaluate on the training data with dropout disabled.

This depends on the context, but normally you would report the test accuracy.


This looks like a "proportional stacked area chart." The R Graph Gallery gallery has a similar one here:

They have great sample sample to provide guidance on how to re-create it (https://www.r-graph-gallery.com/136-stacked-area-chart.html).


How to approximate constant parameters of any generalized function?

There are general techniques around.  E.g., https://en.wikipedia.org/wiki/Non-linear_least_squares, https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

For your specific example, there's a linearization (depending on the values that $y$ can take on):

$$
\begin{align*}
y &= \left[1+\left(\frac{x_1-x_2}{p_6}\right)^{p_7}\right]^{-1} \\
\frac1y-1 &= \left(\frac{x_1-x_2}{p_6}\right)^{p_7} \\
\frac1y-1 &= \left( \frac1{p_6}\right)^{p_7} (x_1-x_2)^{p_7} \\
\underbrace{\log\left(\frac1y-1\right)}_{y'} &= \underbrace{-p_7\log p_6}_{a} + \underbrace{p_7}_b\underbrace{\log(x_1-x_2)}_{x'}
\end{align*}$$
At this point, your original problem is expressed as a linear model.  Of course, the error terms are now skewed from what they originally might have been, and taking the logarithm won't have worked if $y\notin(0,1)$.


How can we apply PCA to reduce dimensionality of a heterogenous dataset?

Yes, Since machines cannot quantify qualitative (or) sometimes called categorical data. We manually have to quantify data when processed along with quantitative data. One way of converting these categorical data is One hot encoding, By performing one hot encoding, you convert categorical data into numerical data, something that a machine can understand
There are some cases where one hot encoding fails. When the number of categories are high, you will get highly sparse features which makes no sense Ex: Countries as categorical labels. In these types of situations you can calculate distance of all countries from a certain point on the earth Ex: Calculating the distance of each country from equator. I'm just a beginner in DS.


Graduate Degree Choices for Data Science

Why not do an MSc in ooh... Data Science?

I wrote a quick review of UK Data Science Masters' offerings recently. That should help you get an idea what is offered. Mostly they are mashups of stats and computing, but there are specialisms (health, finance for example) that might interest you.

Note that list was compiled for courses that have already started, so some of those courses might not be available for starting next October, or have different syllabus contents.


Is feature engineering still useful when using XGBoost?

Let's define first Feature Engineering:

Feature selection
Feature extraction
Adding features through domain expertise

XGBoost does (1) for you.
XGBoost does not do (2)/(3) for you.

So you still have to do feature engineering yourself. Only a deep learning model could replace feature extraction for you.


Minimize absolute values of errors instead of squares

Both the square and the absolute value should work okay for gradient descent but the square will work better.  For calculus based methods, the absolute value method may be intractable.

If you are using gradient descent methods then the square works very nicely in that it tends to form a U shape which takes big steps away from the minimum and smaller steps close to the minimum. Conversely, absolute values tend to form V shapes which take roughly the same step size. The result is that squares have better convergence properties.

Now, calculus based methods tend to take the derivative of a function and set it equal to zero.  Notice that U shapes tend to have well defined derivatives equal to zero at the minima, which works perfectly. By contrast, the V shape has an undefined derivative at its minima which tend to make conjugate gradient methods un-invertible.

Hope this helps!


How to utilize dictionary data set for text classification?

You can try this approach LEARNING TO COMPUTE WORD EMBEDDINGS ON THE FLY. Using additional auxiliary data you can substitute jargons using definitions from dictionary. In this case, jargons  embeddings will become more relevant, in semantic sense .


I guess it has been a mistake. Take a look at here.

  The other author's were Ilya Sutskever and Geoffrey Hinton. So, AlexNet input starts with 227 by 227 by 3 images. And if you read the paper, the paper refers to 224 by 224 by 3 images. But if you look at the numbers, I think that the numbers make sense only of actually 227 by 227.


(Scikit-learn) differences between LinearSVC, 'linear' kernel SVC and poly kernel SVC with degree 1

You can see in the user guide that the first two will not produce the same results.  In the multiclass setting, LinearSVC uses one-vs-rest while SVC uses one-vs-one.  Aside from that, the solver used is also different, and because of that some of the options differ.  See LinearSVC docs, SVC docs.

I suspect the third will be the same as the second.  I don't see anything in the source to indicate it quickly, but it shouldn't take long to test.


Extracting useful features for k-means clustering

There are many ways for features selection:

One possible way (Wrapper methods), is to start by clustering with all related features and save the clusters, then cluster again after removing 1 feature and compare the obtained clusters (without the feature) to the clusters (with the feature). If the difference between the results are negligible then you can eliminate that feature.

Filter methods include what you mentioned in your question, where you remove the feature that changes rarely or doesn't change at all which can be expressed mathematically by "entropy", "information gain", "Chi square test"...etc. these concepts measure the impurity of a feature.

To sum up, "Wrapper" methods usually provide better results but they are computationally more expensive.

For more please see this


What algorithms are good to predict next numbers?

Hi this can be approached as a timeseries regression problem. Deep learning (tools like Tensorflow or Keras a higher level API) can be very effective in this. It depends on many factors like the amount and quality of your data, the architecture, the complexity of the problem and hyper parameters whether it will be useful to you. machinelearningmastery.com has many examples of LSTM have a look at it! Maybe start with this. As you can see in the examples in 2 having the numbers you mentioned and the sequence in which they occured could be enough to predict the next number(s).  Usually adding more information to your neural network should make it improve.


How can i build a machine learning model to identify a specific word?

It is always best to approach a problem using the simplest possible tools. In your case there is no need to use a machine learning (AI) model to detect the word sale.

To find the word sale within a long datastream, I would do the following: parse through the entire String while looking at segments of size 4 at a time, convert the substring to lowercase. Determine if this substring is "sale". If it is save the index of the "s" into a list. Return the list which will include the index of each start of the word sale within your text.

public ArrayList<Integer> getIndexSale(String inputString){
   ArrayList<Integer> indexes = new ArrayList<>();
   String[] queue = new String[4];

   queue[0] = 0;
   queue[1] = 0;
   queue[2] = 0;
   queue[3] = 0;

   for(int i = 0; i<inputString.length(); i++){
      queue[0] = queue[1];
      queue[1] = queue[2];
      queue[2] = queue[3];
      queue[3] = inputString.charAt(i);

      String temp = java.util.Arrays.toString(queue).toLowerCase();

      if(temp.equals("sale"))  indexes.add(i);
   }

   return indexes;
}

As mentioned by Dan Carter: if you do not care about the indices of the word "sale" within your string then it is simpler to use:

public boolean getIndexSale(String inputString){

   if(inputString.toLowerCase().contains("sale")) return true;
   return false;
}


Ordinal Classification As Cost Function - In Keras or Tensorflow

Whilst this question is now quite old, I was just looking for the same and came across this implementation for in Keras:

https://github.com/JHart96/keras_ordinal_categorical_crossentropy


Determining threshold in an area with very few samples of positive label

You need a cost sensitive learning mechanism for these type of tasks. You basically create a cost matrix that determines the cost of getting predictions wrong. For example while predicting whether a patient has cancer or not, the cost of predicting he doesn't have a cancer while he does is much more than predicting than he has a cancer and actually doesn't.
Here's a good article : https://mlr.mlr-org.com/articles/tutorial/cost_sensitive_classif.html

Here's one for python :
https://towardsdatascience.com/fraud-detection-with-cost-sensitive-machine-learning-24b8760d35d9


Clustering of documents using the topics derived from Latent Dirichlet Allocation

Assuming that LDA produced a list of topics and put a score against each topic for each document, you could represent the document and it's scores as a vector:

Document | Prime | Minister | Controversial | TopicN | ...
   1       0.041    0.042      0.041          ...
   2       0.052    0.011      0.042          ...

To get the scores for each document, you can run the document. as a bag of words, through a trained LDA model. From the gensim documentation:

>>> lda = LdaModel(corpus, num_topics=100)  # train model
>>> print(lda[doc_bow]) # get topic probability distribution for a document

Then, you could run the k-means on this matrix and it should group documents that are similar together.
K-means by default is a hard clustering algorithm implying that it classifies each document into one cluster. You could use soft clustering mechanisms that will give you a probability score that a document fits within a cluster - this is called fuzzy k-means.
 https://gist.github.com/mblondel/1451300 is a Python gist showing how you can do it with scikit learn.

ps: I cant post more than 2 links


how come accuracy_score recognizes the positive label and precision_score does not?

Accuracy is symmetric in the naming of positive/negative classes, but precision is not: for accuracy, it doesn't matter which class is "positive."  So accuracy_score doesn't have a parameter pos_label, and will error if you try to pass that parameter; meanwhile precision_score has default pos_label=1, so if your labels don't include 1 and you leave the parameter to the default, you'll get an error.


Convert exponential to normal distribution

You can use sklearn.preprocessing.QuantileTransformer (or sklearn.preprocessing.PowerTransformer) which does exactly what you want:

from sklearn.preprocessing import QuantileTransformer
import numpy as np

ey = np.random.exponential(size=100)
qt = QuantileTransformer(output_distribution='normal')
no = qt.fit_transform(ey.reshape(-1, 1))

You can plot histograms to compare "before" vs "after":

# Plot histograms to see before vs after.
import matplotlib.pyplot as plt
%matplotlib inline
plt.subplot(2, 2, 1)
plt.hist(ey, bins='auto')
plt.subplot(2, 2, 2)
plt.hist(no, bins='auto')
plt.show()

The advantage of this approach is that it will also work for other input distributions, not only exponential.


IPython notebook Keyboard shortcuts:

Shift-Enter: run cell
Ctrl-Enter: run cell in-place
Alt-Enter: run cell, insert below
Ctrl-m: This is the prefix for all other shortcuts, which consist of Ctrl-m followed by a single letter or character. For example, if you type Ctrl-m h (that is, the sole letter h after Ctrl-m), IPython will show you all the available keyboard shortcuts.


How to plan a model analysis that avoids overfitting?

I guess that if I re-run the program until metrics are good, my
  algorithm will be overfitted

Re-running an algorithm does not contribute to over-fitting. Re-run and select the best model no problem. However, when we try to compare two algorithms we must average over large enough number of re-runs. However, in practice, we want the best model not the best algorithm.

  I also guess it would be because of the resample of test/training
  samples, but please correct me if I'm wrong.

Re-sampling the training set only contributes to what is known as model variance, which denotes the fact that different training samples yield different models.

  As I re-ran my program twice (I've only tested KNN which I now know is
  quite not appropriate), I got really different results

Your observation is natural. A general approach to decreasing the variance of KNN is to increase the parameter $K$, higher $K$ means KNN looks at more points around a query point (see these plots).

  If I'm going to try a lot of algorithms to see what I can get, should
  I fix my samples somewhere?

Random sampling is OK.

In python, sklearn.model_selection.cross_validate would build and validate K models for a given algorithm and returns K results; assuming we are feeding 80% of data to K-fold CV, it splits the 80% to 80(K-1)/K% training set and 80/K% validation set for K times.

In summary, first split the data 20%-80%, do cross validation on 80% of data for each algorithm (algorithms would be 1-NN, 2-NN, SVM, etc.), then select a model with the best [validation] accuracy (set return_estimator = True to get K models per algorithm from K-fold CV, so for 3 algorithms we are selecting among 3K models), and finally test the best model on the held-out 20% to get the test accuracy; cross-validation has no meaning here since the best model is already built. The final result is the test accuracy.

Also, take a look at this answer on train, validation, test sets.

  Is it even OK to do try several algorithms? (it would not always be in statistics)

Yes. Always try various algorithms.

A side note on parameter finding

We can apply the above procedure on (1-NN, 2-NN, 3-NN), and if the winner model is always selected from 3-NN algorithm, then we can limit our experiments to only (3-NN, SVM) instead of all the four. Otherwise, if the winner is not consistently from 3-NN, we can experiment with all the four.


Can BERT/ELMo be used (or retrained) to generate a text in both directions?

Neither BERT nor ELMo can be used as-is for next word (or previous word) predictions. BERT is trained on a masked language model (LM) task and can therefore only be used to guess masked words, obtaining this way a contextual representation of them. There have been some attempts to use BERT for text generation, but they have been unsuccessful up to now. ELMo is a bidirectional LM, so you need the whole sentence to use it as input.

Both BERT an ELMo use the whole sentence (except masked tokens for BERT) by construction, so they can hardly be trained on "causal" LM tasks without modifying the models:

BERT would need to be added a self-attention mask to force causality. This would be equivalent to using BERT's pre-trained weights as initial values for a normal Transformer (causal) language model or any text-generation model, like a neural machine translation (NMT) one; actually, this has been attempted for NMT here and here.
ELMO should be stripped from one of the directions' LSTM. This would be equivalent to using one of the direction's parameters to initialize an LSTM for text generation.

Another option would be to use them for knowledge distillation. This has been done with BERT.


TensorFlow - Resume training in middle of an epoch?

The network doesn't store its training progress with respect to training data - this is not part of its state, because at any point you could decide to change what data set to feed it. You could maybe modify it so that it knew about the training data and progress, stored in some tensor somewhere, but that would be unusual. So, in order to do this, you will need to save and make use of additional data outside of the TensorFlow framework.

Probably the simplest thing to do is add the epoch number to the filename. You are already adding the current step within the epoch, so just add in the epoch multiplied:

saver.save(sess, 'my-model', global_step=epoch*1000000+step)

When you load the file, you can parse the filename to discover what epoch and step you were on and use those as the start point for the xrange functions. To make this easier to re-start from any given checkpoint, you could use argparse to allow your script to take the name of the checkpoint file you want to use.

In brief, it might look like this:

# Near top of script
import argparse
import re

# Before main logic
parser = argparse.ArgumentParser()
parser.add_argument('checkpoint')
args = parser.parse_args()

start_epoch = 0
start_step = 0
if args.checkpoint:
    saver.restore(sess, tf.train.latest_checkpoint(args.checkpoint))
    found_num = re.search(r'\d+', args.checkpoint)
    if found_num:
        checkpoint_id = int(found_num.group(0))
        start_epoch = checkpoint_id // 1000000
        start_step = checkpoint_id % 1000000

# Change to xrange:
for epoch in xrange(start_epoch, 25):
    for step in xrange(start_step, 1000000):
        sess.run(..training_op..) # etc

    # At end of epoch loop, you need to re-set steps:
    start_step = 0

You may want to reduce the number of checkpoints you are creating - as it stands you would have 25,000 checkpoint files generated by your code.

Another option would be use a single checkpoint file, and  to save and restore a Python pickle of a simple dict containing the state at the time you made the checkpoint, with a similar name.


Hot Encode vs Binary Encoding for Binary attribute when clustering

Not much of difference in your case. The difference is in just 1 dimension which does not affect much. The only point I can add is that if the number of BUY and SELL values are not the same, you can replace them with their frequencies i.e. if 40% BUY and 60% SELL, then replace BUY with 0.4 and SELL with 0.6


Python Sklearn TfidfVectorizer Feature not matching; delete?

You should not experience this problem if you use TfidfVectorizer properly.

Demo:

In [58]: from sklearn.feature_extraction.text import TfidfVectorizer

source text

In [59]: text = """I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.
    ...:
    ...: The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error"""

let's tokenize it to a list of sentenses:

In [60]: from nltk import  sent_tokenize

In [61]: vect = TfidfVectorizer()

In [62]: data = sent_tokenize(text)

yields:

In [63]: data
Out[63]:
['I trained a classifier using TfidfVectorizer in Sklearn.',
 'I then pickled the model for future use.',
 'The new x_test that I want to make predictions on, has more features than the x_train from the model.',
 'This is the resulting error']

now we can fit and transform our data set:

In [64]: X = vect.fit_transform(data)

result:

In [65]: X
Out[65]:
<4x31 sparse matrix of type '<class 'numpy.float64'>'
        with 34 stored elements in Compressed Sparse Row format>

In [66]: vect.get_feature_names()
Out[66]:
['classifier',
 'error',
 'features',
 'for',
 'from',
 'future',
 'has',
 'in',
 'is',
 'make',
 'model',
 'more',
 'new',
 'on',
 'pickled',
 'predictions',
 'resulting',
 'sklearn',
 'tfidfvectorizer',
 'than',
 'that',
 'the',
 'then',
 'this',
 'to',
 'trained',
 'use',
 'using',
 'want',
 'x_test',
 'x_train']

now let's feed it a data set with unknown words (features):

In [67]: new_dataset = ["let's see what happens to unknown words", "Yet another sentence."]

In [68]: X2 = vect.transform(new_dataset)

In [69]: X2
Out[69]:
<2x31 sparse matrix of type '<class 'numpy.float64'>'
        with 1 stored elements in Compressed Sparse Row format>

it worked properly - all unknown features (words) have been ignored:

In [70]: pd.SparseDataFrame(X2, columns=vect.get_feature_names(), default_fill_value=0)
Out[70]:
   classifier  error  features  for  from  future  has   in   is  make   ...     the  then  this   to  trained  use  using  want  \
0         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  1.0      0.0  0.0    0.0   0.0
1         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  0.0      0.0  0.0    0.0   0.0

   x_test  x_train
0     0.0      0.0
1     0.0      0.0

[2 rows x 31 columns]


When is a time series considered to be "complex"? (not complex-valued)

Most of the times one refers to a complex time-series to indicate that this is not, for example, a predictable periodic time-series or a sequence of repeating patterns with fixed seasonality.

Complex time-series would be those coming e.g. from real-world measurements, and can be non-stationary, very noisy, have different and dynamic behaviors and seasonality, contain outliers and missing values, and so on.


R, xgboost: eval_metric for count:poisson

This is from XGBOOST documents:
“poisson-nloglik”: negative log-likelihood for Poisson regression.

You might want to go and read more about poisson n loglikelihood.

Here is a link for you to get started:
https://onlinecourses.science.psu.edu/stat504/node/31

Hope this can be helpful


how to I replace numeric values with a string in an R dataframe?

From the documentation of is.numeric:

  The default method for is.numeric returns TRUE if its argument is of mode "numeric" (type > "double" or type "integer") and not a factor, and FALSE otherwise.

So for a vector, is.numeric returns a single TRUE, it doesn't test each element as you might expect.

is.numeric(c(5, 4, 3))
[1] TRUE

is.numeric(c(5, 4, NA))
[1] TRUE

That's why either all or none of the values are changed to NOTMISSING.

@eg-r's fix is correct.  Here's a tidyverse way to accomplish the same.

> df<-tibble(myCol=c(5, 4, NA))
> df
# A tibble: 3 x 1
  myCol
  <dbl>
1     5
2     4
3    NA
> df %>% mutate(myCol = ifelse(is.na(myCol), myCol, "NOTMISSING"))
# A tibble: 3 x 1
       myCol
       <chr>
1 NOTMISSING
2 NOTMISSING
3       <NA>


Missing Categorical Features - no imputation

Certain models are able to deal with missing values 'naturally', like certain tree based models. Most models however are just a mathematical function which is shaped after the training data. A very easy example would be:

$$f(x) = \alpha x_1 + log(2x_2)$$

What do you do if one of them is NaN? The function is undefined and no prediction can be made. By imputing the values you make a reasonable guess where this sample lies on the data manifold.


Your train/validation loss curves are a classic example of overfitting.
It looks like you have 1425 data samples to train a model with > 28 million parameters.
I would suggest trying any/all of the follow:

use a smaller model -> less parameters means less complexity and less ability to overfit
data augmentation -> more data samples means more variation in the dataset for the model to capture
early-stopping -> use something like the Keras callback, which will stop the model training once the validation loss doesn't decrease for a number of epochs

If you happen to be using image data, you might take a look at the Keras ImageDataGenerator, which can do things like flip/rotate your images.


Should a bias neuron be connected to previous neurons?

No, bias neurons are not connected to any previous neuron. This is visualized like this:

(source)


Earlystopping in multi-output deep learning

How would one implement this without risking overtraining some output nodes while preventing others from under-training?

This isn't done with early stopping. There are rather other techniques that could prevent that some main nodes are trained in your model such as Dropout. By using Dropout your model can't rely on some main nodes, because they could get shut off at any time. So with Dropout the model  should learn to balance the weights throughout the nodes.

  Should one monitor the net validation loss, or is there any way to implement early-stopping such that m out of n (m goes from 1 to n) outputs' validation losses are monitored and training stops at convergence of all/oscillations within a certain threshold.

You noted that n in your case refers to the number of output nodes. Generally, that is the number of classes in your classifier which is mostly denoted by c. So, in multi-class classification you wouldn't have c different validation losses but average them to obtain one mean validation loss for the whole model.

  How would one implement ...

There is no gold rule to how to implement early stopping.
The simplest way to do it is as follows:

Set a so called patience i.e. after how many epochs do we stop if the loss doesn't improve (usually set to 10)
After each epoch check your validation loss
Then select the model patience epochs before you stopped, because that was the best performing model.

You see that this is not the best approach and there could be better ones. In this paper, some of them are suggested.

  ... training stops all/oscillations within a certain threshold

Could be another idea, that would have to be checked. I wouldn't recommend it though, because with SGD you still have oscillations at the very beginning.


Setting up network communication between R and Python

Apache Thrift or Protocol buffers are common choices for serialization.

If you want to share data in-memory, there is the Apache Arrow format which interoperates between many data types. If you want to share dataframes on disk, there is Feather.


Note that a CNN is a feed-forward neural network. Thus, if you understand how to perform backpropagation in feed-forward neural networks, you have it for CNNs.
A convolution layer can be understood as a fully connected layer, with the constraints that several edge weights are identical and many edge weights are set to 0.
You can also build a pooling layer in this way. For example an average pooling layer is nothing but a specific convolution layer, with fixed weights.
For max-pooling, use the fact that $\max\{x,y\} = \frac{x+y+|x-y|}{2}$.


How do we implement a custom loss that backpropagates with PyTorch?

You should only use pytorch's implementation of math functions, otherwise, torch does not know how to differentiate them. Replace math.exp with torch.exp, math.log with torch.log.

Also, try to use vectorised operations instead of loops as often as you can, because this will be much faster.

Finally, as far as I can see, you are merely reimplementing a log loss in pytorch, any reason why you don't use one that has been implemented by default? (see here or here)

[EDIT]: If after having removed math operations and implemented a vectorised version of the loss, it still does not converge, here are a few pointers on how to debug it:

Check that the loss is correct by calculating the value manually and compare it with what the function outputs
Compute the gradient manually and check that it is the same as the values in loss.grad, after running loss.backward() (more info here)
Monitor the loss and the gradient after a few iterations to check that everything goes right during the training


The behaviour you are seeing is not related to not properly resetting models but the stochastic nature of most of these algorithms. By setting a random seed the same random numbers will be generated every time. See:
How to seed the random number generator for scikit-learn?
However, while this will lead to a reproducible sample, this might still not be fair. If one model randomly gets a good seed and another one a bad one you will unfairly always favor the first one. What you could do is run the models multiple times with the same hyperparameters but with different seed and look for the average performance. This way you get a more fair comparison and the reproducibility. Pick the seeds up front, then you can loop over them. Something like this:
seeds = (1, 2, 3, 4, 5)
performances = []
for seed in seeds:
    performances.append(score(Model(param1=1, param2=2, random_state=seed)))


normalize identification values properly

It is possible that the other variables you're feeding into the NN are simply bad at predicting sales.  Sell prediction is a notoriously hard problem.

Specifically the addressing of mapping a multi-state categorical variable to the NN's {0,1} input range:  Another idea is to change that one, 5-state variable into five boolean variables.  Rather than {0,0.25,0.5,0.75,1.0} on your one variable, make each of the five boolean variables represent a single day and make [1,0,0,0,0] equal Monday, [0,1,0,0,0] equal Tuesday, etc.  I've personally had more success both with training good networks and introspecting the network itself when spreading out states of classes like that.

Other hacks you can try:
 * Take out the the 'day' column all together and see if any of the other variables get used.
 * Plot the distribution of spend as a function of day.  Even if nothing else comes of this current model, it sounds like you've found one interesting insight already.
 * Consider also trying different models.


What is Quantum Convolutional Neural Network?

This answer might not make a lot of sense without a little background on quantum computing

A QCNN (https://arxiv.org/abs/1810.03787) is a type of quantum model that the authors in this paper use to model quantum data. At the core it is just a quantum circuit acting on a set of qubits in order to model quantum data that is on those qubits. The authors use it to predict a particular property of their quantum data. Here is a piece from the paper where they explain the architecture at a high level:

  The circuit's input is an unknown quantum state $\rho_{\text{in}}$. A convolution layer applies a single quasi-local unitary ($U_i$) in a translationally-invariant manner for finite depth. For pooling, a fraction
  of qubits are measured, and their outcomes determine unitary rotations ($V_j$) applied to nearby qubits. Hence, nonlinearities in QCNN arise from reducing the number of degrees of freedom. Convolution and pooling layers are performed until the system size is sufficiently small; then, a fully connected layer is applied as a unitary $F$ on the remaining qubits. Finally, the outcome of the circuit is obtained by measuring a fixed number of output qubits. As in the classical case, circuit structures (i.e. QCNN hyperparameters) such as the number of convolution and pooling layers are fixed, and the unitaries themselves are learned.

To contrast this a little bit with the classical CNN: The QCNN is best used when dealing with quantum data that you wouldn't normally be able to deal with (easily) classically. The authors made a very smart choice here in choosing to model data that could be prepared on a quantum computer, data which might be harder for a classical computer to prepare. If you have classical data (like MNIST or CIFAR) then the CNN is still your best bet over the QCNN for performance. If you have quantum data on a quantum computer the QCNN has a much stronger case.

If you've been using TensorFlow Quantum, one of the tutorials shows this exact performance comparison where a quantum computer modelling classical data has a really hard time keeping up with the scaling of the classical computer (https://www.tensorflow.org/quantum/tutorials/mnist).

We are hoping to use TensorFlow Quantum to better understand quantum data and it's not too likely any QCNNs are going to pop up in serious classical ML pipelines for NLP or Image classification at scale that are going to drastically improve performance..... yet :P


Python - Feature Selection - Should I remove bad variables?

Assuming your features are df[X] and your target is df[y], I would just do the following:

keepfeatures = X[trained_model.feature_importances_ > 0]
keepfeatures.append(y)

df = df[keepfeatures]


how does xgboost handle inf or -inf values?

Since (most) tree-based methods only care about the ordering of values in each feature, replacing your infinite values with very large values (larger than any finite value of the feature) is fine.  Of course, you'll have to do some thinking of whether df[col1]/df[col2] should actually be treated as $\pm$inf when col2 is zero, and nans can just be left in for xgboost.


I think all you need is to build proper features.
For each period and element, I would build a categorical feature. That is 80 categorical features. It looks like there aren't many possible values by feature, let's say there are 3 or 4 possible values for each feature, by doing one-hot encoding you would end up with 240-320 features.
Then you can do some kind of feature selection, like Lasso, and train your model with the selected features.


multipying negated gradients by actions for the loss in actor nn of DDPG

This answer is not mine but a reply I received on Quora here:  https://www.quora.com/Why-is-the-loss-for-DDPG-Actor-the-product-of-gradients-of-Q-values-actions/answer/James-MacGlashan?filter=&nsrc=2&snid3=4129828389

Hopefully it helps someone rather than me just deleting the question and I am sorry for my lack of math jax knowledge.

Let’s forget about DDPG, Q-functions, actors, and actions for a minute. I know, know, blasphemy, but bear with me for a moment and we’ll get back to that.

Let’s just talk about loss functions. We’re all good with loss functions, right? They’re the backbone of our standard supervised deep learning literature. That is, in supervised learning, what we do is we define a loss function that defines how bad a prediction is from what it should be.

As a grounding example, let’s revisit the L2 loss we use for a regression problem. We define this loss as:

L
2
(
p
,
l

)

1
2
(
p
−
l
)
2

Where
p
 is a prediction from our neural net and
l
 is the “label:” the value the prediction should have been. If we can tune our neural net parameters so that this loss is always zero, we win! In other words, we want to minimize this loss function.

And how do we minimize it? The standard way these days is to use some flavor of stochastic gradient descent (SGD). And to do that, what we want to do is differentiate our loss in terms of our neural-net parameters
w
. If we used vanilla SGD, we’d do:

w
t
+

1

w
t
−
α
∇
w
L
2

Okay, but there’s a catch here, our loss function was in terms of
p
, not
w
. However, we describe our neural net predictions as a function of some input and the neural-net parameters w:
p
(
x
,
w
)
 for which we know how to compute
∇
w
p
. Since we know that, we can use the chain rule of differentiation, which says:

∇
w
L

2

∇
w
p
∇
p
L
2

Where since w is a vector,
∇
w
p
 will refer to its (transposed) Jacobian matrix.

In our modern era of deep learning libraries, you typically don’t do that yourself. Instead, your library will use the autograd/backpropagation algorithm to compute
∇
w
L
2
 for you, and it will do it by decomposing it slightly differently than we did above.

Despite that fact that modern autograd libraries will do things slightly differently, it will be important to see that we can express the gradient of our loss in terms of neural net parameters in that way.

Back to Actor Critic

Okay, now that we’ve refreshed our memory about loss functions and what applying the chain rule to a loss function looks like, let’s get back to Actor Critic and DDPG.

Let’s ask our selves: What do we really want actor critic to do?

Suppose someone handed you the actual Q-values for some policy. How can you improve that policy? Simple: by making a new policy that selects actions that maximize the Q-values you were handed. This is the foundation of RL. It’s why in Q-learning you set your policy to greedily select the action with the max Q-value (and maybe add some noise for exploration reasons).

Okay, lets assume I have a Q-function: how do I find a policy that maximizes the Q-function if the actions are continuous?? I can’t do what Q-learning does by simply evaluating each action and choosing the max, because there are an infinite number of actions to evaluate!

Wait a second though. We know how to maximize a function over continuous values. We’ve been doing it for ages in supervised learning! Note that maximizing a function is the same as minimizing the negative value of that function. That is:

arg
max
x
f
(
x

)

arg
min
x
−
f
(
x
)

Okay cool, so if I want to maximize some function, I can just use regular SGD on the negative value of that function.

From that, we now have an insight: what if we think of
−
Q
(
s
,
a
)
 as a loss function, where the actions
a
 are “predictions” from some neural net: the actor. In that case, we can use good ol’ SGD to train our actor to choose actions that maximize the Q-values.

To do so of course requires that we have a representation of Q that we can differentiate. But if we simultaneously train a neural net to estimate Q, and we believe its estimates are good**, then we can easily use it as the “loss” function for our actor!

Indeed, if -Q is our loss function, with actions
a
 acting as our “predictions” from our neural-net actor model
μ
 lets substitute that back into chain rule expression for loss functions that we wrote earlier and with which we’re familiar:

∇
w
−

Q

∇
w
μ
∇
a
−
Q

And there it is!

(You may note that the DDPG paper reverses the order of the multiplication from the above. The order really just depends on what convention the paper takes the gradients and Jacobian matrices to mean. If you go back to the original deterministic policy gradients paper, you’ll see they write it out as we did above.)

So why doesn’t DDPG just say to use autograd on -Q like we do with loss functions ?

Okay, so now you’re probably asking yourself:

Why didn’t the paper just say “use autograd on -Q”???

After all, in supervised learning papers we never write out the chain rule in that way, we just assume practitioners have an autograd library and only write the forward version of the loss.

The reason is if you’re not careful, running pure autograd will burn you!

In DDPG, recall that we’re training two neural networks: Q and
μ
. If you simply feed
μ
 into Q and run autograd, it’s going to produce gradients on the parameters of Q in addition to the parameters on
μ
! But when optimizing the actor, we don’t want to change the Q function! In fact, if we did, it would be really easy to make any actor maximize Q by making Q always output huge values!

To run autograd through Q would be like letting a supervised learning algorithm change the values of labels, which clearly is not right!

You can’t simply “block gradients” on Q either, because then no gradients will flow back onto the actor.

So instead, you simply have to make sure you apply the chain rule through Q to
μ
, without adjusting any of the parameters of Q. One way to do that is to compute the gradients of each manually and multiply them, as written in the paper.

There are other ways to do it in autograd libraries too, but now you’re starting to get really specific about which library you’re using. So for an academic paper, it’s better to simply write out precisely what gradient should be computed for the actor, and then let the practitioner decide the best way to compute it in their library of choice.

** This assumption that a neural net’s estimates of the Q-function are good is really important. Because it’s an estimate, it will have errors, and a limitation of the DDPG algorithm is that your actor will exploit whatever errors exist in your neural net’s estimate of Q. Consequently, finding ways to ensure the Q-estimate is good is a very important area of work.


Is there an alternative to nltk in golang?

No. Not yet

There is no single package in Golang, which acts as versatile as nltk for NLP. However, there are several packages which aim to do it.

Here is a compiled list of such packages: https://github.com/gopherds/resources/blob/master/tooling/README.md#nlp


I think that caret and e1071 serve difference purposes. First lets discuss caret, its closest competitor is the mlr package. Both are meta packages that allow to optimize models across parameters. Take for example a problem where you are not sure whether you would want to use Lasso or Ridge to create a model. As explained here caret allows to choose the optimal lambda based on different type of cross validations.

The e1071 package instead is a bag of functions developed by the TU Wien. It is probably one of the most popular packages for support vector machines which are used by caret. But its goal is very different than caret, it implements learning algorithms and other functions whereas caret seeks to find the best parameter in learning models.


How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states.

In Equation 1 of the Attention is all you need paper, these are just parameters that come from outside:

It just says, what do you do, if get some queries, keys, and values from somewhere outside. This also the case of the unnumbered equation on top of page 5. Here, you only project the keys, queries, and values for the heads of multiple attentions.

Here $W_i^Q$, $W_i^K$, and $W_i^V$ are learnable parameters and they learned by the standard back-propagation algorithm.

Note that although keys and values (and queries in the self-attention) are equal only at the input of the $\text{Multihead}$ function. The $\text{Attention}$ function already gets the projected states.


What are x variable and y variable in word2vec model if it is supervised learning

First I would call it self-supervised in the sense that you dont need human labeling, but still need labels for NN to learn.

Both CBOW and skip-gram are "combined" into word2vec but they are shallow-neural-network architectures (albeit different ones).

Since they are different architectures, they do different things:

CBOW learn to predict the word by the context. Or maximize the probability of the target word by looking at the context.

skip-gram model is designed to predict the context so given the word xyz it must understand it and tell us that there is a huge probability that the context abc__dfg is what "surrounds" xyz.

Given that, you could try to deduce yourself what should input and output data look like to model this desired input output of CBOW, skip-gram (matrix representation of words and than probabilities and a vector representing a word and multiclass output).

Check also this answer


How to classify images Neural Network didn't trained to Understand

You need to modify the dataset by adding 1 label called 'others'. Now, there are 4 output labels: cat, dog, wolf, and others. Then, you train the model again.


Dissmissing features based on correlation with target variable

You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. That's problematic enough to start.

Low correlation means there's no linear relationship; it doesn't mean there's no information in the feature that predicts the target.

I think you're really looking for mutual information, in this case between continuous and categorical variables. (I assume your other inputs are continuous?) This is a little involved; see https://stats.stackexchange.com/questions/29489/how-do-i-study-the-correlation-between-a-continuous-variable-and-a-categorical

If you're attempting to do feature selection then you could perform a logistic regression with L1 regularization and select features based on the absolute value of their coefficients.


CTC can also work with totally random text (and thus without any "word pattern").
I know that because I trained an OCR network with CTC loss on random text which was then able to read non random text (fortunately).
The CTC just sums the probabilities of the all the paths which are "compatible" with the target text (random or not).
The higher the sum of probas of "compatible" paths, the better the prediction.
If not done yet, you should read this very good article explaining the CTC loss.

Edit to clarify and answer questions asked in comments :

do u mean specifically alphanumeric strings ?
Yes I mean random string with any character, alpha or numeric, it does not matter (even Chinese or Arabic alphabet if you want).
it needs to have seen a lot of data with patterns to generalize weights for transition from C->A->T, correct ?
No, when you train on random text, the network is not using context of previous chars to predict current time step but it is still forced to predict each time step (slice of text) correctly because of the CTC loss. The CTC loss pushes the CNN to compute good features and the RNN to give good probabilities by character for all time steps.
In other words, even with random text, the CTC loss still pushes the network to predict each char correctly independently of the other chars of the text.
And actually a system trained on random text generalizes better on unseen words than a system trained on real words because the latter is biased toward words it has seen during training.
For example a system trained on random text with Latin alphabet could read English, French and German but a system trained only on English words won't work well to read another language with same alphabet.
how can we ever generalize A623BBCEF ??
Just because a network trained on random text learnt to read each char independently of other chars.
the output of such a RNN is a matrix containing character probabilities for each time-step" ..how do we calculate character probabilities ?
The RNN takes the feature map for the current time step and compute a "score vector" from it. Sigmoid activation is used to map the scores to probabilities (between 0 and 1).
In the end it is just a fully connected layer with as many units as characters with a sigmoid activation on each unit.


Why KL Divergence instead of Cross-entropy in VAE

Answering with some theoretical understanding  of Variational auto-encoders.
In the general architecture of encoders and decoders, the encoder encodes the input a latent-space, and the decoder reconstructs the input from the encoded latent space.
However, the Variational auto-encoders (VAE), the input is encoded to a latent-distribution instead of a point in a latent space. This latent distribution is considered to be Normal Gaussian distribution (Which can be expressed in terms of mean and variance). Further, decoders samples a point in this distribution and reconstructs the input. Since, VAE encoder encodes to a distribution than a point in a latent space, and KL divergence is use to measure the difference between the distribution, it is used as a regularization term in the loss function.


Parse documents to obtain subjective sentiment

As far as I understand, the task you try to do would require a system to correctly analyze the sentence at a semantic level. In general this is Natural Language Understanding and there's no solution to this problem.
Given your examples it looks like one problem is separating the different parts of the sentence, so that you can analyze the correct part of the sentence for each entity independently. This is doable: you can use shallow or deep parsing methods to extract the chunks or subtrees, and then you could apply sentiment analysis to just to the part which contains the target entity.
In general don't expect perfect results with sentiment analysis or any semantic task. Even though DL has brought great progress in the recent years, these problems are active research questions (and might still be researched for a long time).


Here is what i did. Not sure if this is correct. Will have update next week.

I plotted these 4 points on a graph with x1 and x2 on x and y axis respectively. since its a unit step function, which essentially has 1 or 0 output - i tried to draw a linear regression line such that all "0" result points are on 1 side and "1"  result points are on other. This gave me the possible weights w1 and w2 and i constructed an equation

w1= 1.5 and w2 = 0.5

Then i made a table to check if the equation provided the actual results

i started adjusting the w1 and w2 to try to "fit" to training data but one data was just not working!!

i could bring the error down at w1=1 and w2 =0.39

Then - the threshold value, T
it should be in such a way that the z = x1w1+x2w2 + T and y=0 if z<0 and y=1 if z<=0. The value of T should change the results of (0.2,-0.5 ) to fit but not change the result of other points. So i started at -0.4 and worked my way up to -0.44.

Final equation came out to be.
y = x1 + 0.39x2 -0.44

This fits the current training data.


Is there any alternative to L-BFGS-B algorithm for hyperparameter optimization in Scikit learn?

Scikit-learn package has a limited selection of optimizers. The scipy package has many more optimitizers, including trust-region-reflective algorithm. You would have to use another third party package for Firefly algorithm.


Getting into data science and NLP outside of university

I'm also nearly 40.

2 years ago I decided to change career paths from Engineering to "Data Science".

Despite having many years of Python experience I recognised that I didn't know or understand the latest technology or even the modern terms for things I did everyday.

Therefore I suggest the following:

Put your existing skills on a more formal basis using MOOC like DataCamp or Data Quest.  This will quickly teach you the relevant terms (so you know what to Google) and relatively recent practices.
Read.  Many great books available at low cost or even free, whether it's Python for Data Science or Intro to Sci-kit learn.  Try out the tutorials.  Read articles on Hacker News on the topic.
Practice.  Kaggle is interesting but I found following the tutorials from fast.ai better for real life use.  It's better to apply these skills to something you find interesting. E.g. build a chat bot that turns in the sprinklers in your garden.

After all of this, you'll realise there are opportunities in every business to practice your skills.  Some will use the term "Data Scientist" but I suspect most won't.

You didn't say what you did before, but I suspect if like me you subscribe to the following definition of Data Science then you'd already be there.

Data Science is the combination of Advance Computing Skills, Statistics and/or Maths and "Domain Knowledge".

What is domain knowledge?  Any subject that you know a lot about.  You could be radiologist in a hospital or fraud investigator in an insurance company.


Feature selection with L1 regularization on sklearn's LogisticRegression

Yes, as demonstrated in the documentation: http://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_path.html

Welcome to DataScience.SE


Multicollinearity(Variance Inflation Factor). Variables to remove before doing a model

Here is a code I have written to handle Multicollinearity in a dataset. This code snippet is able to handle the following listed items:

Multicollinearity using Variable Inflation Factor (VIF), set to a default threshold of 5.0
You just need to pass the dataframe, containing just those columns on which you want to test multicollinearity.
This function will drop those columns which contains just 1 value. For a bit more further details on this point, please have a look my answer on How to run a multicollinearity test on a pandas dataframe?.
The calculation of VIF is parallelized over multiple cores.

    from joblib import Parallel, delayed
    from statsmodels.stats.outliers_influence import variance_inflation_factor

    def removeMultiColl(data, vif_threshold = 5.0):
        for i in data.columns:
            if data[i].nunique() == 1:
                print(f"Dropping {i} due to just 1 unique value")
                data.drop(columns = i, inplace = True)
        drop = True
        col_list = list(data.columns)
        while drop == True:
            drop = False
            vif_list = Parallel(n_jobs = -1, verbose = 5)(delayed(variance_inflation_factor)(data[col_list].values, i) for i in range(data[col_list].shape[1]))
            max_index = vif_list.index(max(vif_list))
            if vif_list[max_index] > vif_threshold:
                print(f"Dropping column : {col_list[max_index]} at index - {max_index}")
                del col_list[max_index]
                drop = True
        print("Remaining columns :\n", list(data[col_list].columns))
        return data[col_list]

Good luck !


Doubt in interpretation of Bernoulli Naive Baeyes Algorithm

Bernoulli Naive bayes does not assume gaussian distribution of all continuous features, because it does not make sense. Gaussian Naive Bayes assumes gaussian distribution for continuous features and it is the appropriate way for using Naive Bayes approach if you have continuous features.

On the other hand, if you have binary categorical data then the appropriate approach is Bernoulli Naive Bayes. If your features are categorical but not binary then you could transform them into binary categorical using dummy boolean variables for each available value of the categorical features. The main point of Naive Bayes algorithm is the assumption of feature independence, which in some real world classification problems does not hold.

You need to specify a conditional probability p(x|y) of the feature value x given the class label y. Since Naive Bayes assumes that all features are conditionally independent given the class, you can mix different likelihood models for each feature considering any prior knowledge about it.

For example, considering a continuous feature you might assume that p(x|y) is normally distributed, then you can stimate the mean and variance for this feature under each class in the training set and after that you can use the PDF of the Normal Distribution to estimate p(x|y).
Considering another feature which is categorical, you can estimate p(x|y) using a Bernoulli or multinomial event model and multiply the two conditional probabilities together in the final prediction (since they are assumed to be independent anyway).


Why does categorical_crossentropy work in when input is not 1-hot encoded?

The algorithm converts your categorical labels into one hot encode before calculating loss. So you don't have to carry the burden.


There is bunch of lemmatization solutions for polish language.
One of the best implementation is in polish morphosyntactic analyser, which you can download here.

It has bindings to python, but you have to install them manually. It is "morphosyntactic analyser" which means, that you get all possible lemmas for a given word. If you want also disambiguation, you can use this tool which is provided as docker container.

My team is currently working on implementing polish language support for Spacy, and we will cover lemmatization, so tool with good integration and support will be available soon.


When to use missing data imputation in the data analysis problem?

Generally speaking you have two options:

impute the missing data
discard the missing data

Due to the fact that ML models perform better with more data, the former is usually preferred. However, you should keep in mind that the imputed data should not affect the distribution of the feature.

This is especially the case with features with a high percentage of missing values. If a feature has, let's say, 90% of its values missing, then by imputing it you are dictating what you want its distribution to be like (because all the data in that feature will be artificial). In this case it is better to discard the feature altogether.


Neural network does not converge with negative symbols

I tried this experiment and was able to get some positive results. I will describe what I tried then perhaps you can specify where the differences may lie and we can further explore them. From what I tried I would assume that you are simply not training long enough.

Creating the data

import numpy as np

n = 100000
x_train = np.zeros((n,2))
y_train = np.zeros((n,))
for i in range(n):
    x_train[i,0] = np.random.choice([-1,1])
    x_train[i,1] = np.random.choice([-1,1])
    if x_train[i,0] == 1 and x_train[i,1] == 1 or x_train[i,0] == -1 and x_train[i,1] == -1:
        y_train[i] = -1
    else:
        y_train[i] = 1

x_train = x_train.reshape(n, 2,)

n = 1000
x_test = np.zeros((n,2))
y_test = np.zeros((n,))
for i in range(n):
    x_test[i,0] = np.random.choice([-1,1])
    x_test[i,1] = np.random.choice([-1,1])
    if x_test[i,0] == 1 and x_test[i,1] == 1 or x_test[i,0] == -1 and x_test[i,1] == -1:
        y_test[i] = -1
    else:
        y_test[i] = 1

x_test = x_test.reshape(n, 2,)

print(x_test[0].T)
print(y_test[0])

  [ 1.  1.]
  -1.0

Build the model

As you describe the model is 2 input nodes, 2 hidden nodes and 1 output node. Every node is using tanh as its activation function.

input_shape = (2,)

model = Sequential()
model.add(Dense(2, activation='tanh',
                 input_shape=input_shape))
model.add(Dense(2, activation='tanh'))
model.add(Dense(1, activation='tanh'))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Train the model

Because I generated many instances of the data I am only training for 10 epochs. However, if your input space is only the four possible inputs you may want thousands of epochs. Neural networks do take a long time to converge.

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

  Epoch 10/10 100000/100000 [==============================] - 1s
  9us/step - loss: 9.3983e-05 - acc: 1.0000 - val_loss: 7.9096e-05 -
  val_acc: 1.0000


how Lasso regression helps to shrinks the coefficient to zero and why ridge regression dose not shrink the coefficient to zero?

These diagrams show the "constrained" version of lasso/ridge, in which you minimize the pure loss function subject to a constraint $\|\beta\|_1\leq t$ or $\|\beta\|_2\leq t$.  (Another common version adds a penalty to the loss, and these are equivalent.)
The bluish solid shapes are the set of points with $\|\beta\|\leq t$, on the left with L1 norm and on the right with L2 norm.  $\hat{\beta}$ represents the unpenalized optimum value of $\beta$, and the ovals around it are the level curves of the loss function.
We know that the optimum constrained solution will occur at a point of tangency of the level curves of the loss function and the constraint boundary, specifically with the "smallest" level curve that touches the constraint region.  Then the point of the picture is that, with the pointy L1 constraint, you are much more likely for that point of tangency to occur at one of the corners, which correspond to one (or more, if we try to generalize to higher dimensions) of the coordinates in $\beta$ being zero.  Compare to the L2 boundary, where the tangency being exactly along one of the axes is much less likely.