How to use GridSearch for LinearSVC / Random Forest with time series data

So it´s a classification problem with a grid-search, without cross-validation. Yes, don´t use cv in time series data. There is an option, in which you can use cv, when you slowly start with less data and put more and more data during the process. But it´s complex.
For the grid-search are 2 opportunities. Either use GridSearchCV and define cv as none, or you use ParameterGrid().
For my interest I used this method:
https://stackoverflow.com/questions/44636370/scikit-learn-gridsearchcv-without-cross-validation-unsupervised-learning/44682305#44682305
in which is GridSearchCV defined as none.
import pandas as pd
test = pd.DataFrame({"id":[1,2,3,4,5,6,7,8,9], "age":[20,30,32,40,55,32,20,41,38], "gender":[0,1,0,1,0,0,1,1,0],
            "m1":[12.4, 30,9.4,14,19,20,34,31,16], 'm2':[34,36,22,16,22,27,42,65,13], 'label':[0,0,1,1,0,1,1,1,0]})
test.head()

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score

X = test.drop('label', axis=1)
y = test.label

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)

rf_params = {'n_estimators': [100, 200],
             'max_features': ['auto', 'sqrt'],
             'max_depth': [10, 50],
             'min_samples_split': [2, 20]}

cv=[(slice(None), slice(None))]

rf_clf = GridSearchCV(RandomForestClassifier(random_state=42),rf_params, n_jobs=-1, verbose=2, cv=cv)
rf_clf.fit(X_train, y_train)

#best parameters of model
print(rf_clf.best_params_)

#make predictions
rf_pred = rf_clf.predict(X_test)
print('Accuracy', accuracy_score(rf_pred, y_test))

The GridSearchCV part shows me:
GridSearchCV(cv=[(slice(None, None, None), slice(None, None, None))],
estimator=RandomForestClassifier(random_state=42), n_jobs=-1,
param_grid={'max_depth': [10, 50],
'max_features': ['auto', 'sqrt'],
'min_samples_split': [2, 20],
'n_estimators': [100, 200]},
verbose=2)
So this method works.
Here I used random forest, because in my own experience, random forest is in most cases very good. In big datasets, the SVC takes too much time.
PS: Before I forget, I changed the gender into numbers. You can use one-hot encoding for that or catboost, which can do this automatically. But with catboost you get different results in comparison with rf or other algorithms. So I prefer to transfer gender into numbers.


Machine Learning: Predicting target based on a feature

One quick thought is to use 1-NN (k-Nearest-Neighbors with k=1).
This does not need any training (it is a lazy algorithm) and for each new sample you look up what is the nearest datapoint in your dataset.
In your case, when you try to classify a new sample you can set its price to 0.
If you use Euclidian distance as a distance metric for the 1-NN, then in case there are multiple points with the same feature values, the nearest one will be the one with the smaller price (assuming that prices are in the range [0, inf) to avoid complications with negative prices).
In summary, what you need to do to classify a sample:

Set the sample's price to 0.
For each data point in your dataset:
3. Calculate the Euclidian distance between the sample and the data point.
Use the type of the data point with the minimum distance from the sample as the classification value for the sample.


Explain Binary Classification with output 0.5 (True)

It sounds like your model is saying there is a 50% chance that this data point could be in either class. Since this is a special case, prediction algorithms need to pick which class to choose when there is an equal probability for each class. In this case, it looks like your model defaults to the True class.

There are some models in sklearn like random forest and SVM that have predict_proba functions. You might find more information if you look there or check if the model you are using has documentation on how its probabilities are calculated.


It depends.  If you're evaluating your model's performance without tuning hyperparameters, then a train/test split is sufficient.

If you're tuning hyperparameters, then you need a validation set.  Within each fold, you'll train on the training set (of course), using the validation set to tune hyperparameters.  Then you'll evaluate performance on the test set.


Are stopwords helpful when using tf-idf features for document classification?

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the corpus. If you do add the stopwords, the Idf should get rid of it.

However, working without the stopwords in your documents will make the number of features smaller, which might have a slight computational advantage.


How do I use Machine Learning to analyze process statistics of automated test results?

I want to come up with a script/tool(Python) that leverages machine learning and automatically analyze the results for me, instead of me looking at the results manually every day.

This is the wrong approach. Choose your tools to solve tasks, not because they sound cool. Imagine a craftsman saying "I want to use a hammer". (However, machine learning might still suit to this task.)

  From what I read online, I think I need to [...] train it for a few days.

Depends on how much data, how much features, how much computing power and which algorithm you use. In many cases you can train a model within seconds on a normal computer.

The real question

The problem: You have some characteristics (features) like CPU usage, memory usage, ... for every automated test. For some instances (meaning a test and the associated features) you have the binary information "normal", "abnormal". Now, when you get the characteristics of a new test, you want to automatically decide if it was normal or abnormal.

This is a binary classification problem. If you have the label ("normal", "abnormal") for all instances, you can apply supervised learning methods. If you have the label only for some instances, it is semi-supervised learning. If you don't have any labels it is unsupervised.

The Answer

I assume you are in a supervised learning case. Then, I would suggest to use a decision tree (see sklearn.tree.DecisionTreeClassifier - sklearn is pretty easy to use and has many standard algorithms implemented) for this decision making. It basically automatically generates many nested if/else statements to get to the decision. Hence it is interpretable and you could manually adjust or improve it.


I guess based on the answer here and the point referred here your problem is not from Keras. Where you can read ... we see that PIL is reading in blocks of the file and that it expects that the blocks are going to be of a certain size. It turns out that you can ask PIL to be tolerant of files that are truncated (missing some file from the block) by changing a setting.

Somewhere before your code block, simply add the following:

from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

...and you should be good.

EDIT: It looks like this helps for the version of PIL bundled with Pillow ("pip install pillow"), but may not work for default installations of PIL.


Where can I find datasets with large number of features (100000) but less samples(100-1000)?

Just take a dataset that has a large number of features and ignore as much data as you need to get the ratio you desire. E.g. take imagenet, but randomly select 100 samples per class.


Distance calculation/vector range significance

I created a scoring system ("Thomas Scoring System") to deal with this problem.  If you treat "distance" as a similarity score, this system should work for you. http://exploringpossibilityspace.blogspot.com/2014/02/thomas-scoring-system.html

  Here's Thomas Scoring System (TSS) in a nutshell: Treat each metric and their values as evidence that weigh for or against specific values in the performance index. This is an inference process and not an arithmetic calculation, as in the Usual Method. The output of TSS is an estimation of the Weight of Evidence for all index values.  The Weight of Evidence can be expressed as a probability distribution, where all the evidence weights sum to one.

To use your example, each range is a condition: A) 10 <= x <= 22; and B) x < 10. If A is true, then it weighs in favor of certain similarity (distance) scores.  If B is true, then it weights in favor of other similarity (distance) scores, but would have less weight compared to other evidence.


Preserve colour in convolutional autoencoder

Try training with more epochs. I've seen this same thing where the output after M epochs is a grayscale image but continued training results in more and more color.


How to use Graph Neural Network to predict relationships between nodes with pytorch_geometric?

Seems the easiest way to do this in pytorch geometric is to use an autoencoder model. In the examples folder there is an autoencoder.py which demonstrates its use. The gist of it is that it takes in a single graph and tries to predict the links between the nodes (see recon_loss) from an encoded latent space that it learns. The example is of one large graph, for my purposes I had multiple graphs which meant each one got their edges split and trained separately.


Predict the loss amount for insurance policies using the historical trends and features?

Loss amount - "In insurance. The diminution, destruction, or defeat of the value of, or of the charge upon, the insured subject to the assured, by the direct consequence of the operation of the risk insured against, according to its value in the policy, or in contribution, for loss, so far as its value is covered by the insurance" (not exactly my area of expertise).

If you are looking for "insurance policy datasets", you can try the following link:

insurance data on data.world

You would probably need some regression model, but you are not providing enough information for more specific answer.


Why there is two output in Titanic case in tflearn quickstart?

Since you have only one binary target to predict (survived vs not survived) you can use only one output (that provide the probability of surviving). The problem with this approach is that you can not longer use softmax activation, since it will always be 1. Remember that softmax activation has the form
$$g(x_k)=\frac{e^{x_k}}{\sum_{i=1}^n e^{x_i}}$$
Since you have only one element to run the activation it will be normalized always as 1.
It is used mostly for categorical prediction (i.e. among several possibilities only one is correct).
You are getting $0$/$NaN$ loss because your model always return 1. Instead of using softmax activation you might try sigmoid activation, so you can interpret the result of the network evaluation as the probability of survive.


How to optimize input parameters given target and scoring parameters

If your data have simple linear relationship (or you can transform it such that this is the case) you can do regression on $T$ and $S$, and after that you should have the linear relationship between input parameters and target parameters. After that you can use scipy minimize or any other library to perform constrained optimization. If I follow from my example my idea would be to perform a predefined trade-off between $T$ and $S$ for example $\max 0.8 T + 0. 2S$. This should be easy to code with scipy.


Overfitting - how to detect it and reduce it?

When choosing the validation set and the test set, it is important that it reflects the actual "production environment" of your problem. Since you have "out of time" validation sets, I assume you have some time structure in your data that you need to address when making predictions. If you are developing your model, not taking this time aspect into consideration, you are likely to get issues when this model is put into a realistic environment because the models are fit to an unrealistic scenario where time does not matter.

From the scores you show here, the train and test scores are very similar, while it drops off in the "out of time"-set. This might be an indication that you are not putting enough emphasis on the time dimension while developing your model.

Another point: There is also fully possible to overfit to your validation set, when as in your case, you have a lot of variables. Since some combination of these variables might randomly fit your train and validation set well, it might not be the case for your test set. This effect is also magnified by my earlier point, where the data generating process is not really stationary across time.

Here is a nice article about how to choose a good validation set:
https://www.fast.ai/2017/11/13/validation-sets/


how to find classification accuracy in autoencoders?

Are you using the autoencoder for classification or reconstruction?
if you are pre-training the autoencoder for classification then you use the usual logloss to determine the accuracy of your classifier. If on the otherhand, if you are using the autoencoder for reconstruction, then there is no classification, and you can use something like KL divergence (described in detail here) to measure and track your reconstruction and compression performance.


Pandas dataframe groupby and then sum multi-columns sperately

IIUC:

In [5]: (df.groupby("Groups", as_index=False)
           .agg({"Date":"first", "data1":"sum", "data2":"sum"}))
Out[5]:
  Groups      Date  data1  data2
0    one  2017-1-1      6     33
1    two  2017-1-2      9     27


Help with understanding cross-validation

What is the actual goal here? I get that we take an average of the k rounds of cross-validation that we do, but what is our output? I've read that we discard the result of each round after calculating the validation error, so how do we actually produce a model from this?

The goal is to obtain an accurate estimate of the model's performance. Cross-validation produces unbiased model predictions for every example in the dataset. We use these predictions to compute metrics, like accuracy or log loss, for the training dataset. When we are actually building a model, we will use the entire dataset rather than cross-validation.

  What is the difference between a validation and test set, if any? i.e., we normally separate our data into training and test data, but do we then further divide our training data into validation sets, and keep our test data aside during the cross-validation process, or do we run cross-validation on the entire data set?

A validation set is a set of labeled examples used during the training phase, but not actually for training the model. This can be done in many ways. For example, when building a model through an iterative process (like gradient descent for neural networks), we can use a validation set to decide when to stop iterating. The performance on the training set almost always improves after each training step, but will eventually overfit and produce poor predictions on unseen data. In this scenario, we can test the model on the validation set to detect when overfitting occurs.

A test set, on the other hand, is only used to evaluate the model after training is complete. Optimising our training for performance on the validation set means we are slightly biased towards it, and we should finally test on a truly held-out set of labeled data to evaluate the performance of the model.

Cross-validation is intended to "replace" the need for a test set for providing an unbiased estimate of model performance, so you usually would not bother with having a test set when doing cross-validation. However, if you have a large dataset it is generally recommended to split into train/test rather than to use cross-validation.


It depends. Let me explain you what features are detected when you train a classification model.

Let's take same example as yours - car and truck classification. The CNN will extract features from both type of images - tires, windows, shape, texture etc.

Then it'll separate the features - if it sees something like bigger tires, bigger windows, rectangular shape, it'll give more weight to truck class. Similarly for car class.

Now coming to your image retrieval part. This is what you're doing - passing all the image through CNN and extracting features for each image. Then comparing your test image features with other features to find most similar one.

Now think, what if you pass new image to the same CNN ? What features will be extracted.

CNN will extract the same features as it used to for car and truck - tires, windows, shape etc. Because it's trained for that only. It won't start extracting wings as feature if you pass airplane.

So if you pass airplane as test image - same features as car and truck would be extracted and you will get results based on the comparison of these features. Hence if you pass yellow small airplane, you might get yellow small car having same tire size as your airplane or yellow small truck having shape similar to your plane or in best case yellow small airplane.


Word taxonomies for Facebook likes categories

I think total number of default categories in Facebook is somewhat static. There is around 140-160 categories.

ref: https://stackoverflow.com/questions/4216648/facebook-pages-authoritative-list-of-categories/8576572#8576572 and http://www.marketinggum.com/types-of-facebook-pages-for-business/

You can cluster (manually) them in to 10-12 generic groups.

eg: Sports Event,
    Sports League,
    Amateur sports team,
    Sports Venue,
    Athlete,
    Sports/Recreation/Activities etc
all belongs to Sports category.

Similarly cluster and form generic groups.

And then you can assign a user to any of these generic group based on the like category.


Predict a tree structure out of nodes with different features

Don't use machine-learning for this - you probably don't have millions of training examples.

Instead, what you are looking for is a simple depth-first search.

Begin at the root, then define a priority list, e.g. below-left, below, below-right, in which order to proceed. Don't visit nodes twice, but when first visiting them mark them as seen, and add them to your tree instead.

Alternatively, define a priority pattern of how to define the parent node.  Construct a tree based on this single "best parent" relationship,, but make sure you don't create cycles.


What is the best way to predict multiple outcome from a single entity?

All three models fit to single GPU

Since you have already trained the models and models are separate (do not share the features), you could construct the computational graph in a way that you have only one input (your image), but that input is pushed to three different branches of the computational graph (each branch is one of your three models). At the output of such constructed graph, you will get three outputs (one from each of three branches).

This way you will run all three models at once.

If you are using TF, it will look like this:

output_1, output_2, output_3 = sess.run(output_op, feed_dict:{input_layer: input_image})

where the output_op holds a list of outputs from three models (hence, we unpack them to three variables output_1, output_2 and output_3); input_layer is the tensor operation which takes the image and pushes it to three branches as already described.

This is only possible if your GPU memory is large enough to fit all three models into the memory.

All three models do not fit to single GPU

In this case, assuming you have access to multiple GPUs, you could modify the computational graph which combines three models to use different GPUs for each branch.

Run one after another

Also, this can be always done.

If you are using TF, this link could be useful.


Setting activation function to a leaky relu in a Sequential model

Try this, and tune alpha.

model = tf.keras.models.Sequential([
    keras.layers.Flatten(input_shape=(28,28)),
    keras.layers.Dense(128,activation=tf.keras.layers.LeakyReLU(alpha=0.3)),
    keras.layers.Dense(10,activation=tf.nn.softmax)
 ])


A few simple ideas to observe whether there are any patterns:

Plotting the days with seizure/without seizure together with the days when medication is taken across time. It's quite general but it might show some patterns.
for each seizure, difference with the date of last seizure, i.e. number of days without a seizure. Plotting this value (simple plot of the number of days over time) might show whether there is any regularity and whether there is an evolution in the frequency.
for each seizure, difference with the date of last medication. Plotting the distribution of this number (i.e. how often it's 1,2,3,...) might show if there is a direct relation between the two, e.g. if the number of days is generally more than 10 then it's likely that the effect of the medication wears down within 10 days. The medication might also have a delayed effect, which would be visible in the distribution.
In case there is a long period of data available, you can calculate the average (for instance of days without seizure) over the last N days and plot the evolution of this average across time.

Probably you and your wife can think about other things which could be relevant. The general idea is to represent a potential parameter numerically and then observe how this value behaves.


If A and B are correlated and A and C are correlated. Why is it possible for B and C to be uncorrelated?

Imagine a random point on a plane with coordinates $(x, y)$, where $x, y \in [-1, 1]$.

A = both $x$ and $y$ are positive

B = $x$ is positive

C = $y$ is positive

It is clear A is correlated with both B and C, which are not themselves correlated (assuming uniform distribution).


I still don't know how deconvolution works after watching CS231 lecture, I need help

OP here:

https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d
This has helped me understand how deconvolutional layer or transposed convolution works! Note to self don't google deconvolution or upconvolution google transposed convolution instead.

Update 2: http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html

It turns out "deconvolution" is just convolution but with different arithmetics. You can take the transpose or add enough padding so that

1) You can upsample instead of downsampling.

2) Keep the previously linked relation, what I mean is you need to make sure each upsampled filter still contains a symbolic link with the smaller input.


Best way to store financial data together with news

The short answer is that (almost) any DBMS will be able to store that information for you.

I use Microsoft SQL and you can store the record with a specified or system calculated date-time, a column for the numeric value and you can store the story text in one of four ways:

Plain text as a NVARCHAR
structured text as XML (or JSON if you are on SQL2016 or above)
Filestream if the data format is binary (normally an image, audio or video)
As a reference to a file on disk

MSSQL is an RDBMS (relational database management system) Others in this classification are Oracle, MySql, Postgress.  There is another school of database classified as NOSQL or Document Databases and these include Mongo, CouchDB, DocumentDB and RavenDB.  These differ from RDBMS in the way they store data and manage the relationships (e.g. find all the articles relating to company X in the time period 1st April to 6th June)

I would suggest that if you don't know how you are going to want to process the data in the future, look at storing it in a NOSQL database as it provides much greater flexibility of record structure.  Most can be installed on-premise with a free / open source licence and most are also available on a hosted platform for a relatively small sum.  Many of the RDBMS (esp Oracle and MSSQL) are exceeding expensive for this sort of project.  That said, Azure SQL - the hosted version of SQL can be had for as little as $8 per month depending on your storage requirements.


What does the term "proportional to" mean in Bayes Equation?

It means that $P(\theta | y ) = kP(\theta) P(y | \theta)$, where $k$ is a constant that does not depend on $\theta$.

In fact, the Bayes Theorem states $P(\theta | y ) = \frac{P(\theta) P(y | \theta)}{P(y)}$. This proportion has to be added in order that $\sum_{\theta}P(\theta | y) = 1$ is satisfied (that is, $P(\theta | y)$ has to be a probability).


How to normalize a boolean feature for neural nets?

Actually, it is not clear what you mean by deactivating but if it means the output of neuron would be zero, it is not correct due to having bias term, also known as intercept. Furthermore, we usually use normalisation for features which are of different scales. Your boolean values do not have a large range. You don't need to scale them. If I want to be more precise, you may need depending on the other features' range, because they may change slightly among different input patterns and vary less than let say 1e-5 for different samples, but most of the time, booleans are not needed to be scaled.


Reproducing randomForest Proximity Matrix from R package in Python

I have written some code for this. It can be found here. In answer to your specific questions:

I have tried to optimize for speed. What I did should be a little faster than the code above.
I do not use out of bag records. In fact the original documentation does not suggest this. I created another post to see if the consequences are understood.
This is handled in my code by normalizing by the total possible leaves that could be matched.


What is the relationship between MDP and RL?

What is the relationship between Markov Decision Processes and Reinforcement Learning?

In Reinforcement Learning (RL), the problem to resolve is described as a Markov Decision Process (MDP). Theoretical results in RL rely on the MDP description being a correct match to the problem. If your problem is well described as a MDP, then RL may be a good framework to use to find solutions. That does not mean you need to fully describe the MDP (all the transition probabilities), just that you expect an MDP model could be made or discovered.

Conversely, if you cannot map your problem onto a MDP, then the theory behind RL makes no guarantees of any useful result.

One key factor that affects how well RL will work is that the states should have the Markov property - that the value of the current state is enough knowledge to fix immediate transition probabilities and immediate rewards following an action choice. Again you don't need to know in advance what those are, just that this relationship is expected to be reliable and stable. If it is not reliable, you may have a POMDP. If it is not stable, you may have a non-stationary problem. In either case, if the difference from a more strictly defined MDP is small enough, you may still get away with using RL techniques or need to adapt them slightly.

  Could we say RL and DP are two types of MDP?

I'm assuming by "DP" you mean Dynamic Programming, with two variants seen in Reinforcement Learning: Policy Iteration and Value Iteration.

In which case, the answer to your question is "No". I would say the following relationships are correct:

DP is one type of RL. More specifically, it is a value-based, model-based, bootstrapping and off-policy algorithm. All of those traits can vary.

Probably the "opposite" of DP is REINFORCE which is policy-gradient, model-free, does not bootstrap, and is on-policy. Both DP and REINFORCE methods are considered to be Reinforcement Learning methods.

DP requires that you fully describe the MDP, with known transition probabilities and reward distributions, which the DP algorithm uses. That's what makes it model-based.
The general relationship between RL and MDP is that RL is a framework for solving problems that can be expressed as MDPs.


Avoid hardware limitation while competing in Kaggle?

Yes - a Python script will have less overhead than a Jupyter Notebook
Pickle is the standard way to store a scikit-learn model, see model persistence documentation.
The two primary ways to scale Jupyter Notebooks is vertical (rent a bigger machine on a cloud service provider) or horizontal (spin-up a cluster).


How can I picture an unfolded RNN as a normal Feed Forward Network?

I cannot be sure whether this graphical representation is fully correct or wrong, but your misunderstanding looks like it revolves around what is an LSTM cell and LSTM neurons.
In the second figure, the nodes depicted should be LSTM cells. A cell is in essence a forward neural network consisting of neurons and so an analytic representation of it may look like what the first figure shows, although admittedly it looks very confusing.
See a recent discussion on that exact matter here What is the difference between a "cell" and a "layer" within neural networks?.


Agent always takes a same action in DQN - Reinforcement Learning

This may seem obvious, but have you tried using a Boltzmann distribution for action selection instead of argmax? This is known to encourage exploration and can be done by setting the action policy to

$$p(a|s) = \frac{\exp(\beta Q(a,s)}{\sum_{a'} \exp(\beta Q(a',s))},$$

where $\beta$ is the temperature parameter and governs the exploration-exploitation trade-off. This is also known as the softmax distribution.

Put into code, this would be something like this:

beta = 1.0
p_a_s = np.exp(beta * q_values)/np.sum(np.exp(beta * q_values))
action_key = np.random.choice(a=num_act, p=p_as)

This can lead to numerical instabilities because of the exponential, but that can be handled e.g. by first subtracting the highest q value:

q_values = q_values - np.max(q_vaues)


Max pooling has no parameters and therefore doesn't affect the backpropagation?

Pooling layers does not have parameters which affect the back propagation. Backpropagation is an algorithm to efficiently implement gradient descent in a neural network by using the chain rule and propagating the error from the final layers to the early ones. The parameters that affect backprop are often called trainable variables (in Tensorflow), and they are the weights and biases of the convolutional filters. Adding pooling layers does not change the number of them.

Pooling layers are a way of performing downsampling, and they are used for the following main reasons:

To decrease the computational load of the network: smaller outputs means less FLOPS
To save memory during training: less memory allocation in the GPUs
To increase the receptive field of the neurons in successive layers. You can check this link about this topic: https://medium.com/@santi.pdp/receptive-fields-in-convolutional-neural-networks-6368a699d838
Adding transalation-invariance to the feature maps: when pooling, small translations of a feature will result in this feature located in the same position in the following pooled layer

What is true is that pooling layers may have a great impact in the learning capabilities of a network


You could do that with a neural autoencoder using a custom loss function.
Use a hidden layer, let's call it $l_{encoded}$,  with more nodes than the features of the input data.

You have to code the custom loss:
$loss = corr(o(l_{encoded}))+MSE(o(l_{output}),\  input)$,

where $corr(o(l_{encoded}))$ is the correlation of the output of the encoding layer and $MSE(o(l_{output}), input)$ is the mean squared error of the last layers output and the input instance.

Using this loss your model will try to reduce the correlation of the hidden layer while still making sure that it is able to decode the training instance.

I highly doubt this will be of any use though.


What are alternatives to MLP, when you have rectangular, structured data?

Fancy deep learning architectures mostly work by exploiting structure in your data. Permutation invariance or equivariance, temporal structure and spatial structure come to mind. Maybe some features are based on a set of the same objects? Most of the benefits come from sharing weights and learning a shared representation. Another potential benefit is the flexibility in output, although that is less relevant in your case. If you have a lot more labeled data on another task with the same features, you could pretrain a larger model on that task and then fine tune that network on your current task. For the rest it is difficult to say without showing some examples.


How to handle a valuable feature that is missing on 99\% of the samples in the data set?

Its best to remove such a variable. Reasons are following:

Artificial imputation can add bias and result cannot be justified because 99% data for the particular variable was artificially created.
The variables/features that you choose for building the predictive model should have low correlation with the target/outcome variable/feature. Because, variable that are highly correlated with the target/outcome variable reduces the model predictive performance.
You want a model which has low bias and low variance.


What's the good index to choose number of clusters so that obtained clusters are homogeneous?

Your problem is not appropriate for machine learning. Machine learning will not give robust answer to clustering itself (parameter) or automatic number of clustering (hyperparameter). The number of examples are too few (10-13) and the number of examples to the number of groups (2-5) is also too low.


You forgot to preprocess your data.

K-means is really sensitive to scale and outliers.

Also: clustering is not classification.

It may well be that e.g. one of your classes has a dense subtype. Just because the clustering found something else than your labels (and it's not supposed to find the same, is it?) does not mean it failed. It just didn't classify the data the same way as your labels (but if that is what you wsnt, then you should have been using a classification method instead.)


Working with more than one dummy variable in neural net

I found that this works:

onehotencoder = OneHotEncoder(categorical_features = [0,1,3,4,5,6])
X = onehotencoder.fit_transform(X).toarray()


What is the difference between semantic segmentation, object detection and instance segmentation?

Object Detection : is the technology that is related to computer vision and image processing. It's aim? detect objects in an image.
Semantic Segmentation : is a technique that detects , for each pixel , the object category it belongs to , all object categories ( labels ) must be known to the model.
Instance Segmentation : same as Semantic Segmentation, but dives a bit deeper, it identifies , for each pixel, the object instance it belongs to. The main difference is that differentiates two objects with the same labels in comparison to semantic segmentation.

Here's an example of the main difference.

In the second image where Semantic Segmentation is applied, the category ( chair ) is one of the outputs, all chairs are colored the same. In the third image, the Instance Segmentation, goes a step further and separates the instances ( the chairs ) from one another apart from identifying the category ( chair ) in the first step.
Hope this clears it up for you a bit.


Which NLP library has the most mature Chinese language models?

There are no common libraries that support high quality named entity recognition for Chinese.
Other options include Information-Extraction-Chinese on GitHub or adapting a paper with code.


Differentiating roadmap of a loss function


Machine learning image training set data as whole object but finding part of object

Train with the same kinds of inputs as you wish the network to recognise. There is no easy way to add logical combinations that you think should help the network, because deep neural networks, despite the hype as the best thing to hit AI, are essentially just really dumb complex statistical analysers.

Provided you supply training images, the CNN will learn how to statistically combine components to decide if overall an image is a chair or not. For instance if the image is of a person wearing a padded jacket, perhaps that will look a little like the back of a chair. The NN might output a 0.75 confidence that a back of a chair is in the image. If you manually combine decisions from components, you will need to figure out from that whether to say there is a chair in the image. If you leave it to the network, the CNN itself will combine this data with the additional detection there is a 0.2 confidence that the person's legs are chair legs (and they are in the wrong relative place) and decide - hopefully - that there is no chair.

You don't need to explicitly label components of objects for the NN to learn that there are components and what they look like. The outputs of Deep Dream and other CNN visualisations demonstrate that internally the CNN builds up its own structures that represent parts of objects. They are not necessarily the same parts as a person would focus on, but by allowing the training process to choose, you should be able to  get a good recognition rate.

So, if you give the network a lot of chair parts to recognise separately, it will get better at labelling those chair parts accurately. But that will not really help the network recognise a chair. In fact it could be worse at the main task if the training images are too different from images it is expected to recognise. Plus, it will be your task to then decide how to re-combine the classes to detect the chair later.

There probably are architectures and image-processing pipelines that can be used to combine logical knowledge of parts and their poses into whole objects. It is something I very much expect to be an active area of research in computer vision systems. However, these approaches will not use a simple CNN and altering training data as the solution.


Bias and variance and their effect on overfitting and underfitting summarized in one illustration

Therefore, I think you a fit model, with reasonable variance and bias.


As you would have seen in the research, AUC ROC prioritizes getting the order of the predictions correct, rather than approximating the true frequencies.

Usually, like in the credit card fraud problem you link to, the impact of one or two false negative is more devastating that many false positives. If those classes are imbalanced, like they are in the fraud case, AUC ROC is a bad idea.

It appears that in the competition you are referring to, the hosts are more interested in labeling which comments are more toxic than others rather than rating how toxic they each are. This makes sense since in reality the labels are subjective.


What are some methodologies for performing feature selection for simple feed-forward neural networks?

I usually try to do the following process, I don't know whether it has name or not:

Try to find features which an expert can say what the label is without hesitating.
Plot correlation matrix for your sample to investigate whether each feature has correlation with others or not. I do this process whenever the input features are so much. Then try to reduce the correlated features or try to apply PCA. Consider the point that PCA alone does not care about the labels, so it may find principal components that your data would be so difficult to be separated but not necessarily.


How can I use my laptop to implement analysis in Python but use a different fast server for computation of analysis?

I have a similar set-up; I run both local and remote installations of Jupyter. On the server I have JupyterHub running, into which i can ssh. Locally I have Anaconda set up etc.

If you are meticulous in mirroring any custom libraries across the system (or even better package them for pip installs) this works quite well.


Running SQL-like queries over large schemaless JSON dataset in the cloud?

You want to write structured queries (have your cake) but have unstructured files (eat it too).

Is there any way you can play with the files a bit and have a common schema between them, with some fields just empty? That would make it easier.

That being said, BigQuery is probably your least bad solution as you found. Amazon's new Redshift Spectrum is kind of similar but will also require a schema to be defined.


How to perform Permutation Feature importance?

Ok lets answer this with some basic steps:

Yo have done what you said until 5-th step after that you could do:

from sklearn.feature_selection import RFECV
from eli5.sklearn import PermutationImportance
pi=PermutationImportance(logreg, cv=2).fit(X_train,y)
selector=RFECV(pi, step=10, n_jobs=-1, cv=2, scoring="f1_macro").fit(X_train,y)

now make predictions

selector.predict(X_test)

or plot the most important features

import matplotlib.pyplot as plt
plt.plot(selector.grid_scores_)


Can the same CNN architecture be used for different data sets?

One way is to redesign your CNN architecture to fit your new data input. However, if you want to use the current model you have, you should:

(1) Retraining: Re-train the model with new data set images

(2) Reshape new images: You should add a pre-processing layer to convert 28x28x1 images to 32x32x3 images. This could be done by replicating a single channel in other two channels and adding boundaries to your images. This deformation process should be the same on all new input images.


number of parameters for convolution layers

Actually it's $49C*C$, the first $C$ is the number of input channels, and the second $C$ is the number of filters.

Quote from CS231n:

  To summarize, the Conv Layer:

  Accepts a volume of size $W_1 \times H_1 \times D_1$
  Requires four hyperparameters:

  Number of filters $K$,
  their spatial extent $F$,
  the stride $S$,
  the amount of zero padding $P$.

  Produces a volume of size $W_2 \times H_2 \times D_2$ where:

  $W_2 = (W_1 - F + 2P)/S + 1$
  $H_2 = (H_1 - F + 2P)/S + 1$ (i.e. width and height are computed equally by symmetry)
  $D_2 = K$

  With parameter sharing, it introduces $F \cdot F \cdot D_1$ weights per filter, for a total of $(F \cdot F \cdot D_1) \cdot K$ weights and $K$ biases.
  In the output volume, the $d$-th depth slice (of size $W_2 \times H_2$) is the result of performing a valid convolution of the $d$-th filter over the input volume with a stride of $S$, and then offset by $d$-th bias.

  A common setting of the hyperparameters is $F = 3, S = 1, P = 1$. However, there are common conventions and rules of thumb that motivate these hyperparameters. See the ConvNet architectures section below.


Determining whether a sentence is "cliche" using NLP

IMO. Firstly, you make the hypothesis that cliche sentences are in the same part of the document, which makes sense. But how do you determine the weight of the position and what if the students use different structures (e.g., some write the conclusion and the discussion sections separate, others write them together)?
Then you could get more up to date sentence embeddings algorithms than the one you suggested to provide you with more accurate similarity scores.
Finally, how I would approach it. I would use a clustering technique such as DBScan or HDBScan (with a cosine distance). The reason I would use these is that they are distance-based clustering techniques. Hence, initially, you can start with a small aperture, distance, and view which sentences are very similar to each other, almost identical. Then you can increase your distance, open your aperture, and view which others are quite similar to them. You could of course try other clustering techniques and see how they would work since those two are usually not the best. Then I would take into consideration where each sentence position is and see if it would change the results, like that, I believe, you have more of an insight into what is happening.


Thanks to @Peter. It's called Venn Diagram and here's my needed Javascript Library. https://www.benfrederickson.com/venn-diagrams-with-d3.js/


How Does Sci-Kit Learn Train Regression Neural Networks (MLPRegressor) So Fast?

Scikit-learn use numpy and cython to speed-up code.


How much data warrants building a pipeline?

You should only complicate it as much as necessary to meet your requirements.

What kind of aggregations are you doing? Most DBs can handle basic counts pretty easily. 10 million in a month sounds like something a large MySQL instance could handle.
How quickly does everything have to update? If the users only expect the summarized data to be updated monthly, you have a lot of time. If you have to update it more often, it might make sense to have MySQL collecting it but Redshift presenting it. Redshift's analytical focus and columnar storage means you may not need pre-aggregated data until your volume gets much higher than what would require aggregation in MySQL. I was using a 4 node Redshift cluster with the smallest instance types and it could do a count across >300 million rows in 17 seconds.

Can you not build aggregate tables in MySQL? What is making you want to put Hadoop in between MySQL and Redshift?


How does QUEST compare to other decision tree algorithms?

QUEST stands for Quick, Unbiased and Efficient Statistical Tree.

It uses ANOVA F and contingency table Chi Square tests to select variables for splitting. Variables with multiple classes are merged into two super-classes to get binary splits which are determined using QDA (Quadratic Discriminant analysis). The tree can be pruned using the CART algorithm. It can be used for both classification and regression tasks.

Quest first transforms categorical (symbolic) variables into continuous variables by assigning discriminant coordinates to categories of the predictor. Then it applies quadratic discriminant analysis (QDA) to determine the split point. Notice that QDA usually produces two cut-off points—choose the one that is closer to the sample mean of the first superclass.

An advantage of the QUEST tree algorithm is that it is not biased in split-variable selection, unlike CART which is biased towards selecting split-variables which allow more splits, and those which have more missing values.

Not sure why it is not used as widely as CART or C5.0. It might be due to greater coverage of CART/C5.0 in literature than others.

References:

Quest reference manual: http://www.stat.wisc.edu/~loh/treeprogs/guide/guideman.pdf
http://www.stat.wisc.edu/~loh/quest.html


You can use proximity-based outlier detection methods. These methods fall under 2 different categories, 1) Cluster-based, and 2) Distance-based.

Cluster-based methods will define a point as an outlier if its distance to cluster centroids is large. If you have both categorical and continuous variables, then it is important to ensure that the clustering method you use is appropriate for mixed data-types. Clustering methods appropriate for both types include k-Prototypes or Squeezer.

To use a distance-based method, you could use the k-nearest neighbor algorithm, and define an outlier as a data point whose distance from its k neighbors is much larger than normal data points.


What is the optimal value of a Markov Decision process with Single actions at each state?

Hi guys thanks for your comments and sorry for the slow reply. I cannot reply to them directly because I am not allowed.

@Constantinos, to your point regarding b: b-->b, b-->a being different actions.  I think you misinterpreted what I am asking. There is only one action (what you call it is arbitrary) but two possible outcomes.

@Neil Slater, Yes I think the question is badly written. An MRP is defined as an MDP with only one possible action in each state (or no action dependant on how you look at it). Reward is a real value that is the output of a function that maps a pair of states (so say (a,b) or (b,b) etc ) to a real value. Discount is defined similarly.

Really the crux of the question is: find a linear function approximation parameter $\lambda$ such that $\lambda* \phi(s) \approx v(s)$ where $v(s)$ is the value associated with a certain state $s$. I believe that such values are defined as $v(s) = \sum_{s'}P(s,s')(R(s,s') + \gamma v(s'))$ where $R(.,.)$ is a reward transition function and $P(.,.)$ is a function mapping state pairs to probabilities.

I think the main reason I am lost with this is because the question itself is pretty ambiguous so I am trying to fill in the gaps heuristically.

Thanks Guys


Recommender/Clustering data to support a hypothesis. Is this a valid use-case for unsupervised ML?

Clustering and recommendation are similar tasks, however in recommendation you usually want to recommend several items while clustering usually assigns each sample to only one cluster.
Anyway for your problem a clustering or even a classifier might help.
If labels are assigned on the basis of a similarity metric (and you have a good guess of what this metric might be) you could use a clustering algorithm to impute the missing labels (clustering is based on similarity between samples). However if it's reasonable that samples can be similar according to different metrics (and you are uncertain about it) then a classifier might help - provided you have enough labeled data to train the classifier. Which approach works best will also depend on the number of labeled samples and number of features.
Whichever strategy you choose to use the imputed labels will be noisy (some samples will be mislabeled). This is important to take into account especially if you are going to further process those labels.


Where can I get Insurance claim data for practicing NLP(Natural Language) processing?

I am working in the same industry for a few years now and I can tell you that there are no publicly available datasets because of the nature of the documents. They are quite private and contains sensitive information, that are bound by rules and regulations.


How is it possible to process an image with a few neurons?

There are several ways to make this big number trainable:

Use CNNs
Auto-Encoders (see Reducing the Dimensionality of Data with Neural Networks)

Dimensionality reduction of the input

Scale the image down
PCA / LDA

Troll-Answer

If you really meant "only a few neurons" then you might want to have a look at Spiking neural networks. Those are incredibly computationally intensive, need a lot of hand-crafting and still get worse performance than normal neural networks for most tasks ... but you only need very little of them.


Any idea to what $\bar{X}$ refers to in association rules? Is it "not-$X$"?

Do you have a link to the actual source you are referring to? I haven't myself seen it before in the context of association rules and interestingness.

As was pointed out by @Sean Owen in his comment, it is one of a few standard notations for saying something is not in a set - i.e. the complement. This would make most sense, as we are generally referring to itemsets when computing the associations.

Regarding the complement and set notation, the Wikipedia article contains this:

  The absolute complement of A is usually denoted by $${\displaystyle  A^{\complement }}$$

  Other notations include $$ {\displaystyle A^{\text{c}}}, \hspace{3mm} \overline A, \hspace{3mm} A',\hspace{3mm} \complement _{U}A \hspace{3mm} and \hspace{3mm} \complement A$$

Of course a bar can also mean things such as the mean or expectation value of some values (I have even seen people using it to mean the unobserved or predicted value!), but with the information you provide, the complement seems most likely.

Here also from the same Wiki page linked above for those who are unsure, a visual definition of the complement:


Why is the correlation coefficient of a constant function with function input is not defined?

The correlation coefficient between two random variables is a rigorously defined mathematical parameter. It is undefined when either of the random variables have zero variance. Therefore "NaN" is a very appropriate value to return in this case.

You are welcome to define it to be zero when one of the random variables is constant, if you find this more practical. But this does not agree with my mathematical intuition using limits. For example, if $X$ is $Uniform[-1,1]$, then the correlation coefficient between $X$ and $Y = c + \epsilon X$ is 1 for all $\epsilon > 0$, -1 for all $\epsilon < 0$. You can modify this example to get functions that get closer and closer to a constant function as $\epsilon$ goes to zero, while the correlation between X and its image is any number from the interval [-1,1].

I also like this explanation by David Epstein: the cosine of an angle between a non-zero and a zero vector is fundamentally undefined.


Score remains same during hyper parameter tuning

I would suggest the following changes,

Try KerasRegressor instead of KerasClassifier.
Use other activation functions than relu (Eg: tanh, or just linear)
Normalize both target and regressors.


Can random forest algorithm provide customer churn prediction probability at each customer instead at class level?

I want to know if it is possible to get the churn prediction probability at individual customer level & how by random forest algorithm rather than class level provided by: predict_proba(X) => Predict class probabilities for X.

I think the predict_proba(X) is actually the correct method to accomplish your task.  For each instance (person) the function will display the probability for each class label. So, if you know which class is churn, I'll just assume class 1, you just need to slice the results on that column.  For example:

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from tabulate import tabulate

X, y = make_classification(n_samples=1000, n_features=50,
                           n_informative=2, n_redundant=0,
                           random_state=1, shuffle=False)
clf = RandomForestClassifier(n_estimators=100, max_depth=2,
                             random_state=0)

clf.fit(X, y)

train_class_probability = clf.predict_proba(X)

print(tabulate(train_class_probability[0:4], ['Churn', 'No Churn']))

NewData, y = make_classification(n_samples=10, n_features=50,
                                 n_informative=2, n_redundant=0,
                                 random_state=1000, shuffle=False)
class_prediction = clf.predict(NewData)
class_probability = clf.predict_proba(NewData)

print(tabulate(class_prediction.reshape(-1, 1), ['0 - Churn / 1 - No-Churn']))
print(tabulate(class_probability, ['Churn', 'No Churn']))
# Just print the first column, Churn
print(tabulate(class_probability[:, 0].reshape(-1, 1), ['Probability of Churn']))

Output will be:

Class Probabilities of the Training Data (First four instances)

   Churn    No Churn
--------  ----------
0.595298    0.404702
0.620975    0.379025
0.601251    0.398749
0.610663    0.389337

Class Prediction on New Data

  0 - Churn / 1 - No-Churn
--------------------------
                         0
                         0
                         0
                         1
                         1
                         1
                         1
                         0
                         1
                         1

Class Probability Prediction on New Data

   Churn    No Churn
--------  ----------
0.553698    0.446302
0.602109    0.397891
0.587715    0.412285
0.423982    0.576018
0.419588    0.580412
0.419984    0.580016
0.369798    0.630202
0.572373    0.427627
0.414734    0.585266
0.40422     0.59578

Probability of Churn on New Data

  Probability of Churn
----------------------
              0.553698
              0.602109
              0.587715   <- new customer #3
              0.423982   <- new customer #4
              0.419588
              0.419984
              0.369798
              0.572373
              0.414734
              0.40422

So, the probability that "new customer #3" will churn is ~59% and the probability that "new customer #4" will churn is ~42%.

Hopefully, I have understood your question.

HTH


Model inclines because of imbalanced data

Impact of imbalanced datasets
First I would say that imbalanced dataset impact depends on the type of model you are using.
For instance:

Gaussian Naive Bayes should not be that much impacted if you have a certain amount of data for each class that is enough to approximate your gaussian distributions. (and that your data are normally distributed)
Neural Networks learning is using the error of your predictions to update its model, so having an imbalanced dataset would lead to imbalanced learning (for example 70% of the weights/biais have been updated according to class 'Good'). You don't want this happening a priori.

How to deal with imbalanced dataset ?
There might be other approaches, but you can do at least:

Use an algorithm that is not much impacted by imbalanced datasets

Some algorithms have a class_weight parameter. You can use it to penalize more the minority class during the learning process so the model is forced to pay more attention to the minority class observations. See this post for more details: How does the class_weight parameter in scikit-learn work?

Resample your dataset so it becomes balanced. That could be through undersampling the majority class or oversampling the minority one.

Further reading
How to Deal with Imbalanced Data
8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset
Hope it helps.


$y'(x)$ means you will work with $ROC' = ROC^{-1}$ (inverse of $ROC$), as all true positive will be falsely negative and vice versa. Therefore, $AUC' = 1 - AUC$ (As ROC is an increasing function and inside a unit square), and your answer is yes.


First will explain the terms loss, acc, val_loss, val_acc and then get into evaluating model performance.

Loss - This metric is used to understand the prediction by the network on training data vs the actual value. Given that you are using categorical crossentropy as your loss mechanism, your loss is actually telling you how near or far your predicitions are from the true classification values.

Acc - Is the accuracy of the network on the training data. After training on 1750 samples, your network can accurately predict with 0.98 accuracy.

Val_loss, Val_acc - The metrics are the same but are evaluated on validation data, that is the data that is not part of the training dataset, this will help you evaluate how your network performs on data that it is not trained on, sort of mimicking how your network would work when deployed in the real world on the chosen dataset.

The above metrics help you understand how your network is performing every epoch, giving you an idea on how to improve the performance.

How to perform model performance:

Model performance requires you to create/manage training dataset that would depict what you can expect in the real world when deploying your model to actually predict.

You would look at test accuracy, precision, recall, f1 scores, confusion matrix and other metrics based on the problem you are solving to truly evaluate your model performance.

Hope this answers your question. Have fun with deep learning networks :)


where does the downloaded file gets stored in the R-Studio Server on Ubuntu EC2 Instance when using download.file()

It'll put it exactly where you asked for it:

 destfile = "Haberman.txt"

that is, a file called Haberman.txt in the current working directory at the time you ran download.file. You can get the current working directory with getwd() in R.

So for example if its something you can read with read.csv, you can do:

hdata = read.csv("Haberman.txt")

as long as you haven't changed the current working directory (like with setwd).


Is ROC and AUC the only criteria for choosing a model?

No, area under receiver operating characteristic (AUROC) is just one metric amongst very many possibilities, even assuming you just want to pick a standard approach. There are too many to list in a simple Stack Exchange answer. You can take a look at this list extracted from scikit learn documentantion on metrics for example, which is not by any means exhaustive:

Accuracy classification score.
Area Under the Curve (AUC) using the trapezoidal rule
Average precision (AP) from prediction scores
The Brier score.
The F1 score, also known as balanced F-score or F-measure
The F-beta score
The average Hamming loss.
Average hinge loss (non-regularized)
Jaccard similarity coefficient score
Log loss, aka logistic loss or cross-entropy loss.
The Matthews correlation coefficient (MCC) for binary classes
Area Under the Curve (AUC) from prediction scores
Zero-one classification loss.

What you should choose depends entirely on your model class and goals for the learning work. AUROC is a good choice for a binary classifier when you have different business costs associated with false positives and false negatives. That is because it gives you a sense of how well the classifier can be tuned to be more or less sensitive, and get the best outcomes by changing the class threshold.

There is no minimum AUC or other metric required to select a model in practice. It depends on performance of existing solutions (including non-ML ones) and what the costs vs benefits would be of using the model. Clearly a poorly-performing model (e.g. with AUROC 0.5, no better than random guessing on a balanced data set) is unlikely to gain much from being implemented in a production environment, and you would want to take a new look at the original problem and see if was reasonable to expect anything from ML at all.

To decide which metric to use, you need to define some goal for the eventual model. An ideal metric is one that can be simply converted to terms that the end users will care about. Failing that, one that can be compared to known results from other ML work in the subject area (domain) of the problem.


not quite sure about the difference between RNN and feed forward neural net

The authors state that's the formulation for a feed forward deep learner, so you're exactly right. The two equations at the bottom of the page are where they formulate their recurrent neural net

The response is

$\hat{Y} = \text{softmax}(W^2Z_t+b^2) $

and the hidden state is

$Z_{t-j} = \text{tanh}(W^1[Z_{t-j-1},X_{t-j}] + b^1), j \in\{k,...,0\}$

The authors punt to this paper for implementation details, but the recurrent nature here comes from directly articulating your hidden state off of your most recent hidden state $Z_{t-j-1}$ (which involves the input $X_{t-j-1}$ from the last time step).

Hope this helps!


How to handle large number of categories in a dataset?

Algorithm like Decision Tree, can also work well on ordinal values, i.e. without OneHotEncoding. You can try this one.

Also Im not sure the importance of Title Name, so you have to take decision based upon the requirements. But I have avoided in my use case.

  Also, generally when you have large number of categories, you can give
  a try by clubbing uncommon categories into one.


Dealing with the test set of imbalanced data

For a case like this, it would probably be the easiest to do the train/test split manually. Here is a quick example with Pandas:

import pandas as pd
import numpy as np

# Create an empty dataframe with 60*A, 30*B, 20*C and some random target values
df = pd.DataFrame(columns=['category'])
df['category'] = (["A"] * 60 + ["B"] * 30 + ["C"] * 20)
df['target'] = np.random.rand(len(df))
df.category.value_counts()

> A    60
> B    30
> C    20
> Name: category, dtype: int64

# Obtain the size of the least represented category
smallest_category_size = np.min(df.category.value_counts().values)

# Size of the test split of the smallest category
test_size = 0.1

# Test count (2 in this case)
test_count = int(test_size*smallest_category_size)

# New train/test dataframes
train_df = pd.DataFrame(columns=df.columns)
test_df = pd.DataFrame(columns=df.columns)

for cat in df.category.unique():
    # Select only a part of the dataset containing one category
    cat_df = df[df.category == cat]

    # Create and shuffle boolean mask
    mask = np.array((len(cat_df) - test_count) * [True] + test_count * [False])
    np.random.shuffle(mask)

    # Append the selected values to the train/test
    train_df = train_df.append(cat_df[mask])
    test_df = test_df.append(cat_df[~mask])

train_df.category.value_counts()

> A    58
> B    28
> C    18
> Name: category, dtype: int64

test_df.category.value_counts()

> A    2
> B    2
> C    2
> Name: category, dtype: int64


I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows:

criterion = nn.CrossEntropyLoss()
total_loss = 0.
...
for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
    ...
    loss = criterion(output.view(-1, ntokens), targets)
    loss.backward()
    total_loss += loss.item()
    log_interval = 200
    if batch % log_interval == 0 and batch > 0:
        cur_loss = total_loss / log_interval
        ...
        print('ppl {:8.2f}'.format(math.exp(cur_loss)))
        ...

As @SpiderRico reminded, I got it from this link


Individual feature selection methods assign a numerical value to every feature so that features can be ranked according to this value. The calculated value is chosen to represent how much the feature contributes to knowing the label/response variable: common choices are conditional entropy, but also information gain or correlation.
The actual values assigned to the features are not really useful on their own, what matters is the ordering of the features according to this value. Thus the standard method for selecting the features is not to choose a particular threshold on the value, but simply to choose the number of features one wants to obtain.
Example: in a text classification task, there are 1000 documents and a vocabulary of 20000 unique words as candidate features. Using all the words would certainly cause overfitting, so we decide to use only 100 words as features. We can calculate the conditional entropy of every word with respect to the label, and then select the bottom 100 words according to the corresponding ranking as features (the 19900 other words are ignored).
Since individual feature selection is very efficient, it's often possible (and a good idea) to try a range of values as the number of features, and train/test the model for each of these values. This way one can experimentally determine the optimal number of features (the one which maximizes performance on the data). Note that this a form of hyper-parameter tuning, and therefore one has to use a validation set for the tuning stage, and then the final model (with the selected optimal number of features) should be applied on a fresh test set.


Handling encoding of a dataset which has more than total 2000 columns

To correctly encode your variable you have to know what those variables are about. An algorithm needs to somehow understand the type of your variable to automatically encode them. In such a case, you should have a dictionary for variables about the type of variable (sometimes, dataset guides or readme files, or some txt files contain that). Or you have to know that all the variables are monotype, so you can apply the same encoding. If you don't have these or similar information sources about the data, it is not possible (unless you have a perfect model to categorize the variables according to their type :)) ) to automatically encode them.
Although you cannot categorize the types of categorical variables automatically, it is possible to distinguish the continuous (and also discrete) and categorical variables. When I face a similar situation of having lots of variables, one of the very first things that I do is to have a count and percentage of distinct values for every variable. Thus, for example, if a variable with 200000 samples has ~154000 (unless there is a variable with 154000 categories, which is almost possible) distinct values, then it is a continuous (or discrete) variable. If a variable with 200000 samples has 13 distinct values then it is a categorical variable for sure. Using similar tricks you can identify categorical variables. However, thereafter, it is inevitable to analyze categorical variables one-by-one. When you categorize them within themselves e.g. rank variables, nominal variables, etc. you can altogether encode each variable type.


How to Classify Documents whether they are similar to previous documents or not?

Only way to check which is best model is to try different in practise and compare. Do you have any documents to train with that are not similar? Do you have labels or topics assigned to these 100 ?

Assuming you have some irrelevant documents you can use a "spam" approach to this type of problem.

I would begin with Bag of Words representing of your documents for a Naive Bayes model generation (common, good performing, baseline here) - this link http://sebastianraschka.com/Articles/2014_naive_bayes_1.html#naive-bayes-and-text-classification is a good explanation of this for documents .

I would also try using a library like Python gensim that will allow you to try its various methods, and you could try that library's Doc2Vec method. Then train a linear discriminant classifier like Log. Regression with the document vectors it gives you, and try see if its resulting accuracy is better than the Naive Bayes method.


Theano used to be very popular in the last few years. However, from seeing what the top research labs in the world currently use instead of theano (Facebook AI and Google Deepmind), I would either go for caffe or Torch7, with a stronger preference on the latter.
The reason is that you can implement from Recurrent Neural Nets to Convolutional Neural Networks very easily and there is a wide range of examples on github.
Graduate students in NYU, Stanford, Oxford definitely implement advanced models in their studies; your background will help you a lot, a great place to start and familiarise with such models, and implement them in Torch7, is the lessons of Prof. Nando de Freitas.


How to professionally spell library names such as "scikit learn"?

Scikit-learn has a citation guide: https://scikit-learn.org/stable/about.html#citing-scikit-learn
They spell it with a uppercase "S" and lowercase "l":

Scikit-learn: Machine Learning in Python


Just run

PROC means data=d mean; var a; run;

for overall data


How to interpret coefficients from logistic regression?

They are all significant but for certain thing.
What do I mean? You are predicting evidence, i.e. first column of the following picture:

In other words you have "linear regression part"+ instead of y you have evidence. So changing values of independnet variable X (positive or negative) will influence different binary class (0 or 1), hence different values are significant for different thing. (they add some info)


Is active learning able to detect challenging cases?

In general it depends on the exact method used to select instances and of course on the data. Assuming that the selection is based solely on the uncertainty measure of a single classifier, then by definition the method will prioritize instances predicted with a probability around 50%, i.e. where the classifier is "unsure". As a consequence an instance predicted with a high probability is unlikely to be selected for annotation. However the iterative training process will make the classifier re-estimate the probability of all the instances, so it's possible that an instance wrongly classified with 90% probability at a particular iteration will later be assigned a lower probability, or even the true class. But overall there's no guarantee: like with any statistical system there can be instances misclassified with a high probability.

The rationale behind my question is: does adding more samples to the training set always improve the performance of a classifier?

In active learning, the performance depends more on how many instances end up being manually annotated than on the size of the unlabelled sample. But as usual the performance strongly depends on the data itself.


Should we also include negative instance in cross-validation process of one-class classifiers?

The evaluation design depends on the goal, i.e. how the classifier is intended to be used in production. The goal of a one-class classifier is to find instances of this class among other instances, so it has to be evaluated with data which includes negative instances. The choice of the proportion and type of negative instances should be driven by the final goal.
Note that in terms of evaluation and cross-validation the setting is the same as regular supervised classification, the difference is only that negative instances are not used during training.


Personally, we use S3 on top of GCE and really love it. Depending on how much data you're dealing with, Google Drive just doesn't quite match the 5 TB max that S3 gives you. Also, if you're using python, boto does a pretty fantastic job of making most aws services pretty accessible regardless of what stack you're dealing with. Even if you're not using python, they've got a pretty straightforward API that generally is more accessible than Google Drive.

Instead of google drive, though google did recently release a cloud storage service, apart from drive, that lets you more closely integrate your storage with any gce instance you've got, https://cloud.google.com/products/cloud-storage/

They've got an API which seems to be pretty comparable to S3's, but I can't profess to having really played around with it much. Pricing-wise the two are identical, but I think that the large community and experience with aws in general still puts S3 squarely above both google's cloud storage and google drive.


Is there an appropriate sequence of parameters to be considered when building a pipeline in Scikit learn?

What you are asking for is a gradient based hyperparameter optimization. To enable such a grid search one would have to define a metric on the space of all hyper parameter combinations for the current pipeline.

I don't think this makes sense in your example because of the binary nature of the use_idf flag. Lets say your parameter combinations are (use_idf, alpha). Then you would have to do decide what the distance between (True, 0.00015) and (True, 0.00010) is. In your case it should be smaller than the distance between (True, 0.00015)and (False, 0.00015). So

d((True, 0.00015), (True, 0.00010)) < d((True, 0.00015), (False, 0.00015))

But what about the distance between (True, 0.00015) and (True, 0.05)? Will it be still smaller than d((True, 0.00015), (False, 0.00015))? I think you get the point.

By choosing the metric you will subjectively influence the search for the best hyper parameters. Probably a random search would be more suitable as suggested in the following paper

@article{bergstra2012random,
  title={Random search for hyper-parameter optimization},
  author={Bergstra, James and Bengio, Yoshua},
  journal={The Journal of Machine Learning Research},
  volume={13},
  number={1},
  pages={281--305},
  year={2012},
  publisher={JMLR. org}
}


Would my code be faster if I rewrite it with matrices?

Without seeing the code it's impossible to know, but very likely. Also, I would never model single neurons. Too much overhead without any use. Model layers instead.

  how can I get my performance to be comaparable to that of sklearn

Sklearn is open source. Read the code: https://github.com/scikit-learn/scikit-learn/blob/7389dba/sklearn/neural_network/multilayer_perceptron.py#L682


I want to build a data synchronization scheduler which keeps track of the amount of data sync after every scheduled triggers and auto-adjusts the next schedule.

This doesn't immediately strike me as a "use machine learning" problem to be honest. If you just want the scheduler to schedule the next run to be after an amount of time that's determined by the number of records processed in the current batch then that's quite a simple and deterministic formula. You could have the default gap be 20 mins and then do something like:

next gap = 10 * (10 / n)

Where n is the number of records processed in the last run. That would mean in this case:

  So let at 10 AM Job 1 got executed and got 10 entries.

  At 10 AM job 2 got executed and got 100 entries.
  At 10 AM job 3 got executed and got 200 entries.

The next Job 1 would be scheduled at 20 * (20 / 20) = 20 minutes, the next job 2 would be at 20 * (20 / 100) = 4 minutes and the next job 3 would be at 20 * (20 / 200) = 2 minutes.

If you really wanted to use ML for it, I guess I'd suggest using a time series forecasting algorithm like ARIMA or Prophet to predict the number of samples each job will have to process over the next hour or something, and set the next run-time appropriately based on that.


Machine learning in electrical circuit simulation

For waves one usually uses a Sequence Model but for this purpose I think it will probably be the simplest to use a CNN. I am not very familiar with waveforms, but they seem to be similar to standing waves (time independent) or maybe even better as they also have scale invariance. Assuming the number of input waveforms are the same, I would simply stack them side by side and treat the result as any other image and do image classification using a CNN. You can probably also do the analysis by building a multi input CNN, but you probably have to build it from scratch, I don't know any samples of this kind of architecture available online.


Feed forward neural network, output as list of targets and associated probabilities

If you have more than two targets, then you should use softmax activation instead of sigmoid. Softmax activation gives you the probability associated with each class in the output. The only thing you need to do before applying softmax is to convert your targets into one-hot encoded vectors. If you want to know more about softmax in detail, you can read about it here


Do we need to increase training data size when increasing dropouts?

No, you should only tune one hyperparameter at a time. If you change two hyperparameters and the performance increases, how do you know which of the two parameters is responsible?

If you have the time, do a full grid search on the dropout rate and the training data size.


What is the most efficient method for hyperparameter optimization in scikit-learn?

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, they become a more opaque black box. This is the case for deep neural nets and presumably complex trees as well. A surrogate model attempts to regress the underlying space within that black box. Based on a variety of sampling techniques, they probe the hyper-parameter space and attempt to build a function which represents the true underlying hyper-parameter space.

Bayesian optimization focuses on the surrogate model and how this model is constructed is crucial to BO. Also crucial to BO is choosing a good loss function.

I think performance between random search and Bayesian search varies from dataset to dataset, and model to model. Bergstra & Bengio (2012) made a strong argument for random search over grid search. Shahriari et al. (2016) make a strong case for BO. Model-based Hyperband strategies can potentially perform better than BO, especially for high-dimensions, however it is purely exploration, not exploitation. This can easily result in too early of stopping. However, there have been efforts to combine Hyperband and BO.

I've had good success scikit-optimize, despite there being quite a bit unimplemented. It's easy to prototype with and can easily interface with scikit-learn.

Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb), 281-305.

Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & De Freitas, N. (2016). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1), 148-175.


How to interpret linear trends in residuals?

When I see it correctly in the top figure, there is some " bunching" in your data, meaning that there are a number of companies with the same (or very similar) number of employees. Since you appear to run a regression with only one independent variable (right hand side), this bunching will be visible in the residual.
Your model is:
$$ y_i = \beta_0 + \beta_1 x_i + u_i$$
Now say $\beta_0 = 1$ and $\beta_1 = 0.1$, and with "bunching" in $y$ you will get something like:
y     x      y_hat  u_hat
--------------------------
5     10     2      3
5     20     3      2
5     30     4      1
5     40     5      0

So the linear nature of the model will be reflected in the residual given that there is "bunching" in the data.
Note that using a log-log approach (which is perfectly okay) will change the natural interpretation of the estimated coefficients. In a log-log case, you will interprete the results as "a one percent change in $x$ will be associated with a $\beta_1$ percent change in $y$" (all other things equal).
Since the number of emplyees is bounded (no negative employees -> "count data"), things like Poisson regression could be worth a try.
Overall your model may be underspecified. If you can, include additional $x$-variables in your model, so to better reflect the data generating process. See page 114 in this book.


How many features should I take out of 515 features?

Disclaimer:  I do not know enough about MATLAB to answer the full question but I'll have a go at answering the rest
1 - How many features should you take?
If this is a predictive problem you're trying to solve, how about you try and assess the accuracy of your predictions by using features sets of 7 and 10 respectively? You might find that there is very little difference in predictive power so taking 7 might be OK.
NB: I am quite surprised that your feature selection algo selected such a small proportion of your available features - you're right to want to dig a little deeper on what's happening under the hood..
2 - Will any ML algorithm be able to capture the characteristics of the data with just these 7 features?
If your feature selection algo is working fine then there's all the likelihood that a ML would be able to characterise the underlying structure of your dataset relatively easily.
There's a number of problems out there that have a large amount of explanatory variables but are best characterised by a select few (for instance GDP is usually relatively well explained by what it was last quarter, with all other related variables having small impacts on prediction accuracy).
3 - MATLAB QUESTIONS: TBC
Sorry I can't answer all your questions - maybe someone will come by with MATLAB knowledge and we can roll our answers together. Otherwise you might be better splitting this into two questions to get more detailed answers on the MATLAB algo.


Does GridSearchCV not save the best parameters?

It doesn't, please try the following code
CELL1:
import numpy as np
from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV
iris = datasets.load_iris()

CELL2:
%%time
parameters = {'kernel':('linear', 'rbf'), 'C':np.linspace(0.1,100,1000)}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters)
clf.fit(iris.data, iris.target)

CELL3:
%%time
clf.best_params_

Wall time of CELL2 will be about 7-9 seconds.
Wall time of CELL3 will be 0ns. ( instantaneous )
This is because best_params_ is an argument of GridSearchCV. It is however only created (and accessible) once you run .fit method.


I believe this is the answer: https://github.com/dmlc/xgboost/issues/651
the sklearn api uses n_estimators= 100 as default whereas xgb.train is using n_boost_rounds=10

As both refer to the same parameter this could explain the huge difference.


How do subsequent convolution layers work?

I am not sure about the alternatives described above, but the commonly used methodology is:

Before the application of the non-linearity, each filter output depends linearly on all of the feature maps before within the patch, so you end up with $k_2$ filters after the second layers. The overall number of parameters is $3 \dot{} 3\dot{}k_1 + k_1\dot{} 5 \dot{} 5 \dot{} k_2$.

Bonus 1: Pooling is done per feature map, separately.

Bonus 2: The order of "sliding" does not matter. In fact, each output is computed based on the previous layer, so the output filter responses do not depend on each other. They can be computed in parallel.


How to deal with a skewed data-set having all the samples almost similar?

This is the so-called "Class Imbalance Problem". Fortunately, there are a number of possible approaches.

Probably easiest is going to be to use the class_weight option present in a number of scikit-learn's classifiers. For example, LogisticRegression.

You can either set this option to "balanced" (or "auto" if you're using an older version of sklearn), in which case it automatically adjusts weights inversely proportional to class frequency (i.e. it tries to compensate for the class imbalance); or you can manually set the weights for each class label using a dict, e.g. {0:1, 1: 20}.

For a worked example using an SVM, see here.

See here for more information: https://stats.stackexchange.com/questions/131255/class-imbalance-in-supervised-machine-learning


Why is it the Most significant feature from xgboost model is not at all significant when i build a model with same data set but using catboost?

There is no guarantee that you get the same subset of features from any feature selection algorithm. There might be multiple subsets that all perform approximately equally.  For instance, perhaps feature C is highly correlated with feature I.  So, there is no reason to be surprised by what you are observing.  There is no notion of "too low".


Are the raw probabilities obtained from XGBoost, representative of the true underlying probabilties?

It depends on the definition of accurate model, but in general the answer to your question 1) is No.

Regarding your second question (based on results in the paper of Niculescu-Mizil & Caruana linked below):

boosted trees and stumps - NO
Naive Bayes - NO
SVM - NO
bagged trees - YES
neural nets - YES

You can test whether it is the case for your particular model and dataset by looking at the so called reliability plot:

Create N bins based on the model output (e.g. 10-20)
Create a scatter plot with average model output for each bin along X axis and average true probability for each bin along Y axis

Ideally, your X-Y points should lie near the diagonal Y=X, otherwise the output of your classifier cannot be interpreted as a probability of an event.

However, not all is lost and if needed, one can try to modify (calibrate) the output of the model in a such way that it better reflects the observed probability. In order to assess whether the calibration exercise was successful one might look at the reliability plot based on the calibrated model output (instead of using raw model output).

Two most widely used techniques for the calibration of classifier outputs are Platt scaling and isotonic regression, see the links below.

Note, that it is not advisable to cailbrate the classifier using the training dataset (you might need to reserve a separate subset of your data for the purpose of calibration).

Some relevant links

Predicting Good Probabilities With Supervised Learning

CALIBRATING CLASSIFIER PROBABILTIES

Classifier calibration with Platt's scaling and isotonic regression


activation = ‘softmax’ should be used for multiclass classification whereas ’sigmoid’ for binary classification.

You can refer to: http://dataaspirant.com/2017/03/07/difference-between-softmax-function-and-sigmoid-function/

If changing the activation function does not help, I will be around for alternative solutions.


Transfer learning by concatenating the last classification layer

It turns out that it was an issue related with the order of classes being loaded in the model.

Let's assume I have the following structure

root
  -train1
    +a
    +r
  -train2
    +b
    +y
  -all
    +a
    +r
    +b
    +y

Where a, r, b and y are classes (folders with images). I first train with the ones at train1, so the network assigns each output to a class. Then I train a second network (everything but last prediction layer frozen) with the second training folder train2.

The order of the outputs of the prediction layer will change depending on the algorithm, but assuming it's loading the paths in a sorted way:

In the first network, the first output will be for a and the second for r.

In the second network, the first output will be for b and the second one for y, and when concatenated to the first network, it will be a, r, b, y

When I load the whole (merged) model, and I load a folder will all classes inside (all), the network assumes the outputs are for a, b, r, y in this order, whilst the concatenated network had a, r, b, y as the output order.

In this example, the outputs for a and y will be in the same position, while r and b will be reversed, which will lead to a bad accuracy.

TL;DR:

The order in which classes are loaded while you do the "incremental learning" needs to be the same always, so be careful in which order your classes are being loaded.


Help required to implement the below model using Bi-GRU

from keras.layers import LSTM
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense,BatchNormalization
model = Sequential()
model.add(LSTM(800, return_sequences=True,
               input_shape=(1, 64)))  # returns a sequence of vectors of dimension 32
model.add(BatchNormalization())  # returns a sequence of vectors of dimension 32
model.add(LSTM(800))  # return a single vector of dimension 32
model.add(BatchNormalization())
model.add(Dense(204, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='Adam',metrics=['accuracy'])

model.summary()

I figured it out on my own.


What is an appropriate machine learning technique to analyse development of status over time?

Yes, K-means is an option. But you need to do some data processing first.

My first concern is about quantifying those statuses (are they linearly depended, like does B means something in the middle of A and C)? then numerate them (A to 1 and B to 2 and C to 3 etc)
Represent your data set in a dimensional way, melt years columns into three columns

Year (with values of 2000, 2001, 2003 etc) so you get a time series dimension.
Year_status (status value).
Year_percentage (percentage of the year).

Try to treat pattern by moving window (kernel), and try different sizes.. for example size of 3 will cluster every 3 followed statuses into patterns centroids (for example.. having a common occurrence of A, E, A)


Image Classification task with unevenly sized images

The size of the input image is usually decided by computational resources you have. If input image has larger dimension you need to increase filter size, then model will have more parameters, hence you need more computational resources and longer training time.
Usually, I resize the images to min(heights of all images), min(widths of all images).
If there are larger samples of particular dimension in the dataset and you have enough computational resource, it might also makes sense to resize to that dimension.
Try with different settings, pick the one that best works for you.


Your problem looks to me more like a classification problem than a time series problem. My suggestion: Split the date into several sub-variables (year, month, day, week-of-day (maybe). Then just use this and the other values as input for a classification algorithm. Ideally you try several. I can recommend sklearn for this (http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html)

One advice: Depending on the classification algorithm you need to first normalize your data. I find the following blog useful for entry-level problems (with code examples--  if you want to approach this from a time-series perspective he also talks about this) and documentation http://machinelearningmastery.com/blog/.

Hope that helps.


One method would be to take many subsets of your dataset, i.e.
bootstrapping, build your models, perform cross-validation and calculate the average performance.  This is a good explanation of how the amount of data affects the model outcomes: https://stackoverflow.com/questions/25665017/does-the-dataset-size-influence-a-machine-learning-algorithm

Play around with the size of your subsets until you start getting stable results.


Sensitivity to scaling of features in a multivariate gaussians

There are many pitfalls to not scaling your data and it is generally very advisable to scale it. It is so easy to do, it is reversible, and it useful in other operations like removing outliers.

Upon further analysis of the specifics of Hidden Markov Models using Multivariate Gaussians the theoretical accuracy should not suffer as a result of drastic differences in the scales of your features.  But, an important operation involving multivariate Gaussian distribution is matrix inversion.

Though the theoretical invertibility won't change, the practical numerical solution to your problem will likely suffer inaccuracy due to the difference in scales.  Iterative methods will have issues with convergence and direct solve methods will suffer from stiffness.  This is especially true when common complications like linear dependence are present.

Here is a set of 3 lectures on HMM with some specifics on multivariate Gaussians (1-2-3)

I know we've already discussed this in the comments, but I wanted to close out this question, so have added it as an answer.

I hope this helps!


Convolution layer dimensions in deeper layers?

The output shapes for convolutional layers are calculated in the following way:

Let's say that the input shape for some convolutional layer is WxHxC:

W - width
H - height
C - channels

Now, assume that you have only one convolutional kernel, eg. size 5x5 (width and height). That kernel will actually be of size 5x5xC (C is the number of channels in the input shape) because one kernel must multiply all channels in the input when it is fixed in some place of the input. As you know, for that one fixed position, you get only one output number. When you repeat this for all input positions, you get only one output feature map (ie. with shape WxHx1 assuming that you keep the input dimensions by using padding ...).

In order to get more feature maps on the output, you need to repeat the process with new convolutional kernels (all those kernels must have the channel number equal to input shape channels). So, if you repeat this K times, your output feature map will have dimensions WcHxK. And the size of the kernel for that convolutional layer is said to be 5x5xCxK.

You can consult the following image:

In the image, one convolutional kernel (in the image it's called "Convnet Filter") outputs one output feature map (in the image it's called "One Feature Map").

Hence, in your case, for the first convolutional layer, the size of kernel is 7x7x1x100, and the output shape is 25x64x100. For the second convolutional layer, kernel size is 5x5x100x150 and the output shape is 13x32x150.

The above-described convolution is the "classic" convolution. There are other types of convolution, eg. Depthwise separable convolution, which use different ways of calculation. Consult A Basic Introduction to Separable Convolutions tutorial for more information about different types of convultion (it includes also descrption of "classic" convolution in the "Normal Convolution" section).


Fine-tuning a CNN for recognizing two classes, but also being able to tell if none of them is present in an image

Actually you are in the right path but in the question you are wrong in the second paragraph.

What you should do is as follows:

change the output layer to 4 classes, dogs, cats, dogs and cats, none.
If you have so many data, don't freeze the convolutional layers and train them all but if not, freeze them and fine tune the last layers, maybe just the dense layers.


Concat two tensors of different dimensions

For that, you should repeat b 200 times in the appropriate dimension this way:
c = torch.cat([a, torch.unsqueeze(b, 1).repeat(1, 200, 1)], dim=2)
c.shape

As desired, the shape of the result is torch.Size([500, 200, 15])


How to preprocess an ordered categorical variable to feed a machine learning algorithm?

One way to do is to use target encoding:

https://medium.com/analytics-vidhya/target-encoding-vs-one-hot-encoding-with-simple-examples-276a7e7b3e64
https://towardsdatascience.com/target-encoding-and-bayesian-target-encoding-5c6a6c58ae8c

(There are a million resources to learn target encoding)
This way your categories will not only be ordered by the number but for the target value (what is at the end what you want, to give better predictions)


Finding the most phonetically similar word from WordNet

You could use e-speak to convert your words into a phonetic alphabet. What you can also do is use fuzzy-matching. Here is a nice blog post about this.


Can you l2 normalize word2vec vectors for density clustering?

You can do DBSCAN, OPTICS, HDBSCAN with cosine similarity - they do not expect or require Euclidean distances.
Yes, one version of cosine distance corresponds nicely to Euclidean distance of L2 normalized vectors. So there is some theoretical support to L2 normalize the vectors and try Euclidean then.
Given how word2vec is trained, I don't think Euclidean nor cosine distance is the right thing to do. Instead, use the raw dot product as a similarity measure (beware that it is not limited to [0;1] but allows negative similarity - values may be too spread out to use them intuitively).
t-SNE, which was discussed in the comments and another answer, also relies on distances to the nearest neighbors. So it does suffer from the same problem - cosine, Euclidean, dot product? The fast ELKI implementation does support cosine distance as well as Euclidean, but you appear to be set on Python already. Also, if you have a large vocabulary, t-SNE may give you some scalability trouble. The standard algorithm will need O(n²) memory and time.