Converting a nominal attributes to numerical ones in data set

By converting a nominal attribute to a single numeric attribute as you described, you are implicitly introducing an ordering over the nominal labels which is a bad representation of the data, and can lead to unwanted effects from a classifier. Does it make sense to say that UDP should be inbetween TCP and ICMP? (no!) Imagine you are training a $k$-NN model on this data. It doesn't make sense to say that ICMP should be "further away" from TCP than UDP, but if you adopted the mapping that you suggested, the representation of the data has this assumption built-in. Alternatively, what if you are training a decision tree-based model? Usually, in decision trees, binary split points are chosen for numeric attributes. There could be some randomness in your training data where splits at certain values of the numeric attribute results in overfitting to noise.

Typically when converting a nominal attribute to numeric, one numeric attribute per nominal label is created. Each attribute is set to one if the corresponding nominal label is set, and zero otherwise. For example, if a nominal attribute called protocol has labels {tcp, udp, icmp}, then this dataset:

$$
\begin{array}{ccl}
\text{inst.} & \text{protocol} & \text{other attributes} \\ \hline
1 & \text{tcp} & \dots \\
2 & \text{icmp}& \dots \\
3 & \text{icmp}& \dots  \\
\vdots & \vdots & \ddots
\end{array}
$$

could be converted as follows:

$$
\begin{array}{ccccl}
\text{inst.} & \text{tcp} & \text{udp} & \text{icmp} & \text{other attributes} \\ \hline
1 & 1 & 0 & 0 & \dots \\
2 & 0 & 0 & 1 & \dots \\
3 & 0 & 0 & 1 & \dots \\
\vdots & \vdots & \vdots & \vdots & \ddots \\
\end{array}
$$

This is what the NominalToBinary filter does in WEKA. As you mention, the downside of this is that a large number of additional attributes can be introduced if the number of distinct nominal values is high.

If the dimensionality is too high after the conversion, you may want to consider using a dimensionality reduction technique such as random projection, PCA, t-SNE, etc. Note that this will reduce the interpretability of your model. You could also use feature selection techniques to remove some of the less useful attributes. It is possible that some of the nominal labels are not useful for your model, and you will improve performance by removing them. Another thing you could try is to use your domain knowledge to reduce the number of categories. For example, TCP and UDP are both transport protocols, maybe for your application the distinction between TCP and UDP is not that important and you can put instances with protocol $\in$ {tcp, udp} into a new category, removing the old ones.


Evaluating likelihood of egg breaking when falling in random container on concrete

This question might be better answered on the physics stackexchange, but I'll take a crack at it.

Although the factors you mentioned probably affect the likelihood the egg will break, some of them are not specific enough to be descriptive features. For example, how do you quantify "shape"? I will try to provide some more features that can be quantified, but some of them might take effort to retrieve.

Egg Features

Eggshell thickness

the thickness will probably not be constant, so you can take the mean, standard deviation, and even isolate some key points on the egg to sample such as the top/bottom points.
some studies indicate that "eggs with thin but more uniform eggshell were stronger than those with thick but less uniform eggshell" from this paper
you might be able to measure thickness using optical techniques such as imaging or lasers.

Temperature

both of the egg and the surroundings

Time since egg was laid

this might be approximated by the expiration date, particularly when only using a single brand

Density

you can calculate this by putting the egg in water (fresh eggs sink) to first determine the volume, then divide the mass by this volume.

Calcium deposits

the bumps you can see and sometimes feel on chicken eggs are calcium deposits that range in frequency and size. I am not sure how you would quantify these, but they could be another factor.

Wikipedia indicates some abnormalities with eggs that you might want to account for or remove from your samples. I copied the abnormalities here:

double-yolk eggs
yolkless eggs
Double-shelled eggs: where an egg may have two or more outer shells, is caused by a counter-peristalsis contraction and occurs when a second oocyte is released by the ovary before the first egg has completely traveled through the oviduct and been laid.
Shell-less or thin-shell eggs possibly caused by egg drop syndrome

Container Features

ideally the impact energy would not be absorbed by the egg, so somehow you need to measure this using estimates of the material metrics as well as shape parameters. One such measurement might be:

Average surface area that contacts the ground

You might find looking at the container's convex hull useful for visualization purposes

Many other factors can contribute such as:

Hardness
Resilience
Elasticity
Stiffness
Ductility

Lastly, there are probably many features that can be calculated by combining the egg and container, but those seem to be much more complicated. For example, measuring the amount of movement, on average, an egg can endure within a container by applying pressure to certain areas. I am not sure how you would easily extract these types of data, but they certainly exist.


Comparison with transfer learning and CNN from scratch on a small dataset

They have better performance for now.

If you continued to re-train your pre-trained NN you would spezialise more and more for your dataset at hand and converge to 89%.

But here is the kicker, if you used some pre-trained models for example based ImageNet dataset you would be able to generalise a lot more better. In other words for any images in the test data that were different from the ones in the train (or from the holdout/new set) you would perform better with the transfer learning.


Number of parameters keras dense layer with a 2D input

Yes, it takes only to the last dimension, accordingly to the source code (comments are mine):
https://github.com/keras-team/keras/blob/88af7d0c97497b5c3a198ee9416b2accfbc72c36/keras/layers/core.py#L880

def build(self, input_shape):
    assert len(input_shape) >= 2
    input_dim = input_shape[-1]  # uses last dimension


On regression to minimize log distance rather than distance

There is a loss called Root Mean Squared Log Error (RMSLE): $\sqrt[]{\frac{1}{n}\sum_{i=1}^n{(\log(y_i + 1) - \log(\hat{y_i} + 1))^2}}$ (do not forget the $+1$ as the $log$ is not defined at $0$)

You will find a brief explanation and discussion here. It has also been used in competitions as for example here.


What has to be considered when creating a radar chart?

Whenever it comes about a specific graph critique, I check the data-to-viz.com (I have not affiliation with it)

About the Radar or Spider Charts, you can find some drawbacks here. Let me summarize them for the readers.

Circular layout = harder to read

This is a common problem in every circular layout. The human eye can easily understand and compare data in a single vertical or horizontal axis, but not with the same convenience in circular plots.

Supporting the ranking

Bar charts and other similar plots, such as lollipops allow you to instantly graph the ranking of the data. Which is higher and which lower. This is not easy at all in Radar Charts where the are no starts and ends.

Category order has a huge impact

You need to be really careful with the order you will put your categories. Readers meant to focus a lot on the shape of the radar which will change significantly when you change the order of your categories.

About scales

It's common in Radar Charts to compare different variables with different scales at the same time. In those cases, you need to clearly specify the scale for each of them, otherwise, readers expect to have the same scale and unit everywhere.

Overplotting

One series of data is nice. A second one adds some eye-catching details. If you need more than that, it will become unreadably quite easy.

Over-evaluation of differences

Humans are able to understand the difference in horizontal or vertical axis linearly. However, when it comes to the area of a shape, it becomes quadratically. As a result, it's easy to overestimate a difference between two points in the Radar Chart.

Workarounds

Unfortunately, there aren't many things to do. If you believe your Radar Chart is not readable enough, then you can only change to a bar chart or a lollipop plot.

In your example, you might try to put the scale outside to keep the inside area more clear. Like this one

Image source: https://blog.scottlogic.com/2011/09/23/a-critique-of-radar-charts.html#occlusion


What's the best way to tune the regularization parameter in neural nets

Wikipedia lists some well-known approaches to hyper-parameter searches.

The brute-force scan/search, or a grid search across multiple parameters, is still a very common and workable approach. As is random search, just trying some variations of parameters automatically and picking the best result from cross validation.

A guided search by intuitive feel of what might best work is also still quite common approach, despite being characterised as "optimisation by graduate descent" (a pun based on the fact that a senior researcher will hand off the tuning work to their smart junior researchers who make educated guesses on hyperparameter values) - for regularisation parameters you can look at training curves and difference between training loss vs cv loss to help decide where to look. You generally want to increase regularisation if there is a large difference and decrease it if the values become close but not very good. What counts and as "large difference" and "not very good" is subjective though, and typically you only get a feel for what works after a few random attempts and immersing yourself in the problem so that you recognise good vs bad behaviour of a model.

There are some claimed "smart" schemes that apply Bayesian optimisation or similar, and even some automated services or paid tools that attempt to make these available for tuning models automatically. However, I don't see these used much in practice.

Unless you are trying to win a Kaggle competition, you quickly hit diminishing returns attempting to fine-tune your parameter. I've rarely found tuning the second significant digit of a meta-param like L2 loss to be worth much attention. Maybe with the exception of testing between e.g. 1.0 and 1.5 - often useful to think in terms of geometric progression, not linear, even when fine-tuning.

You also have to be careful that you are not simply tuning to your cross-validation set, with no meaningful or justifiable improvement for test cases or in production. You can mitigate this in part by using k-fold cross validation (which can reduce error on metrics estimates effectively by using more data to estimate them), although of course that increases time/CPU costs.


Applying neural network for simple x^2 function for demonstration purpose

I re-implemented your set-up in python using keras. I used a hidden layer size of 25, and all my activations were sigmoid's. I got to an accuracy of 99.88%. Try running your algorithm for a greater amount of epochs. Use binary cross entropy as the loss function and try decreasing the learning rate of your gradient descent algorithm. This should help increase your accuracy. My only explanation for the poor performance would be that you are getting stuck at a local minimum, if that is the case different initiations of your weights should fix that problem.


Clustering mixed data: Find optimal number of clusters

You should select 9 as you can see from plot that for the WSS value there is a dip.

It doesn’t matter if you have 2 features or 9 features or n features. Clustering is on Data present in those features(it might depend on the amount of data).


Experiment click to lead prediction with Azure ML

This looks like a two class boosted decision classifier.  That should perform quite well.  I used it several times about 2-3 months ago and got excellent results.  I'm guessing you are not training/testing the right features.  What is your target variable?  Is it 'hasLead'?  I'm not sure what will be statistically influential on that dependent variable, but you  can run a feature engineering exercise to see what independent variables have the most influence.  Run the code below; change to suit your specific needs (obviously, use your specific data).

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# matplotlib inline

df = pd.read_csv("https://rodeo-tutorials.s3.amazonaws.com/data/credit-data-trainingset.csv")
df.head()

from sklearn.ensemble import RandomForestClassifier

features = np.array(['revolving_utilization_of_unsecured_lines',
                     'age', 'number_of_time30-59_days_past_due_not_worse',
                     'debt_ratio', 'monthly_income','number_of_open_credit_lines_and_loans',
                     'number_of_times90_days_late', 'number_real_estate_loans_or_lines',
                     'number_of_time60-89_days_past_due_not_worse', 'number_of_dependents'])
clf = RandomForestClassifier()
clf.fit(df[features], df['serious_dlqin2yrs'])

# from the calculated importances, order them from most to least important
# and make a barplot so we can visualize what is/isn't important
importances = clf.feature_importances_
sorted_idx = np.argsort(importances)

padding = np.arange(len(features)) + 0.5
plt.barh(padding, importances[sorted_idx], align='center')
plt.yticks(padding, features[sorted_idx])
plt.xlabel("Relative Importance")
plt.title("Variable Importance")
plt.show()


Why does image segmentation benefit from fuzzy clustering?

If you take aerial images for example, you might need to identify overlaps.

Lets say your goal is to segment water, vegetation and rock areas. More to be assigned to the closest centroid, there are hidden memberships to other centroids that can be identified by fuzzy c-means image segmentation.

On this thesis at https://zone.biblio.laurentian.ca/handle/10219/2616, at section (4.5 Analysis - Task 1) you can find detailed analysis using both (k-means & fuzzy c-means).


Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

Short answer: NO.

The moment you convert the spark dataframe into a pandas dataframe, all of the subsequent  operations (pandas, ml etc.) will be run on a single-core as those algorithms and programs are written in native-python and doesn't support multi-core distributions. In a nutshell, someone has to rewrite the whole sklearn to again to be compatible with spark.

Said that there are some progress to bring e.g. to distribute scikit-learn on a Spark cluster. Some functionalities like the expensive GridSearch are rewritten in spark and you can use it together with native sklearn in databricks, see this post, or Joblib Apache Spark Backend (previously known as spark-sklearn). This blog post is also worth reading as well.


Suitable modelling approach to incorporate different signals

Combing multiple models together is called typically ensembling.
You are describing hierarchical ensembling. There are simpler methods for ensembling, such as voting and weighted voting.


Is dimension reduction helpful to select features for a classification problem?

Well, let's say it depends on the distribution of your data. In approaches like PCA the approach does not care about the labels of the data in hand. This is why PCA may lead to data which are sometimes difficult to be separated or vice versa. PCA just cares about which direction leads to more variance and take that direction as a new basis. Not caring about the labels is why you cannot say it may lead to a better space for classification or not. You have to employ that and after that, investigate whether it's helpful or not. Approaches like LDA or other variants of that take care of the labels but they are linear classifiers which are not strong at least in a current feature space where you've not done any feature engineering.


Multi target classification for different types of target variables

You have one classification task and one regression task, but sklearn's multioutput meta-estimators only support two tasks of the same type.

The best solution here is to train two models:

A binary classifier to predict $target1$
A regressor to predict $target2$

For example:

from sklearn.cross_validation import train_test_split
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor

# X, y = load training data

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf = RandomForestClassifier()
clf.fit(X_train, y_train[:,0])
print("classifier accuracy:", clf.score(X_test, y_test[:,0]))

regr = RandomForestRegressor()
regr.fit(X_train, y_train[:,1)
print("regressor R^2:", regr.score(X_test, y_test[:,1]))


Neural Network Timeseries Modeling with Predictor Variables

I think this thesis https://dspace.mit.edu/bitstream/handle/1721.1/99565/924315586-MIT.pdf?sequence=1 is a good starting point for building a model for time series forecasting with so called leading indicators and different machine learning models. Basically you need just to training sets where your inputs are past values of the variable you wish to predict plus the additional indicators (i.e. past_value_t-6, past_value_t-5 ..., past_value_t-1, additional_variable_1, additional_variable_2, ..., additional_variable_n) and the output is either single value in future (t_1) or multiple values (t_1, t_2, ... t_n), in case you wish to predict for example the next twelve moths' sales.

It is also possible to use recursive strategy to generate future values for your prediction variable. In this case you generate next value for the prediction value based on your previous predicted value. However, at least based on my experience the first mentioned direct strategy is way more efficient.


Convert graph database relationships into node properties

The cypher query that ended up working for me was MATCH (n) OPTIONAL MATCH (n)-[r]->(m) RETURN n,r,m

function makeAllBlocksQuery(cb){
  db.cypher({
    query: "MATCH (n) OPTIONAL MATCH (n)-[r]->(m) RETURN n,r,m",
  }, function(err,results){
    var processedResults = handleAllBlocks(err, results);
    cb(processedResults);
  });
}

function handleAllBlocks(err, results) {
    if (err) throw err;
    if (!results) {
      console.log('No blocks found.');
      return [];
    } else {
      //first, form [_fromId, _toId] relationshipArray
      var relationshipArray = new Array();
      for (count = 0; count < Object.keys(results).length; count++) {
        if(results[count].r !== null) {
          var relationship = [results[count].r._fromId, results[count].r._toId];
          relationshipArray.push(relationship);
        }
      }

      var resultArray = new Array();
      var repliedIdsArray = new Array();
      //add nodes found in results to resultArray
      //and add id property to properties for later
      for (count = 0; count < Object.keys(results).length; count++) {
        results[count].n.properties.id = results[count].n._id;

        //iterate through the relationshipArray for each result
        //and add all the replied Ids to the repliedIdsArray
        for (relationshipcount = 0; relationshipcount < relationshipArray.length; relationshipcount++){
          if (relationshipArray[relationshipcount][1] == results[count].n.properties.id) {
            repliedIdsArray.push(relationshipArray[relationshipcount][0]);
          }
        }
        //add the repliedIdsArray to the result's properties
        results[count].n.properties.hasreplyto = repliedIdsArray;
        repliedIdsArray = [];
        //stringify so it can be sent to client more easily
        resultArray.push(JSON.stringify(results[count].n.properties));
      }
      return resultArray;
    }
}


suitable algorithms for very high dimensional ”binary pattern" recognition?

For feature selection, you may want to look at PCA and I would suggest that you try Naive bayes Classifier and Random Forests.


Don't give to use in java :)

I had the same problem, try to change the scale value, I had the same labels from java and python after that.

        // - The model was trained with images scaled to 224x224 pixels.
        // - The colors, represented as R, G, B in 1-byte each were converted to
        // float using (value - Mean)/Scale.
        final int H = 224;
        final int W = 224;
        final float mean = 117f;
        final float scale = 255f;


Classifiers are commonly scored on the cross entropy or some other divergence, since we're talking about distributions. A multi-class classifier will give you a "score" associated with each class (option, outcome). It is usually easy to transform this score into a probability, so that the options add up to one. Furthermore, it is possible to associate with each class a confidence distribution. I don't know if this is what you are asking about, but here's a great blog post on it.


Determining if time series follows a pattern

First of all, digitalize your data on a regular time scale with a time delta shorter than the shortest assumed period. For instance, in the case of the milkman, you could create a table with one record per day having the value 1 if the man come and 0 if not.
The table would look like a regular time-serie :

day, presence
  1, 1
  2, 0
  3, 1
  4, 0

Then perform a autocorrelation analysis of your time-serie. This is a mathematical tool often used in signal processing that consists in, as suggested by @MichelGrogan in his comment, calculating the correlation between the original serie and copies of the serie delayed by distinct time deltas (1 day, 2 days, ...). The result can be plotted as values of correlation in function of the lag value and is called an autocorrelation spectrum. Read autocorrelation for more details.

The max values of the spectrum will give you the values of the most important frequencies observed in your data and confirm that pattern exists. If you do not see clear max values, you can conclude there are no pattern in your data.

Here is an example in Python from your data :

import numpy as np
import matplotlib.pyplot as plt

day =['mon','tue','wed','thi','fri','sat','sun', 'mon','tue','wed','thi','fri','sat','sun', 'mon','tue','wed','thi','fri','sat','sun' ]
num =[    1,    2,    3,    4,    5,    6,    7,     8,    9,   10,   11,   12,   13,   14,    15,   16,   17,  18,    19,   20,   21]
pres=[    1.,    0.,   1.,   0.,   1.,   0.,  0.,    1.,   0.,   1.,   0.,    1.,  0.,    0.,   1.,    0.,  0.,    1.,   1.,    0.,   0.]

corr=np.correlate(pres,pres,'full')
lag=[i-(len(corr)/2) for i in range(len(corr))]
plt.plot(lag,corr/np.max(corr))

And the spectrum plot :

As you can see there are clear maximum at 2 days and 7 days lags.


Where is the majority of Tensorflow memory usage?

Assuming your samples have a fixed size and you use constant batch size, you should never run out of memory - that is: as soon as you have managed to train a full batch (forward and backward propagation complete) you should be fine to assume it will train until completion!

Extra info / tips

The total memory requirement will be a product of the size of the model (number of parameters) along with the batch size. Each single sample that gets sent through the network will effectively influence each of the $N$ weights. A batch of size $B$ will mean that there needs to be roughly $N \times B$ memory available.

If you model is already fixed, you can start with a large batch size, decreasing it slowly until you no longer receive an outOfMemory error. You could also try computing exactly what your memory requirement are, but there are some other factors that might make that a little difficult, so it is faster to try the first approach.

If you are using a GPU, you ca look at the output of the terminal command nvidia-smi, you can see the available memory of the GPUs. You will notice it essentially becomes all used as soon as training begins. This is because Tensorflow, by default, will occupy all available memory. There are ways around that, so search for allow_growth in the Tensorflow docs for relevant information and configuration tactics.


How to protect data from internal data scientists?

You could checkout the OpenMined Pysyft library which is a library for encrypted, privacy preserving deep learning built over Pytorch. PySyft decouples private data from model training.

Github link to the Pysyft library - https://github.com/OpenMined/PySyft


Clustering documents - how to evaluate results?

None of the internal evaluation metrics will work well on text in my experience. Probably because of the curse of dimensionality.

Furthermore, DBSCAN does not cluster everything, but can also produce noise points. Few evaluation methods (and even fewer  implementations...) handle it well that noise is not a cluster.


Machine Learning to predict risk of items

This sounds like a homework problem and those are frowned upon on this web site; so I'm going to be cautious about helping you and just give you general guidance:

1 - be very, very careful about picking models before you do anything else. That is a bit of Researcher Bias and you should try to avoid that; let the data tell you where to go, not the other way around.

2 - you have to think about what "factors" you can use to describe each chair. How old is it? What color is it? Does it have wheels or not? Some of these features will have an effect on your models, others will not. But it should be your goal to start with a long list of factors and then try to model them out and see which ones have an effect on your results.

Start putting a list of factors together and then you will be in a better position to continue your studies.


Out of the two pipelines you mentioned, I'd recommend the second (i.e. real-time augmentation). This is better than the first, because by performing random augmentations the network sees different images at each epoch.

I'd recommend imgaug, which is a python library for performing data augmentation. I've found it very helpful as it can work with keras' ImageDataGenerator very well. The way can do this is:

from imgaug import augmenters as iaa

seq = iaa.Sequential([...])  # list of desired augmentors

ig = ImageDataGenerator(preprocessing_function=seq.augment_image)  # pass this as the preprocessing function

gen = ig.flow_from_directory(data_dir)  # nothing else changes with the generator

One final note I'd like to make is that $7 \cdot 50 = 350$ images are very few for deep learning, even with augmentation. Make sure you use a pre-trained network or else you will have a serious overfitting problem.


What’s the most suitable programming language for AI development?

Disclaimer: this is massively subjective topic, and below is just my opinion.

I would say it depends on your focus. You could do work in these areas with any major programming language, but some have better suited capabilities and/or better support from a community.
My biased and very simple answer would be to use Python for basically everything, and learn C++ for times when performance is important. And then, if you get really serious, you'll need to learn CUDA (extensions to C++ to programme directly for Nvidia GPUs). If you are proficient at Python and C++, there isn't really anything you couldn't do.
In summary

If you want to do research, you could probably do everything with just Python. There are libraries such a NumPy/PyTorch/Tensorflow which do all the heavy lifting for you.

If you want to go the way of robotics and embedded hardware/software, you'll likely need C/C++.

If you want to make models to deploy in a browser, perhaps Javascript would be useful.


How to transform time series data to apply supervised learning algorithms to it?

You need to transform your timestamp column into several numerical/categorical columns representing that specific timestamp. After converting that timestamp column into features, you can simply throw supervised algorithms on the problem.

You could leverage for example the fast.ai function add_datepart that you can find here:


How can i tell if my model is overfitting from the distribution of predicted probabilities?

Such a plot doesn't really tell you much about overfitting.
First, check that your calibration has worked well; it's possible that an incorrect calibration has pushed the probabilities to the extremes.  Otherwise, the distribution of probabilities being so extreme suggests the data just naturally separates into a segment of easy-to-detect cancers and the rest.  Among the latter, it looks like you get reasonably good but not great rank-ordering of cases.


How to feed two (or more) time-series data sets to a neural network?

The whole idea of NN is that it learns by itself to give the appropriate weights and biases to the input neurons, so yes - just give your network the 2 series and it should return the appropriate weight and bias for each neuron in the input.


Loss function when the output is a single probability

At first I was going to say:

  It doesn't make sense to use use cross entropy loss in a regression
  problem!

  See explanation here.

But then I realised that if you are really trying to do regression on probabilities it could have some sense.

But still, why would you use it instead of L1, L2? So maybe try it and let me know if it works better!


How to fix the index value that I use for X_test and y_test?

I've changed the range values in the loop and indexing for training and test data. (Also, it seems you are using the dataset instead of data by mistake.)

i:(999 + i) contains thought rows locations of which will be incremented by one as the loop progresses.

for i in range(0, 999):

    X_train, y_train = data.iloc[i:(999 + i)], target.iloc[i:(999 + i)]
    X_test, y_test = data.iloc[i + 1000], target.iloc [i + 1000]


Using Neural Networks to extract multiple parameters from images

A CNN could be a good choice for this task if you expect variation in the original image scale, rotation lighting etc, and also have a lot of training data.

The usual CNN architecture is to have convolutional layers close to the input, and fully-connected layers in the output. Those fully-connected layers can have the output arranged for different classification or regression tasks as you see fit. Predicting the values of parameters describing the image is a regression task.

If you want accurate measures of size, you may need to avoid using max pooling layers. Unfortunately, not using pooling will make your network larger and harder to train - you might get away with strided convolution instead if that is a problem for you.

If your input images are very simple and clear (because they are always computer generated), then other approaches may be more reliable. You may be able to reverse-engineer image production and derive simple rules such as identifying lines, corners, circles and other easy-to-filter image components, and make direct measurements. There may also be a middle ground in complexity where extracting this data as features and using it to train a simple NN (or other ML model) will have good performance.


Time of trainig vs time of prediction, which one is used during classification algorithms comparison?

I get this answer, if it may helping you or if you have any other information to add :

If we are talking only about computational time then prediction time is more important if the training time is finite and accuracy is comparable. Training time can be handled easily by using a good machine.

This is important because any machine learning system is expected to generalize or at least have higher accuracy on the testing set. To achieve this better performance training time is often not considered. If one algorithm has significantly higher training time but good accuracy on testing test then it is considered better because once the training is done, the algorithm doesn’t need much tweaking. Obviously in this case we assume that the training time is finite which can be reduced by using a better machine.


Plotting libraries for Scala on Zeppelin

Vegas is a library that strives to substitute for matplotlib in Zeppelin/Spark environments. It works on Spark DataFrames.


Reinforcement Learning different patients

Let's first begin with a primer to reinforcement learning. Reinforcement Learning is a method of learning that predicts the action to perform at a given state or predicts the value associated with a given state and action pair.

So, Neil is correct that this problem is not explicitly a reinforcement learning problem. This begs the question, how does one define a reinforcement learning problem. It is defined as a 5-tuple containing the following elements. $S$ is the set of states, which must be finite. $A$ is the set of actions available at a given state, it is also sometimes written as $A_s$. $P_a(s, s') = \Pr(s_{t+1}=s' \mid s_t = s, a_t=a)$ is the probability that action $a$ at state $s$ will lead to state $s'$. $R_a(s, s')$ is the reward received when using action $a$ to transition from state $s$ to $s'$. $\gamma$ which is the discount factor which is a value between $0, 1$ inclusive to discount rewards at certain steps.

Now, we have to frame the question in the form of a reinforcement learning problem. You need to establish a state, in your case this would be the information about the patient that you believe has a high correlation with their happiness (or is a causal factor for their emotions). Now, you have a state of your patient with these features you've engineered. Now you must decide how to frame your question appropriately. Here you have two choices, either, you can choose what their emotion is and update your weights (your model weights, or the weights you want to output) as necessary (based on the true labels) or you can use your model to choose an action (in this case it would be a class of 1-7). These approaches are labelled as $Q$ learning or $\pi$ Policy methods. These are the primary methods present in RL, one chooses the value to assign to a given state and action $Q(a, s)$ and the other takes in a state $\pi(s)$ and maps it to an action to perform. You can decide which one you'd like to use based on convenience/experience.

Once this is done, you'd have to decide on an architecture you'd like to use to update the gradients, this is totally up to you. You can do a literature review to see what's going on in this domain and use what you think is appropriate. Just as an example, let's say you'd want to use a Policy method that is updated by a vanilla CNN. You would pass in the state to your CNN in some form that would be interpretable by the CNN and then choose an action based on the class labels. Now, you would have a reward function which would choose how far you are from the true class (you can do this heuristically, like Happiness is 10 units from Sadness). The reward function would be propagated to the network and it would update the weights as required.

PERSONAL COMMENTS: I think this is completely unnecessary for the current task. In my opinion a simple vision classifier should do perfectly well. There are an abundance of datasets that are available which takes in facial features and classifies sentiment/happiness. Also, a lot of statisticians believe that numeric values for happiness/sadness is a bad metric because questions like, "How different is very happy from happy" are hard questions to answer in the context of associating a reward with them. Do take Neil's comments into consideration.


Doc2Vec, Mikolov's paper will solve your problem. Here is the paper. You can find a gensim implementationhere. While using RNN, using GLOVE or Googl Word2Vec will be always useful even if your sequences are of the same length. To solve the variable sequence problem you can pad the sentences and use bucketing or just pad to one uniform length, truncating longer sentences and padding shorter sentences.


calculating average of each session by unique itemID

Assuming you can use dplyr package

data %>% group_by(session_id) %>% summarise(avg = length(unique(item_id))/length(item_id))

You have asked many questions on basic grouping and summarise operations in R. It would be better to go through tutorials like this to learn all these tricks in one go.


As you are working on image classification and would also like to implement some data augmentation, you can combine the two AND load the batches directly from a folder using the mighty 'ImageDataGenerator` class.
Have a look at the execellent documentation!
I won't copy and paste the example from that link, but I can outline the steps that you go through:

create the generator class:   data_gen = ImageDataGenerator()

If you want it to perform on-the-fly augmentation for you, that can be    specified when creating the class: data_gen = ImageDataGenerator(samplewise_center=True, ...)

If you use augmentation processes that require some statistics about the dataset, e.g. feature_wise normalisation (not sample-wise), you must prepare the generator by showing it some of your data: data_gen.fit(training_data). This fit method simply precomputes things like the mean and standard deviation, which is later used for normalisation.

the generator goes into the model's fit_generator method, and we call the flow_from_directory method of the generator:
model.fit_generator(training_data=data_gen.flow_from_directory('/path/to/folder/'), ...)

You can also create a separate generator using ImageDataGenerator for your validation data, where you should then not apply the augmentation, so that validation tests are done on real data, giving you and accurate picture of the model's performance.
In any case, these generators will theoretically run forever, generating batches from your folder. Therefore I recommend using a callback function from Keras to stop when a certain criteria is met. See the documentation for the EarlyStopping class. You could also do this manually, but Keras makes it very simple!
If you want fine-grained control, you could do all of the above manually, loading enough samples from disk for a single batch, performing some augmentation and then running the model.train_on_batch() method. If you want to get into the details, you might be best first learning the Keras way, then progress to your own detailed models combing Tensorflow as required. The two can be used toegether very nicely!


In addition to what Lupacante conceptually and nicely showed such that the added feature(s) has(have) to be informative for the model otherwise it can get ignored by majority of models (perhaps easily by regularized models), I would like add that you also increase the dimensionally of the feature space synthetically using many simple mathematical expressions as well. Concertedly, let's say your only feature (column) is:

x

You may easily construct other features like (it is common practice actually in physical sciences):

x$^2$, x$^3$, x$^{0.5}$, $sin(x)$, x$^2$sin(x),...

Till you hit the so-called the curse of dimensionality for your exercise. Although I am not yet sure what features/num_samples ratio exactly causes the curse of dimensionality! What I have gathered so far about the curse of dimensionality has been very subjective.


How to solve the gradient descent on a linear classification problem?

It appears to me that your gradient calculation is off:

$$  \frac{\partial E_{in}(a,b)}{\partial{b}} = \frac{1}{|D|} \sum_i \frac{\partial e(y_i,h_{a,b}(x_i))}{\partial b} $$

Where (with some lenient notation) :

$$  \frac{\partial e(y_i,h_{a,b}(x_i))}{\partial b} = \frac{\partial e(y,s)}{\partial{s}} * \frac{\partial h_{a,b}(x_i)}{\partial{b}} $$

We can show that :

$$ \frac{\partial e(y,s)}{\partial{s}} = -y $$ if $$1-y.s >0$$ and, otherwise: $$ \frac{\partial e(y,s)}{\partial{s}} = 0 $$

This give you an expected dependency on x, as the value change the criterion.

And :

$$ \frac{\partial h_{a,b}(x_i)}{\partial{b}} = -a $$

So you have :

$$  \frac{\partial E_{in}(a,b)}{\partial{b}} = \frac{1}{|D|} \sum_i   \mathbb{1}_{y.x<1}*y*a $$


Scikit Learn OneHotEncoded Features causing error in classifier

scores = cross_val_score(mnb, Y, chk, cv=10, scoring='accuracy')

You have your Y and chk switched. That's it. :)

The signature of cross_val_score is sklearn.cross_validation.cross_val_score(estimator, X, y).

X is a matrix and y is a 1D vector with your class labels.

unlike in R, most (or all?) sklearn models do not support categorical variables. Most of the time, encoding your feature matrix X into what is called one-hot encoding is good enough.

Notice that, in some models, this hack is not the same as true native categorical support, and the performance of the model will be worse.

Invert One-Hot Encoding

Sklearn does not seem to have an easy method to invert the one-hot encoding.

It is not trivial how to do this. I found this suggestion:

def inverse(enc, out, shape):
    return np.array([enc.active_features_[col] for col in out.sorted_indices().indices]).reshape(shape) - enc.feature_indices_[:-1]

Example:

import numpy as np
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder()
X = np.array([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]]))
Z = enc.fit_transform(X)
print(inverse(enc, Z, X.shape))
# [[0 0 3]
#  [1 1 0]
#  [0 2 1]
#  [1 0 2]]
print(X)
# [[0 0 3]
#  [1 1 0]
#  [0 2 1]
#  [1 0 2]]

Notice:

This only works when HotOneEncoding(sparse=True) (default) because it uses scipy sparse matrix methods (this could be changed by making the code only use numpy methods), but this is probably what you want since working with a dense matrix will kill your memory anyhow
I think this will only work if your variables are within the range [0,something] because you lose that information in the transformation (no work-around for this other than you using something like DictVectorizer which offers you more control over the transformation.


How to add previous predictions for new predictions in LSTM?

For your problem, Keras is telling you that the input vectorhas 4 dims, which is invalid, because for a LSTM model you need a input vector with (batch_size,num_time_steps,feature_space_dimension). Check the shape of the input vector, you may be adding an extra 1 dimension with the reshape operator

Other thing is the approach your are taking, which may be valid, but usually this problem is tackle with a specific type of models: Sequence-to-Sequence models, using LSTM encoder-decoder architecture.

This models try to predict a sequence

$Y = [x_{(t=1)}, x_{(t=2)},...,x_{(t=t+N)}]$

from a sequence of previous values

$X = [x_{(t=0)}, x_{(t=-1)},...,x_{(t=t-M)}]$

Note that the number of samples in each sequence can be different

In these models, the input sequence $X$  is fed into a LSTM encoder, which captures its dynamics creating a new context feature vector $F$. This new feature vector are then fed to the LSTM decoder as initial state, and then, the prediction $\hat y(t)$ of cell for time $t$ is used as input for the cell for time $t+1$ to generate the predicted value $\hat y(t+1)$

One key difference is, that Seq2seq models should be more efficient than your approach, since it performs the prediction of the N future values with a single run of the model. It could be interesting if you try this approach out and to compare it with yours

You can visit a Keras tutorial for these type of models here


With expected values you have a fair bit of freedom to expand/resolve or not.

For instance, assuming the distributions $X$ and $Y$ are independently resolved (i.e. the values are not correlated):

$$\mathbb{E}[X + Y] = (\sum_x xp(x))+ \mathbb{E}[Y]$$

$$\mathbb{E}[XY] = \sum_x xp(x)\mathbb{E}[Y]$$

Each time step of a MDP is independent in this way, so you can use this when handling sums and products within expectations in the Bellman equations (provided you separate terms by time step).

For the Bellman equation, the goal is to relate $v_\pi(s_t)$ to $v_\pi(s_{t+1})$, and the definition of value is given as an expectation, so it makes sense to preserve the second expectation rather than expand it.

Something has to change though, as within the second sum a time step is effectively taken from $s$ to $s'$, so the new expectation has to include that. It has in some sense been expanded, just not fully broken down into the full product of every following policy decision and state transition.

You could try to write out the full expansion from expected to products of sums over distributions using some container like $\Pi_{n=t+1}^{T}$ - showing how to calculate the expected value over full tree of all possibilities - and the maths would still work. But it would be a very longhand way of showing the same relationship.


How is Amazon using Deep Learning in the new Amazon Go?

" Our Just Walk Out technology automatically detects when products are taken from or returned to the shelves and keeps track of them in a virtual cart. "

Out of this sentence I understand that camera's will record everything you do. Based on these videos they will record the things you bought etc. To get the information out the video deep learning will be used.

A quick search on google about "video analysis deep neural network" will give you results like: https://arxiv.org/abs/1512.03131


Sklearn feature selection stopping criterion (SelectFromModel)

Some regression/classification models can also calculate feature importances - for example RandomForestClassifier models have a property feature_importances_ and LogisticRegression models have a coef_ property. There are many more models that can provide feature importances - but all of them either have a coef_ or feature_importances_ property.

What SelectFromModel does is to check if the object you pass as "estimator" has one of these two properties and to return features where this property is higher than the specified threshold (or the mean/median).

For example if you pass a RandomForestClassifier to SelectFromModel, it will return all features where the random forest's feature_importances_ property is higher than the specified threshold. The same happens if you pass a LogisticRegression model, except that it'll compare the coef_ property with the threshold instead.

Selecting the best value for the threshold can be done using a grid- or randomized search.


Relationship Between Independent and Dependent variables

Yes you can use neural network for any non linear function. Maybe your variables don't have a strong linear correlation but you can train it using neural network to find the best pattern possible . Although it is tough to say whether it will generalize or not


Stacking/Concatenating/Combining two vector space models

It is perfectly valid to concatenate the vectors from two different models. It will be necessary however to experiment with the two approaches, i.e. either using individual vectors from one of the models or concatenating the vectors, before you can determine which method will produce better clustering results.  It is also valid to normalize word vectors, but again requires experimentation to determine whether this approach will provide superior results.


Keras input for multivariate classification with LSTM using current features and previous timesteps features and y values

Since you want to get a classification/output per each time step, you should set return_sequences=True in you LSTM layer.

Read more about return_sequences here in LSTM documentation.


What methods to create singular content classification from inconsistent inbound info?

You could import each companies data into a specific  table and then develop regular expression Scripts two change specific Expressions into your own taxonomy.
https://en.m.wikipedia.org/wiki/Regular_expression


Is there any data set containing list of windows software and the category each one of them belongs to?

What kind of categories are you looking for? Could you give more insight into why you need such a dataset? I did a prima facie search and your question seems true. I can create the dataset based on web scraping based on your requirements.


Gender identification task on instance or user level?

This depends heavily on your intended use case:

When 1) is better

If you want to identify the gender as best as you can while always having access to a data set containing multiple documents/tweets then this is the way to go.

There are multiple features that you can only engineer from the full data set (like average length of tweets, tweet frequency, etc.) that might be very predictive.

When 2) is better

If however your use case is to predict the gender of user based on a single tweet (maybe even without knowing the id/origin of the tweet, only the tweet text) then you need to train a model that can make a prediction out of a single tweet.

In this case I would also encourage you to get a diversified data set which contains more tweets from several different users instead of a large class of tweets from few users.


How does a Decision Tree handle an unknown category in a split?

There are two perspectives to this question, from a mathematical or machine learning perspective and from a technical perspective. From a technical perspective this depends on the implementation of the decision tree. An unseen value is not that different from a missing value, and sklearn for example does not deal with unknown values well and will fail on unseen or unknown values. Other tree implementations might deal better with this.

From a mathematical or ML perspective, without making some assumptions you cannot solve this problem of course. However an assumption you could make is that unknown or unseen values are average values. With that assumption you could follow both paths on the split where the unknown value occurs. Then you collect both results from the split and weigh them by the number of training samples that happened there. That way you can still make a prediction for these unknown values.


Relationship between KS, AUROC, and Gini

The Wikipedia entry for Receiver operating characteristic references this paper for the Gini=2AUROC-1 result: Hand, David J.; and Till, Robert J. (2001); A simple generalization of the area under the ROC curve for multiple class classification problems, Machine Learning, 45, 171–186.  But I'm afraid I don't have easy access to it to see how close it comes to what you want.


How to preprocess with NLP a big dataset for text classification

Let me first clarify the general principle of classification with text data. Note that I'm assuming that you're using a "traditional" method (like decision trees), as opposed to Deep Learning (DL) method.
As you correctly understand, each individual text document (instance) has to be represented as a vector of features, each feature representing a word. But there is a crucial constraint: every feature/word must be at the same position in the vector for all the documents. This is because that's how the learning algorithm can find patterns across instances. For example the decision tree algorithm might create a condition corresponding to "does the document contains the word 'cat'?", and the only way for the model to correctly detect if this condition is satisfied is if the word 'cat' is consistently represented at index $i$ in the vector for every instance.
For the record this is very similar to one-hot-encoding: the variable "word" has many possible values, each of them must be represented as a different feature.
This means that you cannot use a different index representation for every instance, as you currently do.

Vectors generated from those texts needs to have the same dimension Does padding them with zeroes make any sense?

As you probably understood now, no it doesn't.

Vectors for prediction needs also to have the same dimension as those from the training

Yes, they must not only have the same dimension but also have the same exact features/words in the same order.

At prediction phase, those words that hasn't been added to the corpus are ignored

Absolutely, any out of vocabulary word (word which doesn't appear in the training data) has to be ignored. It would be unusable anyway since the model has no idea which class it is related to.

Also, the vectorization doesn't make much sense since they are like [0, 1, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3] and this is different to [1, 0, 2, 3, 4, 1, 2, 3, 5, 1, 2, 3] even though they both contain the same information

Indeed, you had the right intuition that there was a problem there, it's the same issue as above.
Now of course you go back to solving the problem of fitting these very long vectors in memory. So in theory the vector length is the full vocabulary size, but in practice there are several good reasons not to keep all the words, more precisely to remove the least frequent words:

The least frequent words are difficult to use by the model. A word which appears only once (btw it's called a hapax legomenon, in case you want to impress people with fancy terms ;) ) doesn't help at all, because it might appear by chance with a particular class. Worse, it can cause overfitting: if the model creates a rule that classifies any document containing this word as class C (because in the training 100% of the documents with this word are class C, even though there's only one) and it turns out that the word has nothing specific to class C, the model will make errors. Statistically it's very risky to draw conclusions from a small sample, so the least frequent words are often "bad features".

You're going to like this one: texts in natural language follow a Zipf distribution. This means that in any text there's a small number of distinct words which appear frequently and a high number of distinct words which appear rarely. As a result removing the least frequent words reduces the size of the vocabulary very quickly (because there are many rare words) but it doesn't remove a large proportion of the text (because the most frequent occurrences are frequent words). For example removing the words which appear only once might reduce the vocabulary size by half, while reducing the text size by only 3%.

So practically what you need to do is this:

Calculate the word frequency for every distinct word across all the documents in the training data (only in the training data). Note that you need to store only one dict in memory so it's doable. Sort it by frequency and store it somewhere in a file.
Decide a minimum frequency $N$ in order to obtain your reduced vocabulary by removing all the words which have frequency lower than $N$.
Represent every document as a vector using only this predefined vocabulary (and fixed indexes, of course). Now you can train a model and evaluate it on a test set.

Note that you could try different values of $N$ (2,3,4,...) and observe which one gives the best performance (it's not necessarily the lowest one, for the reasons mentioned above). If you do that you should normally use a validation set distinct from the final test set, because evaluating several times on the test set is like "cheating" (this is called data leakage).


Using tensorflow for any type of dataset

TensorFlow is a general purpose library for numerical computation using data flow graphs. It is primarily used for neural networks but can be used for any mathematical operations on multidimensional data arrays (tensors).

Thus, TensorFlow can be used to estimate binary logistic regression with explanatory categorical variables. An example can be found in the TensorFlow tutorials here.


When you want to use Auto-Encoders (AEs) for dimensionality reduction, you usally add a bottleneck layer. This means, for example, you have 1234-dimensional data. You feed this into your AE, and - as it is an AE - you have an output of dimension 1234. However, you might have many layers in that network and one of them has significantly less dimensions. Lets say you have the topology 1234:1024:784:1024:1234. You train it like this, but you only use the weights from the 1234:1024:784 part.

When you get new input, you just feed it into this network. You can see it as a kind of preprocessing. For the later stages, this is a black box.

This is manly useful when you have a lot of unlabeled data. It is called Semi Supervised Learning (SSL).


Clearest way to visualise temporal data?

How about using a boxplot, so you can also visualize the variance to some degree. Each boxplot could represent one weekday.

When you look only at the mean (and be at the busstop at this time) you have a good chance to miss the bus (about 50% for a well-shaped non-skewed distribution).

You want to make sure that you be at the stop at the right time (95% chance of getting the bus or so). Therefore variance is important.

The image is from this page: https://www.datamentor.io/r-programming/box-plot/

Alternatively, you could train a model on the arrival time, make predictions for each day, and plot the predicted value including a 95% confidence interval. But this probably is too much here.


Modelling promotions for demand forecasting

To create dummy variables for days promotion holidays, you might find this example useful.

If you are sure you do not have an out-of-stock problem anytime in the history you could use an autoregressive model to predict future sales (demand) for any product that has historical data. Depending on your data you could choose a model. Following code is an example which combines four different models by giving different weight to different models. This type of models capture seasonality and trends of your data. For more details about the models please check Rob Hyndman's forecast package documentation.

choose_model<-function(x,h,reg,new_reg,end_train,start_test){
      library(forecast)
      library(tidyverse)

  #train data

  x_train <- window(x, end = end_train )

  x_test <- window(x, start = start_test)

  #train and test for regressors

  reg_train <- window(reg, end = end_train )

  reg_test <- window(reg, start = start_test)

  h1=length(x_test)

  #model1

  stlf(x_train , method="arima",s.window= nrow(x_train),xreg = reg_train, newxreg = reg_test, h=h1)-> fc_stlf_xreg

  #model2
  auto.arima(x_train, stepwise = FALSE, approximation = FALSE,xreg=reg_train)%>%forecast(h=h1,xreg=reg_test) -> fc_arima_xreg

  #model3
  set.seed(12345)#for nnetar model
  nnetar(x_train, MaxNWts=nrow(x), xreg=reg_train)%>%forecast(h=h1, xreg=reg_test) -> fc_nnetar_xreg

  #model4
  stlf(x_train , method= "ets",s.window= 12, h=h1)-> fc_stlf_ets

  #Combination

  mod1 <- lm(x_test ~ 0 + fc_stlf_xreg$mean + fc_arima_xreg$mean + fc_nnetar_xreg$mean + fc_stlf_ets$mean)
  mod2 <- lm(x_test/I(sum(coef(mod1))) ~ 0 + fc_stlf_xreg$mean + fc_arima_xreg$mean + fc_nnetar_xreg$mean + fc_stlf_ets$mean)

  #model1

  stlf(x, method="arima",s.window= 12,xreg=reg, newxreg=new_reg, h=h)-> fc_stlf

  #model2
  auto.arima(x, stepwise = FALSE, approximation = FALSE,xreg=reg)%>%forecast(h=h,xreg=new_reg) -> fc_arima

  #model3
  set.seed(12345)#for nnetar model
  nnetar(x, MaxNWts=nrow(x), xreg=reg)%>%forecast(h=h, xreg=new_reg) -> fc_nnetar

  #model4
  stlf(x , method= "ets",s.window= 12, h=h)-> fc_stlf_e

  #Combination

  Combi <- (mod2$coefficients[[1]]*fc_stlf$mean + mod2$coefficients[[2]]*fc_arima$mean +
              mod2$coefficients[[3]]*fc_nnetar$mean + mod2$coefficients[[4]]*fc_stlf_e$mean)

  return(Combi)
}

The usage of the function:

coose_model(x,h,reg,new_reg,c(2018,02),c(2018,3))

$x$ is a time series

$h$ is time horizon to predict

$reg$ is the historical promotions, dummy date variables, holidays...

$new_ reg$ is the promotions, dummy date variables, holidays that are that you know it is going to happen

If you know that there is out-of-stock problem then take a look to this paper.


breaking joined words into meaningful ones during text mining

The compound split is not trivial, but there are solutions that kind of work (as you can read on the link @Aditya shared).

Another way of dealing with the matter would be to tokenize the strings into character n-grams.


Why training and validation similar loss curves lead to poor performance

to prevent overfitting in a model the training curve in a loss graph
  should be similar to the validation curve.

That's not always the case, maybe the train set has slightly a different distribution from the validation set, therefore, our focus is just when training loss decreases while validation loss slightly or never decreases.

In the former graph, you notice there was a slight increase in the validation error while the training error was decreasing, while the latter graph has the same behavior due to a less complex model, the model was not able to fit the training data well.


How to test trained PCA used for compression?

Found the answer in a different stackexchange question here: https://stats.stackexchange.com/questions/142216/zero-centering-the-testing-set-after-pca-on-the-training-set

Answer: Yes, zero center the test data with the mean from the train data


Do not apply PCA to categorical data

PCA attempts to find the dimensions which contains the most variance in a dataset. When you have categorical variables, the distance between points and the variance captured by a variable are ill-defined. First of all, you have no proper distance measure to tell how far apart two categories are, and secondly, the ordering of your categories will have a big impact. Ordering months categorically will result in a small distance between January and February, but a large one between December and January. If you're categorizing things like broadcaster or channel, there is no natural order to put things in, so your distance measure will be essentially meaningless. Your PCA plot looks strange because you have applied it to categorical data - this is not a meaningful representation of your data.


Non Deterministc Dimensionality reduction

Autoencoders are non-deterministic, since they rely on a random weight initialization.


How to define a neural model as linear or non linear

You are right, for the model to be non linear, the activation function must be non linear.

If the activation function is for example the identity function, your model won't be non linear, even if you stack multiple hidden layers. Indeed, the output of the neurons will only just be a linear composition of the inputs combined with weights.

I think the author says this neuron model is non-linear because in practice a linear activation function is almost never used. Some functions look like linear functions, but in fact these are not. Here is a demonstration for the ReLU activation function, which look like a linear function but is mathematically not : https://datascience.stackexchange.com/a/26481/73209

I guess it is the same for the Piecewise Linear Unit (PLU) activation function.


Are linear regression models with non linear basis functions used in practice?

Yep, that's a thing. It's called a "Generalized additive model (GAM)":

https://en.wikipedia.org/wiki/Generalized_additive_model

You may also be interested in "multivariate adaptive regression splines (MARS)":

https://en.wikipedia.org/wiki/Multivariate_adaptive_regression_splines

EDIT: Regarding demonstrating that these are actually used, I'm not sure what you're looking for. I've never seen a survey to try and gauge the popularity of specific models, and I'm not sure how useful such a thing would be. I could speak from my subjective experience, but I don't consider that particularly meaningful either.

If you just want examples:

Hits for GAM models on PubMed - 2114 hits
Hits for MARS models on Pubmed - 190 hits

For context:

Hits for Random Forest models on PubMed - 5604
Hits for Deep Learning on PubMed - 4435 hits
Hits for Neural Network on PubMed - 57170 hits

Take these counts with a solid dose of salt: in my experience random forests are in much wider use in practice than neural networks. But like I said earlier, you shouldn't really trust my subjective experience either since that's just anecdotal.


A trick used in Rademacher complexity related Theorem

This requires hell of a derivation, but I liked the question :)

  My question is, why can we swap $z_i$ and $z'_i$?

The key insight is that notation $S \sim  \mathcal{D}^m$ is equivalent to $Z_1 \sim  \mathcal{D}, \cdots, Z_m \sim  \mathcal{D}$. This translates to
$$E_{S,S'}[.]=E_{Z_1,\cdots,Z_m,Z'_1,\cdots,Z'_m}[.]$$
which remains the same by reordering the $Z$s.

Therefore the swap argument can be shown by

Singling out an arbitrary term $k$ from $\sum_{i=1}^{m}$,
Switching $E_{Z_k,Z'_k}$ to $E_{Z'_k,Z_k}$, and
Renaming the variables.

That is,
$$\begin{align*}
&E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} g(z'_i)-g(z_i)\right] \\
&\overset{\text{expand}}{=}
E_{\color{blue}{Z_1,\cdots,Z_m,Z'_1,\cdots,Z'_m}}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} g(z'_i)-g(z_i)\right]\\
&\overset{\text{separate}}{=}
E_{\cdots,Z_k,\cdots,Z'_k,\cdots}\left[\underset{g \in \mathcal{G}}{\text{sup}} \frac{1}{m}\left(\color{blue}{g(z'_k)-g(z_k)}+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right] \\
&\overset{\text{switch}}{=}
E_{\cdots,\color{blue}{Z'_k},\cdots,\color{blue}{Z_k},\cdots}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\left(g(z'_k)-g(z_k)+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right] \\
&\overset{\text{reorder}}{=}
E_{\cdots,Z'_k,\cdots,Z_k,\cdots}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\left(\color{blue}{-(g(z_k)-g(z'_k))}+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right] (*)
\end{align*}$$
Now by renaming $Z_k \leftrightarrow  Z'_k$, and thus $z_k \leftrightarrow z'_k$ in $(*)$ we have:
$$\begin{align*}
&E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} g(z'_i)-g(z_i)\right] \\
&\overset{\text{rename}}{=}  E_{\cdots,\color{blue}{Z_k},\cdots,\color{blue}{Z'_k},\cdots}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\left(-(g(\color{blue}{z'_k})-g(\color{blue}{z_k}))+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right]\\
&\overset{\text{collapse}}{=} E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\left(-(g(z'_k)-g(z_k))+\sum_{i=1; \neq k}^{m} g(z'_i)-g(z_i)\right)\right]\\
\end{align*}$$

which corresponds to $\boldsymbol{\sigma}=(\sigma_1=1,\cdots,\sigma_k = -1,\cdots,\sigma_m=1)$.

We have proved this equality by switching the sign of arbitrary index $k$. Therefore, by defining
$$f(\boldsymbol{\sigma}):=E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} \sigma_i(g(z'_i)-g(z_i))\right]$$
we have shown that
$$\forall \boldsymbol{\sigma_1}, \boldsymbol{\sigma_2},f(\boldsymbol{\sigma_1})=f(\boldsymbol{\sigma_2}).$$
For example:
$$\begin{align*}
f(\boldsymbol{1})&=f(\sigma_1=1,\cdots,\sigma_k=1, \cdots,\sigma_m=1)\\
&=f(\sigma_1=1,\cdots,\sigma_k=-1, \cdots,\sigma_m=1)
\end{align*}$$
Knowing that there is $2^m$ equi-probable vectors $\boldsymbol{\sigma_i}$, we finally have
$$\begin{align*}
E_{S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m}  g(z'_i)-g(z_i)\right]&=f(\boldsymbol{1}) \\
&=\frac{1}{2^m} f(\boldsymbol{\sigma_1})+\cdots+\frac{1}{2^m}f(\boldsymbol{\sigma_{2^m}})\\
&=E_{\boldsymbol{\sigma}}[f(\boldsymbol{\sigma})] \\
&=E_{\boldsymbol{\sigma},S,S'}\left[\underset{g \in \mathcal{G}}{\text{sup}}\frac{1}{m}\sum_{i=1}^{m} \sigma_i(g(z'_i)-g(z_i))\right]
\end{align*}$$
$\square$


Implication of a dominant Principal Component in PCA analysis

The principal components describe the amount of the total variance that can be explained by a single dimension of the data.

This is equivalent to the spread of the datapoints in a given dimension. The dimensions are (of course) direction that are orthogonal i.e. at 90 degrees to one another.

Have a look at this example of data points, where the red lines show the breasth of the data in two dimensions:

The dimensions don't have to be in X and Y - they could be pointing in any direction, but must be orthogonal. (more detail in this answer)

We can clearly see a bigger spread in the horizontal X-dimension, so I might expect it to account for 80% of the variance in the dataset. The vertical Y-dimension has less variance, less spread, so explains a smaller amount of the total variance. In this simple 2d example, it would explain the remaining 20% of the variance (it must sum to 100%).

In practical terms, if principal components have all very similar values, you might expect the data to form a circle (in 2d), and this means there is little directionality in the feature-space. You might like to think in terms of correlation between the features; a movement in one direction of the space does not guarantee a movement of a certain direction in the second feature. The opposite would be true if e.g. the first component had a normalised value of ~1 i.e. explained approximately 100% of the variance.

I say normalised, because the raw values that come out of PCA do not necessarily between 0 and 1 - so you can normalise them to help interpretation.

In a higher dimensional space, say with 10 variables (so 10d feature space), PCA computes eigenvectors and eigenvalues, which look for orthogonal dimensions that explain the variance of the data points, but these are not all all constricted to the dimensions of your features themselves! This means that you cannot just say that the first component (e.g. with a value of 0.6) is there because of a feature X, i.e. not due to a single feature, but a mixture of the features.

About the dataset

If the first PC explains nearly all of the data's variance, it is likely that you can express you data a lot more succinctly. PCA is often used as a dimensionality reduction method - so in this extreme case, you can reduce a high-dimensional space and convert it to a lower dimensional space without (theoretically) losing much of the explanatory power i.e. a model should be able to learn as much about the feature space from the one predominant PC, compared to the rich high-dimensional space. This of course saves compute time for any model.

I said *theoretically) above, because using PCs to model will make the interpretation of the resulting model itself - in terms of the original input features - more difficult.

For a more thorough introduction, read through a blog post like this one.
For some practical examples, check out the Sci-kit learn documentation.


Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)

Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes or labels are soft probabilities (like [0.5, 0.3, 0.2]).

Formula for categorical crossentropy (S - samples, C - classess, $s \in c $ - sample belongs to class c) is:

$$ -\frac{1}{N} \sum_{s\in S} \sum_{c \in C} 1_{s\in c} log {p(s \in c)} $$

For case when classes are exclusive, you don't need to sum over them - for each sample only non-zero value is just $-log p(s \in c)$ for true class c.

This allows to conserve time and memory. Consider case of 10000 classes when they are mutually exclusive - just 1 log instead of summing up 10000 for each sample, just one integer instead of 10000 floats.

Formula is the same in both cases, so no impact on accuracy should be there.


Semantic Annotation in text with curlie.org

So, I'm still not sure about what you consider to be an annotation on curlie.org. What I understood, you'll correct me if I'm wrong, is that you would like to annotate some text, more specifically you would like to annotate/identify concepts, and you would like to follow the structure on curlie.

From what I see, curlie doesn't contain any annotation, it's more like a website ontology, which has at the bottom real url and as hypernyms domain-specific topics and then at the top level broad topics. So the only thing you might consider an annotation, at document level, i.e. a single label for a whole website, is the domain-specific topic.

If you want to create an ontology similar to curlie, Protege could be a good choice. Also, if you're interested in retrieving some general taxonomy information for a word, you probably want to look at some lexical databased like WordNet or ConceptNet.

If this is not what you want and you're interested in making your own annotations at word level, e.g. "I bought an iPhone[object]" then probably the taxonomy of curlie is too generic and you will definitely need to narrow down the scope of your annotations. Take a look at some literature of NLP tasks like Named entity recognition, it's plenty of papers. Also, if you will go down this road (creating your own annotations) a good tool is Prodigy. Consider that even at word level, annotations can defined at several levels for each document. For example, in opinion mining usually people are interested in annotating not only a chunk of words representing an opinion, but also other aspects like the target of the opinion, e.g. "[This [GPS]$_{target}$ really sucks]$_{opinion}$, it never works!".
But in my experience people stop at max 3 layers of annotations, and it truly depends on the purpose for which the annotations are made in the first place.


Match an image from a set of images : Combine traditional Computer vision + Deep Learning/CNN

In my opinion, if you want a hybrid of Convolutional Neural Networks and the classic feature extraction techniques that would be redundant. Mainly because the architecture of a Convolutional Neural Network is composed of convolutions. I won't go much into detail of the whole architecture but these convolutions actually extract the good features for you and then those convolutions are connected to a classic Neural Network that does the classification task. Hence, extracting features using SIFT and using Convolutional Neural Networks would be redundant. In addition, the features that CNN will extract are better as compared to simple SIFT features. Though if you want to push through with combining SIFT and Deep Learning, you can instead substitute convolutions and use SIFT for feature extraction and then feed it into a Neural Network. That would work as well.


Does CRNN use sparse tensor value for its label?

My assumption here is that if the input vectors are of variable length, you should use something like keras.preprocessing.sequence.pad_sequences  (as well as a masking layer to ignore these padded values) as a means of reformatting all input samples to the same size so that the input to your recurrent model meets the requirement of having a fixed num_timesteps length.
Function Documentation:

Pads sequences to the same length.
This function transforms a list of num_samples sequences (lists of
integers) into a 2D Numpy array of shape (num_samples, num_timesteps).
num_timesteps is either the maxlen argument if provided, or the length
of the longest sequence otherwise.
Sequences that are shorter than num_timesteps are padded with value at
the end.
Sequences longer than num_timesteps are truncated so that they fit the
desired length. The position where padding or truncation happens is
determined by the arguments padding and truncating, respectively.


how to explain the behaviour: linear svm does better than non-linear RBF

The quality of your features might actually be better than you think. If they provide linear separability, nonlinear kernels will overfit more readily than a linear kernel, leading to your result.


To which category does this algorithm belongs?

Ensemble Methods as defined in Wikipedia:

  In statistics and machine learning, ensemble methods use multiple
  learning algorithms to obtain better predictive performance than could
  be obtained from any of the constituent learning algorithms alone.

All those methods you mentioned are tree-based ensemble models:

Bagging (Breiman, 1996): Fit many large trees to bootstrap-resampled versions of the training data, and classify by majority vote.
Random Forests (Breiman 1999): Fancier version of bagging (only a subset of features are selected at random, unlike bagging where all features are considered for splitting a node).
Boosting (Freund & Shapire, 1996): Fit many large or small trees to reweighted versions of the training data. Classify by weighted majority vote. There is a nice article explaining gradient boosting trees.

In general (in terms of prediction capability, boosting to be the best):

Boosting > Random Forests > Bagging > Single Tree

You might be wondering where AdaBoost fits?

Adaptive Boosting (or in short AdaBoost, was the first really successful boosting algorithm) works on improving the base learner esp. where it fails on predictions. Please note that the base learner can be any machine learning algorithms upon which the boosting is applied to obtain a strong learner. When Decision stumps are used as base learner, AdaBoost is comparable to the above-mentioned boosting trees. You might be asking, again, what are their differences, see below (taken from this book):

Modern Boosting Trees

Due to the success of gradient boosting trees, there are types of boosting algorithms, namely: Gradient Boosting, XGBoost, and Catboost. They are conceptually very similar, yet they differ in e.g. sampling methods, regularizations, handling categorical features, performance etc. Strongly recommend checking this article out if interested to learn more.

Personal Note: About 1.5 year ago I was a fan of XGBoost (for many reasons), till I experimented Catboost. Now I really like Catboost. First of all, it easily handles a mixture of numerical and categorical features EVEN without coding the categorical ones. And the default hyperparameters give comparable results to fine-tuned hyperparameters in XGBoost, thus less hassle. At present Catboost community is smaller than let's say XGboost, kind of makes it less attractive, but it is growing. Last note: I am not affiliated to any of these methods/implementations.

Hope, it is now better! ;-)


How to suggest medical treatments based on injury subjective and objective notes using ML?

There is a lot of research being done around exploiting biomedical text data in general and clinical notes in particular. I'm not up to date with the whole domain (it's a big one) but let me sketch a few possible directions.

The standard text classification approach: consider every possible treatment as a class, the goal being to predict for each instance of subjective+objective notes the correct class(es). Note that the design can either be standard multiclass classification or multi-label classification. All the regular text classification techniques can be used, from traditional methods like decision trees to using DL with word/text embeddings.
The sequence to sequence approach: the principle here is for the model to represent how an input sequence (here subjective+objective notes) is transformed into an output sequence (treatment). The standard example for this kind of task is machine translation, but this design is used in many other problems.
The semantic representation with a third-party ontology. This approach relies on existing biomedical resources, especially normalized vocabularies like Mesh and/or UMLS. One would use an automatic annotation tool (e.g. cTakes) to extract the medical terms from the subjective+objective notes, and the treatments could also be encoded with the controlled vocabulary. This can greatly facilitate the job of the model by providing it with normalized features and classes. Once encoded, the data can be used in a classification setting similar to the first approach.

About software/data resources: there are a lot, but as far as I know nothing which does exactly what you need out of the box. SciSpacy is a Python module for processing biomedical data. There are datasets available (e.g. shared tasks data), and various research prototypes and resources. See also this answer for pointers to various biomedical resources.


Deploy pretrained model to simple web app

Flask: https://flask.palletsprojects.com/en/1.1.x/

It's a Python framework for dynamic web resources. Should be reasonably easy to load up the estimator and create an endpoint that you can post an image to and get the output you want.


How to deploy a deep learning model in Flask (Python)?

You need to write a simple web application with one endpoint. I've checked the mmf doc you can load a model using it's keys:

from flask import Flask, jsonify, request
from mmf.common.registry import registry

model_cls = registry.get_model_class("<DESIRED_MODEL_KEY">)
model = model_cls.from_pretrained("<DESIRED_PRETRAINED_ZOO_KEY>")

@app.route('/predict', methods=['POST'])
def predict():
    data = request.get_data()
    prediction = model(data)
    return jsonify(list(prediction))

app.run(host='0.0.0.0', port=8000)

I haven't ran that code but I know for a fact that this won't run out of the box. data is a dictionary and you need to get the right key and convert the value
to a numpy array or Torch tensor. And predict will be a numpy array (or Torch tensor - but can be converted back to numpy) and you will need to convert everything back to a standard python format (because numpy or torch data types are not json serialisable).


When you do model.predict(X) the first axis in X is always an index in a batch. So if you want to predict on one sample, do something like X = np.expand_dims(X, axis=0)

In your case, this should work:

old_qval = model.predict( np.expand_dims(old_state_m, axis=0) )


Computing Image Similarity based on Color Distribution

Try using Earth Mover's Distance. It measures how much it "costs" to optimally reshape one histogram into another, where an elementary transform (using the terminology of edit distance-like distance measures) is moving a unit of mass from a bin to a bin. If you have an implementation of EMD, then your distance is dist = EMD(H1, H2, D), where
 H1 = [ 14, 13, 40, 31 ]; -- histogram #1 (first image)
 H2 = [ 82, 8, 7 ]; -- histogram #2 (second image)
 D  = [ D11, D12, D13; ... ; D41, D42, D43 ]; -- (cross-bin) ground distance

Dij is a distance from the i'th bin of the first histogram to the j'th bin of the second histogram. For example, D21 is a distance between 'White' and 'Pink'. It may be defined differently, based on which color space you want to use, and further, based on how you define what it means for two colors to be similar. If I use RGB, and I do not care much as to how human eye perceives colors, then D21 can be the \ell_2 norm of the difference [1, 1, 1] - [1, 0.05, 0.7]. (As far as I remember, the paper I mentioned uses LAB instead of RGB.)
P.S. Besides color, you may include spatial information into comparison. EMD can handle it well, as long as you properly combine similarity of colors with spatial similarity in the definition of ground distance D.


Fittting histograms with combination of an unknown number of Gaussians

You're almost done with your results.
Try to substitute the AIM for the Likelihood of the sum of Gaussian with your data, and as soon as your Likelihood value stabilizes and doesn't increase with the number of Gaussians you've found the right value for your N.

You can even try to look for the EM algorithm for the estimation of the Mixture of gaussians that fit your data.

The value of L = f(N) increases as a log function and the so called "knee" is where your optimal value of N lies on!


What is Pruning & Truncation in Decision Trees?

Your understanding is correct. xgboost has nice explanation in the docs.

Reading the original papers is always great idea. Here's one for LGBM and here's one for XGBoost.

As for dessert, here's catboost paper.

Boosting is by far one of the most important concepts in hard machine learning. It's good to know it by heart.


Any efficient way to build non-linear regression model for polynomial features?

You are probably finding yourself in one of the most interesting problems in data science, the part which is more art than science.

I will give you some ideas that can give you hints on how to solve this problem:

Prices, Salaries and other variables who have information on "accumulables" many times have a distribution which is skewed to the left (many individuals have a little, a few have a lot), what is reccomended to do is to take logarithm of it. Your new variable should be $Ln(Y)$, with this, you will close the gap between areas with greater avg_price and areas with lower avg_price. When that happens you find a less-skewed-normal-like distribution of your $Y$ variable.
The idea of taking Logarithm also applies to the $X$ variables you have (because crimes also accumulate in certain areas).
The Standard Scalling is not necessary when you run a linear regression, because the relativeness of the variable is not influencial in the regression:

The regression $Y = \alpha_0 + \alpha_1X_1+...+\alpha_nX_n$ (no
    scalled) is mathematically equivalent to $Y = \beta_0 +
\beta_1Z_1+...+\beta_nZ_n$ (scalled)

If you want to use other models, you data seems suitable for it, maybe a regression tree or XGBoost might work well on your problem.

I would bet that getting Logarithm in the avg_price, in some exogenous variables and not scalling would get better results for you.

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import pandas as pd

regDF = pd.read_csv('exampleDF')

X_feats = regDF.drop(['Avg_Price_2012'], axis=1)
y_label = regDF['Avg_Price_2012'].values

X = log(X_feats)
y = log(y_label.reshape(-1,1)).flatten()
regModel = LinearRegression()
regModel.fit(X, y)
regModel.coef_


GPU shows 0 utilization even when tensors and model are mounted on the gpu?

The problem was due to some modules I was importing without ensuring the tensors are mounted on cuda through that part of the pipeline. This had turned into the bottleneck blocking the GPU utilization.


Feature extraction of accelerometer data for machine learning

Actually you could treat your acceloremeter signal like normal audio signals. There are endless possibilities for processing audio data (e.g. chroma features).

Another way would be to directly process your raw signal with the help of a neural network (1D-convolution or LSTM).


A decision tree will automatically select the attributes that are predictive of the targets. The automatic learning of meaningful features is one the primary advantages of machine learning in general and decision tree in particular.


Strategy for dealing with giant sample size

My link in the comment has useful advice. I'd like to emphasis:

This is a very well known fact in statistics
Large sample size is good. There's nothing wrong with more and better quality data.
It's silly to split up the data in your second point. You are voluntarily giving away information and statistical power. People do that for training an unbiased model, but not to reduce sample size for the p-values.
There's no reason to switch to another model. The standard error in your logistic model should be small, which makes it ideal for predictive modelling. Your model should be robust and stable.
You should do your analysis on effects size if you have a large sample. Read @Sympa's answer in the link I post. What's your magnitude of the effect size? What's the relative effects?
The threshold you have for your first point is domain specific. If you write a paper, you should know it very well. You could compare your data with the literature. Simple descriptive statistics like percentage and how it compare with the standard deviation (which is independent to sample size) would be useful.

Please also take a look at https://stats.stackexchange.com/questions/125750/sample-size-too-large.

There is a long but excellent article on Bayes factor and p-value:

  https://replicationindex.wordpress.com/2015/04/30/replacing-p-values-with-bayes-factors-a-miracle-cure-for-the-replicability-crisis-in-psychological-science/

I should draw to your attention the following paragraph:

  The more interesting argument against p-value is not that significant results in large studies are type-I errors, but that these results are practically meaningless. To make this point, statistics books often distinguish statistical significance and practical significance and warn that statistically significant results in large samples may have little practical significance. This warning was useful in the past when researchers would only report p-values (e.g., women have higher verbal intelligence than men, p < .05). The p-value says nothing about the size of the effect. When only the p-value is available, it makes sense to assume that significant results in smaller samples are larger because only large effects can be significant in these samples. However, large effects can also be significant in large samples and large effects in small studies can be inflated by sampling error. Thus, the notion of practical significance is outdated and should be replaced by questions about effect sizes. Neither p-values nor Bayes-Factors provide information about the size of the effect or the practical implications of a finding.

The reviewer was correct. Neither p-value nor the bayes-factor reveals anything about the effect sizes.


What is slowing down classic DBSCAN algorithm

Write the DBSCAN code yourself.
Run the code.
Observe that your code likely will be a lot slower than the libraries code.


According to me your approach of oversampling(as mentioned in comment) is not correct.
When you apply SMOTE on each FOLD(assuming K-FOLD), it will over sampled the minority class. And you will always get the mixed response from K-FOLD, which is not the purpose of K-FOLD validation.

Let me try to explain you with an example:

Consider you have a dataset of 4 classes A, B, C and D.
Samples of each class, Class A has 10000, B has 2000, C has 1000, D has 6000.
During K Fold cross validation, sample picked up by system for training having majority of A and D classes.
Whereas for testing it has majorly B and C. Since you have used SMOTE it will over sampled the other two classes and you will get decent results in this case as well.

NOTE:It could be the one of the reason.

My suggestion:
 1. Shuffle the whole data set.
 2. Divide into training and test data.
 3. Apply SMOTE on training data.
 4. Follow other steps of Model training, without manipulating the data at this stage.
 5. Evaluate model on the actual test data.


RandomForest - Reasons for memory usage / consumption?

Both the amount of data and the number of trees in your forest will take up memory storage.

For the amount of data you have, this is obvious why it takes up storage - it’s data, so the more of it you have, the more space it takes up in memory. If your dataset is too large you may not even be able to read it all into memory - it may need to stay on the disk for training (I don’t think scikit supports this).

For the number of trees, each tree that is grown is trained on a subset of random features (so that the forest can avoid variance and avoid all the trees growing the same way). Once the forest is trained, the model object will have information on each individual tree - it’s decision path, split nodes, thresholds etc. This is information, and the more information you have, the more storage it will need. Therefore more trees = more information = more memory usage.

There are other factors that determine how much memory the model object (the trained forest) will take up; for example, max_depth which sets the maximum number of layers / levels any tree can grow to. If this is large, trees can grow deeper and therefore the model object will be bigger and therefore will need more storage. This shouldn’t be too large though, to avoid overfitting.


Understanding why ensembling only improves marginally

From what you are saying on can make up that you predict scores for each class and that the highest score is your class prediction.

A question you may ask is whether or not the class prediction is affected by the ensembling at all. For the sake of the example consider an example where Model A predicts a class always with a normalised score of 1.0 (i.e. full confidence, recommended reading). Suppose Model A is right 80% of the time. Model B however, is less certain. It places his bets on the top 5 classes that it deems likely with an equal normalised score of 0.2. Averaging the scores results in the same top predicted class as for Model A alone. You could easily check this by calculating the correlations between the predictions of the models with the ensembled predictions.


Training model on a Balanced vs Imbalanced dataset?

It's an interesting problem, but I think the answer might be disappointing:

Usually problems about maximizing recall are considered in the context of a trade-off with precision, i.e. the goal is to sacrifice some precision by predicting more positives, whether true (TP) or false (FP). Usually in a binary problem this is possible because we focus on one class of importance and consider the other as irrelevant, so its performance doesn't matter.

Here we have a binary problem where we want to maximize recall for both classes, so we cannot sacrifice one for the sake of the other: any gain in recall for a class is likely to cause a loss in recall for the other class, since predicting more instances as positive for class A would mean more negative instances for class B, and conversely. If we were talking micro-recall we could still use the higher importance in proportion of class A, but with macro-recall we can't. Note that this is a typical case where accuracy could be used, since it would give the same weight to both classes and would be a much simpler metric.

So the only way to improve macro-recall is to increase true positives. Let's look at the options:

  1) A: 10*N observations, B: N observations

Pro: performance for class A is maximized. Cons: class B proportionally disadvantaged, so possible loss in its performance.

  2) A: 5*N observations, B: N observations

Cons: class A a bit less advantaged; pro: class B a bit less disadvantaged.

  3) A: N observations, B: N observations

Pro: maximum performance for class B; cons: class A underperforms.

Actually the best option is probably to use all of N*10 instances for A and repeat 10 times the N instances for B, so that:

the learning method can benefit from all the available training data for A
class B is not proportionally disadvantaged.

But my guess is that it's unlikely to make a very big difference anyway. When it's a matter of increasing true positives, it's usually the features and the ML method which can have an impact.


Here, a very good article: http://machinelearningmastery.com/time-series-forecasting-supervised-learning/

In a few words, define a window of size n and that is the size of your feature vector. Reshape the dataset and play.


My Simple suggestion is to install python , Anaconda on the Linux machine from command line.
As Jupyter Notebook gets installed with Anaconda package.
Just give command
jupyter notebook
Now we can connect to ipython notebook in the Virtual Machine from the token generated from above command from any browser in any other machine locally.


Pandas: how can I create multi-level columns

Finally, I found a solution.

You can find the example script below.

#!/usr/bin/env python3
import pickle
import pandas as pd
import itertools
import numpy as np

data = pd.DataFrame(np.random.randn(10, 5), columns=('0_n', '1_n', '0_p', '1_p', 'x'))

indices = set()
groups = set()
others = set()
for c in data.columns:
    if '_' in c:
        (i, g) = c.split('_')
        c2 = pd.MultiIndex.from_tuples((i, g),)
        indices.add(int(i))
        groups.add(g)
    else:
        others.add(c)
columns = list(itertools.product(groups, indices))
columns = pd.MultiIndex.from_tuples(columns)
ret = pd.DataFrame(columns=columns)
for c in columns:
    ret[c] = data['%d_%s' % (int(c[1]), c[0])]
for c in others:
    ret[c] = data['%s' % c]
ret.rename(columns={'total': 'total_indices'}, inplace=True)

print("Before:")
print(data)
print("")
print("After:")
print(ret)

Sorry for this...


TF-IDF: How to handle terms not part of the corpus

In my opinion, there is no way to deal out-of-vocabulary terms in TF-IDF as it only works using the test corpus. It's like you are trying to find certain word like technology in sports corpus.


In Word2Vec, how to the vector values translate back to related words?

In case you use Gensim in Python, you can create a dictionary with the word and the vector.

from gensim.models import Word2Vec

# we create a Word2Vec model using a window size of 4 and word vectors dimensionality of 100
model_1 = Word2Vec(window=4,size=100,sg=1,min_count=1, workers = -1)
# prepare the model vocabulary
model_1.build_vocab(this_is_your_text_dataset)
# train the model
model_1.train(this_is_your_text_dataset,total_examples=model_1.corpus_count,epochs=model_1.iter)

#we create a dictionary with the word and its vector as the value
vectors_model = dict(zip(model_1.wv.index2word, model_1.wv.syn0))

The above will create a dictionary in the format:

{
"word_1": vector_1,
"word_2": vector_2,
"word_3": vector_3,
....
}

With this, you can go back and forth from words to vectors and from vectors to words.


Does the Bayesian MAP give a probability distribution over unseen data?

Bayesian MAP gives a point estimate $\mathbf{w}_{MAP}$ for model parameter $\mathbf{w}$, which implies
$p(\mathbf{w}|\mathbf{t},\cdots) = \left\{\begin{matrix}
1 & \mathbf{w}=\mathbf{w}_{MAP}\\
0 &\text{o.w.}
\end{matrix}\right.$
However, distribution $p(t_*|x_*, \mathbf{w}_{MAP}, \sigma^2)$ is the probabilistic model of target $t_*$ given input $x_*$ and parameters; after eq. (8), paper explicitly says $x$'s are omitted from the givens of target likelihood $p(t|.)$, but I think they should have been kept there to avoid confusion. It might also be helpful to remind that $(\mathbf{x}, \mathbf{t}) = \{(x_1, t_1),...,(x_n, t_n)\}$ are the observed (input, target) pairs.
Paper says when $\mathbf{w}_{MAP}$ is calculated, which would be the same as $\mathbf{w}_{PLS}$, mean of target distribution will be the same as the deterministic target $t_* = y(x_*; \mathbf{w}_{PLS})$ from Penalized Least Squares. That is,
$\mathbb{E}[t|x_*,\mathbf{w}_{MAP}, \sigma^2] = \int tp(t|x_*,\mathbf{w}_{MAP}, \sigma^2)dt = y(x_*; \mathbf{w}_{PLS})$
In summary, in (Penalized) Least Squares, both target and parameters are deterministic. In non-Bayesian and Bayesian MAP, parameter is deterministic but target is probabilistic. In (true) Bayesian, both target and parameters are probabilistic. In full Bayesian, hyper-parameters (parameters of distribution of parameters, e.g. $\alpha$) are also probabilistic.


You can move the title closer to the first figure. Just add the following two lines at the end of the code

fig.tight_layout()
fig.subplots_adjust(top=0.95)