Why do we move in the negative direction of the gradient in Gradient Descent?

Consider a simple example where the cost function to be a parabola $y=x^2$ which is convex(ideal case) with a one global minima at $x=0$
Here your $y$ is the independent variable and $x$ is the dependent variable, analogus to the weights of model that you are trying to learn.
This is how it would look like.

Let's apply gradient descent to this particular cost function(parabola) to find it's minima.
From calculus it is clear that $dy/dx = 2*x$. So that means that the gradients are positive in the $1^{st}$ quadrant and negative in the $2^{nd}$. That means for every positive small step in x that we take, we move away from origin in the $1^{st}$ quadrant and move towards the origin in the $2^{nd}$ quadrant(step is still positive).
In the update rule of gradient descent the '-' negative sign basically negates the gradient and hence always moves towards the local minima.

$1^{st}$ quadrant -> gradient is positive, but if you use this as it is you move away from origin or minima. So, the negative sign helps here.
$2^{nd}$ quadrant -> gradient is negative, but if you use this as it is you move away from origin or minima(addition of two negative values). So, the negative sign helps here.

Here is a small python code to make things clearer-
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-4, 4, 200)
y = x**2

plt.xlabel('x')
plt.ylabel('y = x^2')
plt.plot(x, y)

# learning rate
lr = 0.1
np.random.seed(20)
x_start = np.random.normal(0, 2, 1)
dy_dx_old = 2 * x_start
dy_dx_new = 0

tolerance = 1e-2
# stop once the value has converged
while abs(dy_dx_new - dy_dx_old) > tolerance:
    dy_dx_old = dy_dx_new
    x_start = x_start - lr * dy_dx_old
    dy_dx_new = 2 * x_start
    plt.scatter(x_start, x_start**2)
    plt.pause(0.5)
plt.show()


Which colour channel from a TIFF image do I have to use?

From the readme of your kaggle link;

All images are provided in .tif format with 3 channels per image.
For 101 cases, 3 sequences are available, i.e. pre-contrast, FLAIR, post-contrast (in this order of channels).
For 9 cases, post-contrast sequence is missing and for 6 cases, pre-contrast sequence is missing.
Missing sequences are replaced with FLAIR sequence to make all images 3-channel.
Masks are binary, 1-channel images.
They segment FLAIR abnormality present in the FLAIR sequence (available for all cases).

So you could use all three channel.


How modelling is affected by similar feature distributions across classes?

EDA
What you are doing there, falls under Exploratory Data Analysis (EDA). A better way to investigate how your features between them and are distributed across classes is a correlogram, sometimes referred to as pairplot.
A correlogram helps with

investigating relationships between pairs of numerical features, via scatter plots for each pair of features
inspecting the distribution of each feature, using a histogram or a density plot in the diagonal of the pairplot.

You can easily create one of those in seaborn:
# library & dataset
import seaborn as sns
import matplotlib.pyplot as plt
df = sns.load_dataset('iris')

# Basic correlogram
sns.pairplot(df, kind="scatter")
sns.plt.show()

# Annotate classes with in different colours
sns.pairplot(df, kind="scatter", hue="species")

# Use regression instead of scatter
sns.pairplot(df, kind="reg", hue="species")

You can see how classes may or may not form clusters which an algorithm could potentially draw hyperplanes by combining and transforming features.
Your question

Can it mean that the feature isn't that significant to data prediction and can be dropped?

The y axis maximum value differs between the two plots you have posted, and so it is not easy visually inspect fairly.
One could potentially assume that the model may not "work" with that particular feature to begin with, should there be other variables in that dataset that are easier to separate. This assumption however (a) can be considered naïve given how ML algorithms work to transform the feature space in order to make classes separable and therefore (b) does not constitute grounds for dropping this feature.
A less useful feature in your dataset can still increase the performance of your model. In the case of unbearably large feature space you can use dimensionality reduction techniques (PCA, kernel PCA, autoencoders).
Dropping features usually relates to how well these relate to the ground truth e.g. noise, or how they may affect feature importance and model stability e.g. multicollinearities.

Image from: https://stackoverflow.com/questions/59212378/how-do-i-get-the-diagonal-of-sns-pairplot


The churn rate is users the ratio of active users that stop being active after a given period.

The churn rate for application user is exactly the one used for other cases, like employees.
However, when dealing with employees it is clear whether the employee is active,
The problem with application users is the identification that they are no longer active.

What you can do is to identify a period of in activity after which, the vast majority of these user never return.

For example, take the users that were active during January of the previous year and were not active for 7 days since then. If 99% of them were not active for the rest of the year than 7 days are a good indication of being in active.

The suitable period is usually determined by the domain (e.g., some applications are used in periods longer than 7 days) and the required confidence level (99%, 90%).
Note that you don't want to be too safe because if you'll decide that a user is inactive only after 2 years, this definition will prevent you from identifying users that are practicality lost and take actions to reduce the churn rate.


How to prepare the training data for SVD-based Recommendation?

I consider this answer as a general approach for the data preparation for CF algorithms and based on my experience. I do not know the specific case of Your data.

While preparing the data for CF I would take care about those things:

dataset size and sparsity - there are many CF algorithms and they are giving different results depending on dataset size (users and items) and its data density. Simple intuition: memory-based KNN methods will be suffering from data sparsity while on the same setup the MF can perform better. Good comparison is presented in this article: A Comparative Study of Collaborative Filtering Algorithms. Moreover, CF methods can depends on the number of users and items differently. To reduce sparsity, You removing items/users which have lower number of ratings/click than selected threshold. This is iterative process. As it's said in the question, when You remove particular item/user it can happen that different user/item respectively can be below the threshold. For research purposes, I even have prepared the function which calculate items/user threshold in order to receive the given sparsity.
outliers in the meaning of filtering unusual users and items. Those, which can influence Your model greatly, but in a wrong way. In my cases (e-commerce website) it were cases like: really heavy users - other shops, bots (scrapping scripts) that was collecting data, internal traffic (blacklist IPs), bidding automation tools, funny offers which lands on a digg-like webpages. Someone can say that this can be removed by user/item bias. Yes, as soon as Your CF method is using it :-)
correct slicing of the dataset. For some reasons (e.g. computational) You have to divide You data on smaller parts. And this is reasonable. Divide and conquer approach. But in such cases before doing this, check if You are not loosing a lot of the information by such slicing. Here is the example: we are slicing offers by categories, categories are formed into a category tree, for this reason the same leaf can be part of few main categories. Thus, slicing by main categories can result in having only a part of the whole user-item interactions about viewed/bought together items in this leaf category.

Those are only few tips from my experience.
But I hope this would help.

I would appreciate any edits and sharing the results :-)


Logitic Regression cost function - what if ln(0)?

In practice, an offset is used to avoid log explosion due to values close to zero. For example $\hat{\text{log}}(x)=\text{log}(x + \text{1e-6})$.


There are nested loops in training machine learning models. The outer loop is selecting the hyperparameters for the model, including the architecture. The inner loop trains the model's parameters while holding the hyperparameters constant.
It is possible to try many possible options on outer loop and make those options dependent on the results of the inner loop.


Can a neural network recognize a letter B as an A if your trained it so?

Does the network recognize A as an A as good as before?

Yes, it can. The reason is that if you try to illustrate the confusion matrix you can see that the FP, FN, TP and TN can be kept at a good level. What you have done is actually changing the distribution of your input data. Suppose that you train a network and it should recognize cats and dogs. You can train it just using white cats or with cats of different colours. The point here is that what you've done is not telling that $A$ and $B$ are of the same class. What you have done is classifying the first two letters as the same entity. Consider that what your network learns depends on the kind of network you use. Avoid using just Dense nets. Having image inputs, Dense nets try to find something like a mask while convolutional layers find appropriate features and those features which are better for classification would then passed to the dense layers.

  Does the network recognize B as an A too?

Again yes! But it depends on the distribution of your data. It can recognize $B$s, because you have trained it in that way. But should consider that the performance depends on the data you have. How your data is obtained to be good for generalisation.


Machine Learning: Balanced training set but highly unbalanced prediction set? How to adjust?

Over- or undersampling should be your second choice. Currently, the best method to deal with class-imbalances is to use the weights argument which sklearn- and keras-classifiers support (see for example the DecisionTreeClassifier).

On a general sidenote, I would recommend focusing on the f1-score,  AUC, and the confusion matrix to evaluate the model's performance, as accuracy might be not as informative in this case.


Machine learning algorithm for a dynamic set of attributes

I would train a gamma-distributed GLM; one for each language. Now, I assume the set of parameters is fixed within each language. You can incorporate ordinal variables (basic -> intermediate -> expert) using dummy coding (i.e., treat them as categorical variables). If there is no intermediate level one bit will do (0 = basic, 1 = expert). And so on for the other properties.

You can find free software to implement GLMs in various language, including python and R.


Linear Regression + KFold cross validation

Cross-validation

An extremely important concept to understand is that:

Cross-validation works independently of the model you use.

Cross-validation is just the process of splitting your data into multiple pairs of training and test sets. Once this is done, you can train that data using whatever model you like. In sklearn, which I assume you are using given your tags, you can use cross-validation with any classifier/regressor you'd like.

Cross-validation is way to evaluate how well the model will perform on data that it hasn't seen before. When you train your model on all your data, you do not have any data that the model hasn't seen before to test on. This is why you cross-validate so that you can get multiple estimates of what the performance might be. This is more robust than just splitting your data once between a training and test set because in this case, it might happen that your test set is easier very difficult or very hard. By cross-validating, you smooth this out.

Linear vs Logistic regression

Apparently, your output is ages, and therefore, you could use linear or logistic regression. If you use linear regression, you will try to guess the age directly. If you use logistic regression, you could round the output to the closest integer to obtain the proper target classes.

I would advise you to use logistic regression because it will allow your model to learn better. If you use linear regression, your classes are all as different as each other (from a mathematical point of view). However, my guess is that the 13 class is closer to 14 than it is to 20+.


How do I setup a server in the cloud for machine learning?

If you are looking for an external storage, then I would suggest you Redshift.

Redshift is a central warehouse and a columnar data store. It allows complex and huge data aggregations and joins; so it is a nice bet for serious Kaggle participants. (I personally use Redshift as data science architecture for kaggle.)

Combinations from some other vendor?

No, a combination of Redshift and S3 is enough. The data is stored in S3, and then loaded into Redshift using the COPY command.

Is there any way to interact with a GUI rather than a command line to set this up?

Yes, there is a CLI interface for Amazon AWS. However, you can use the boto library on Python for handling this stuff.

Would I have to install Anaconda or some other Python distribution before getting started?

Not a compulsion.  Anaconda is a wrapper of numerical and scientific libraries. You can install them on your own or intall Anaconda.

Check out PySpark, which allows you to do handle BigData in Python.


Why do convolutional neural networks work?

Actually I guess the question is a bit broad! Anyway.

Understanding Convolution Nets

What is learned in ConvNets tries to minimize the cost function to categorize the inputs correctly in classification tasks. All parameter changing and learned filters are in order to achieve the mentioned goal.

Learned Features in Different Layers

They try to reduce the cost by learning low level, sometimes meaningless, features like horizontal and vertical lines in their first layers and then stacking them to make abstract shapes, which often have meaning, in their last layers. For illustrating this fig. 1, which has been used from here, can be considered. The input is the bus and the gird shows the activations after passing the input through different filters in the first layer. As it can be seen the red frame which is the activation of a filter, which its parameters have been learned, has been activated for relatively horizontal edges. The blue frame has been activated for relatively vertical edges. It is possible that ConvNets learn unknown filters that are useful and we, as e.g. computer vision practitioners, have not discovered that they may be useful. The best part of these nets is that they try to find appropriate filters by their own and don't use our limited discovered filters. They learn filters to reduce the amount of cost function. As mentioned these filters are not necessarily known.

In deeper layers, the features learned in previous layers come together and make shapes which often have meaning. In this paper it has been discussed that these layers may have activations which are meaningful to us or the concepts which have meaning to us, as human beings, may be distributed among other activations. In fig. 2 the green frame shows the activatins of a filter in the fifth layer of a ConvNet. This filter cares about the faces. Suppose that the red one cares about hair. These have meaning. As it can be seen there are other activations that have been activated right in the position of typical faces in the input, the green frame is one of them; The blue frame is another example of these. Accordingly, abstraction of shapes can be learned by a filter or numerous filters. In other words, each concept, like face and its components, can be distributed among the filters. In cases where the concepts are distributed among different layers, if someone look at each of them, they may be sophisticated. The information is distributed among them and for understanding that information all of those filters and their activations have to be considered although they may seem so much complicated.

CNNs should not be considered as black boxes at all. Zeiler et all in this amazing paper have discussed the development of better models is reduced to trial and error if you don't have understanding of what is done inside these nets. This paper tries to visualize the feature maps in ConvNets.

Capability to Handle Different Transformations to Generalize

ConvNets use pooling layers not only to reduce the number of parameters but also to have the capability to be insensitive to the exact position of each feature. Also the use of them enables the layers to learn different features which means first layers learn simple low level features like edges or arcs, and deeper layers learn more complicated features like eyes or eyebrows. Max Pooling e.g. tries to investigate whether a special feature exists in a special region or not. The idea of pooling layers is so useful but it is just capable to handle transition among other transformations. Although filters in different layers try to find different patterns, e.g. a rotated face is learned using different layers than a usual face, CNNs by there own do not have any layer to handle other transformations. To illustrate this suppose that you want to learn simple faces without any rotation with a minimal net. In this case your model may do that perfectly. suppose that you are asked to learn all kind of faces with arbitrary face rotation. In this case your model has to be much more bigger than the previous learned net. The reason is that there have to be filters to learn these rotations in the input. Unfortunately these are not all transformations. Your input may also be distorted too. These cases made Max Jaderberg et all angry. They composed this paper to deal with these problems in order to settle down our anger as theirs.

Convolutional Neural Networks Do Work

Finally after referring to these points, they work because they try to find patterns in the input data. They stack them to make abstract concepts by there convolution layers. They try to find out whether the input data has each of these concepts or not in there dense layers to figure out which class the input data belongs to.

I add some links which are helpful:

Understanding convolution operation
Understanding arithmetic behind ConvNets
Useful toolboxes for tracking what happens in these nets


How to use InceptionV3 without retraining

can I just provide an image to an untrained Inception and get a label back?

You will get a random output from the network because it hasn't been trained and has been initialized with random values for the weights.

  Is there an API for this?

Cloud Vision is an API with already trained models. If you want to train your own model you can use tensorflow, theano, caffe... there are several frameworks for it. None of them is easy.

  why does Google open source this if they're trying to make money off of google cloud vision?

Open sourcing tools and models gives to the community a lot of knowledge, in exchange the researchers community use the Google tools to improve the models, more people use the models, so it is easier for Google to hire Machine-Learning developers and, most important, because the real value of machine learning today is in the data and the infrastructure you have. The more data you have the better your models. The better the infrastructure the faster and cheaper you can train and deploy your models. And people will pay you to use your infrastructure.


Binary classification of partially labeled data

You of course cannot use them as training data, but they can still have some potential uses, for example:

-You can make predictions on that data and see if the distribution of the classes are similar to those in the labeled data.

-You can use them for feature engineering – e.g. if you are using something like principal components, or if you are using averages for some categorical variable (e.g. you have a variable like state and another like income, and you augment the state variable by creating a feature that is the average income for that state).

-If you need to impute missing values in the predictors, it’s better to use the whole dataset for that.

-You can see if the distributions of the predictor variables are similar, and if you see that, for example, in your test data you tend to predict poorly for observations with a low value of variable ‘x1’ and the data with no labels has mostly low values of ‘x1’, then you should perhaps change your model.


Could I add a one hot encoding to each feature representing "has data" versus "has no data"

If you are using Python, the Imputer class could be a way to fill in the holes. You can use it to define what counts as a missing value and define a method for replacing those (mean, median, mode, constant value). Without knowing what the data is, it's hard to say whether this is an ideal solution, but it is a very quick one.

The Imputer class also includes a MissingIndicator method which would ouput a Boolean array based on the data. This could be another option, and then you could stack the Boolean array onto the original data.


How to discretize certain features with a feature set?

Not sure about your question, but maybe something like the following could help:

Create dummy variables for your KeyCodes variable
Normalize only those variables using the StandardScaler.

import pandas as pd
import random
from sklearn.preprocessing import StandardScaler

# Let's assume the following dataframe
data = {
    'KeyCodes': [random.randrange(2, 223, 1) for _ in range(10000)],
    'age': [random.randrange(1, 100) for _ in range(10000)],
    'id': [i for i in range(10000)]
}

# First you need to create the dummy variables based in KeyCodes
df = pd.DataFrame.from_dict(data)
df.head()
dummies = pd.get_dummies(df['KeyCodes']).rename(columns=lambda x: f'KeyCode_{x}')
df = pd.concat([df, dummies], axis=1)
df.drop(['KeyCodes'], inplace=True, axis=1)

# Then you can apply the normalization to the subset of features you wish to normalize
normalized_df = df.copy()
col_names = [f'KeyCode_{i}' for i in range(8, 223)]
features_normalized = df[col_names]
scaler = StandardScaler().fit(features_normalized.values)
features_normalized = scaler.transform(features_normalized.values)
normalized_df[col_names] = features_normalized

# Explore the output
normalized_df['age'][:10]  # you can see it was not normalized
set(normalized_df['KeyCodes_8'])  # normalized version of the feature
set(df['KeyCodes_8'])  # not normalized version of the feature


Use spark_csv inside Jupyter and using Python

Assuming the rest of your configuration is correct all you have to do is to make spark-csv jar available to your program. There are a few ways you can achieve this:

manually download required jars including  spark-csv and csv parser (for example org.apache.commons.commons-csv) and put them somewhere on the CLASSPATH.
using --packages option (use Scala version which has been used to build Spark. Pre-built versions use 2.10):

using PYSPARK_SUBMIT_ARGS environmental variable:

export PACKAGES="com.databricks:spark-csv_2.11:1.3.0"
export PYSPARK_SUBMIT_ARGS="--packages ${PACKAGES} pyspark-shell"

adding Gradle string to spark.jars.packages in conf/spark-defaults.conf:

spark.jars.packages    com.databricks:spark-csv_2.11:1.3.0


First thing that comes to my mind is to do different encodings. There are some ways to deal with high cardinality categorical data such as: Label Encoding or the famous target encoding. Before anything else I will recommend changing the encoding type.
But, since your question about which predictor use with small and space data. I will go still with logistic regression, decision tree or SVM. When data is small all algorithms tend to work quite similar.
Things like Random Forest might perform well since they do bootstrapping what tends to be a way to sample your data with replacement.


The standardization of the pixel values comes from the same logic standardization data to make Gradient Descent (or some other similar optimization algorithm) faster. When you provide an image to the NN, it is no different from other data types, meaning you provide a sample with different features (or let's say variables). When an image is a case, your features are your pixels. That is, if you have a 32x32 image this means you have 1024 features. If your input is RGB, it creates 3 dimensions which make your feature size 32x32x3 that is, 3072. In other words, your variable in such a case is a specific pixel of all the images in your dataset. Your image is just a single row of data with 3072 features like in a simple regression problem where for example, house price is predicted using 5 features location, size, age of the building, number of rooms, and floor.
Now, before going into details, let's first think why distribution might be different. Depending on your image type, your specific channel values may have a bigger variance than others. For example, if your data samples, images, are about plants, trees, or forest then your Green and Blue channel values will probably vary a lot, but your Red channel will always have a value close to 0. Thus, the features that represent these channels will have a different distribution than others.
Another thing that needs to be considered is pixels' (even if it is a grayscale image) having dissimilar distribution due to natural reasons. For example, consider the dataset where all images are identity card photos of people, where the background is white. In such a case, some of the pixels, probably the top left and top right ones will have a variance close to zero due to having almost the same color value for all images.
These two cases cause the possibility of a specific pixel having, for example, values 0-10, another one having 3-240, another one have 126-255, etc. In such a case, what needs to be handled is the same problem as Andrew NG talks about in this video. Since the ranges of your values will vary in 3072 (for example) dimensions, gradient descent will need more steps to find optimal value in one dimension when compared to the other dimension. This is because your step size is fixed. Consider, you have 100 rungs in one ladder and 12 rungs in another ladder. With normal human step size 1, which one would be faster to climb? The latter.
To briefly summarize, since for all features in the NN same parameters are used, to expect those parameters affect the features in the same way (or objectively), these features should display similar characteristics. If not, some features might advantage more from these parameters when compared to others.


If you have the eigenvalues in a single vector you can simply use base R commands to only select values greater than one. The linked page uses the following commmand:
food.pca.eigen$values[food.pca.eigen$values >=1]

which simply uses base R syntax. food.pca.eigen$values >=1 returns a true/false vector depending on if the value is greater than or equal to 1 or not, which is then used as a mask for the food.pca.eigen vector. The only values selected are the ones where the mask has the value of true, which are then ones with a value equal to or greater than 1.


What is the scalability of linear regression and decision trees?

If you use the standard OLS formula :

$$ \beta = (X^TX)^{-1}X^Ty$$

the overall linear regression complexity is $O(np^2)$, where n is the number of exemple and p the number of features.

For simple trees the theoretical complexity if of the same order $O(np^2)$.

In practice other techniques are used, like gradient descent for the regression. Also you may have simplifications depending on your problem ($n>>p$ for exemple, may lead to the use of sparse matrices).

Practical implementation like sklearn have following complexities (see https://www.thekerneltrip.com/machine/learning/computational-complexity-learning-algorithms/):

LinearRegression : $O(n^{0.72}p^{1.3})$

ExtraTreesRegressor: $O(n^{1.03}p^{0.88})$

Overall they are relatively simple algos and thus are pretty much as scalable as you can get. (or conversely we wouldn't use algorithms that are not scalable)


Dense Layer vs convolutional layer - when to use them and how

As known, the main difference between the Convolutional layer and the Dense layer is that Convolutional Layer uses fewer parameters by forcing input values to share the parameters. The Dense Layer uses a linear operation meaning every output is formed by the function based on every input. In other words, we "force" every input to the function and let the NN learn its relation to the output. As a result, there appear n*m connections (or weights) where n denotes the number of inputs and m denotes the number of outputs. On the other hand, the Convolutional layer uses a filter to operate the convolution operation which has a small size most of the time. An output of the convolution layers is formed by just a small size of inputs which depends on the filter's size and the weights are shared for all the pixels. That is, the output is constructed by using the same coefficients for all pixels by using the neighboring pixels as an input.
In the convolutional layer, as known, the filter's center is located in the pixel and then the linear operation is processed using the neighboring pixels of that pixel. That is, we, in advance, know that there is a strong relationship between the neighboring pixels. If we would not be sure that the neighboring pixels have a strong relationship with neighboring pixels it would not be logical to use Convolutional Layer, instead we would force all the pixels (inputs) to the function to learn the relationship by using Dense Layer. So, by using the Convolutional Layer we assume that the neighboring pixels are the main representative of the center pixel and as we go far away from the pixel, that is, the pixels that are away from the center pixel do not really possess the same characteristics as the center pixel. They might even be a different object thus may lead to a spurious result or would cause your function to learn redundant information which in reality has no relationship with.
In short, since we have prior knowledge about our data and information in it, we not only take the heavy load from our model by using a convolutional layer but also show it the exact location of the data that might be useful for it to learn while keeping it away from redundant data. However, frequently we use both in the same model, it is often simply due to the fact that we do not know what is going on in layer 10 (for example). Namely, we do not have that prior information about data anymore, because we do not know what it learns in those deep layers. Thus, we use Dense Layer, by giving all the inputs, we give the "full responsibility" to learn. In other words, we say to a Dense Layer that "Here are my features (pixels maybe), I don't know the true relationship between them, please find it yourself". We say to a Convolutional Layer that "Here are my features (pixels maybe), I am sure that it is enough just to look the few pixels around the center pixel" and that relationship is preserved through all the pixels.
So knowing, how it works, in different contexts, you can apply them to different scenarios. If you have a good knowledge of your data and its structure then you can choose the relevant layer while designing your model.


Issue while predicting multiple values which possess different order of magnitude (regression)

I am maybe thinking about creating one loss per target. Freeze all the
  network except the output layer and each epoch let one of the loss
  train the whole network. But I am not sure it is a good idea.

This is one option. It should work but it may be slower to converge as you only make use of only a third of your losses each epoch. Also, it is not completely unlikely that your losses may work "against each other". Meaning that they could somewhat cancel each other out, which can make convergence take longer.

  Currently I am normalizing the target before predicting it, but I don't like this solution very much. I would prefer having a slightly more complicated architecture than normalizing and denormalizing (afraid of data leakage) and it seems more robust to directly output the right value.

That's what I would do. I don't think you should be afraid of data leakage. Computers have enough precision in them that normalizing/denormalizing shouldn't create any issues. Also, keep in mind that computers have greater precision for floats close to 0. So normalizing your data may actually be good.

And when you say, I'd rather directly output the right value. You can do both! Your network could directly output the denormalized value, but the loss used to train the network can be computed on any layer before that.


Test Binary Classifier on a Test-set that includes only one class

Give cross-validation or bootstrapping a shot!  Also, for metrics, look at per class performance and macro averaged.


Dealing with zeros when plotting log-scaled data

Your suggestion is a valid one, encoding variables with a known outcome once the scaling is applied. Log(1) will become zero, so just keep that in mind for your next stage. You can use clip or replace for this:
df.clip(1, df.max())

or try replacing with a NaN
df.replace(0, np.nan)

Alternatively you could do one of the following:

Drop the zero value rows e.g. df = df[df['column'] !=0] but then you lose some data.
Fill the zero values with a statistically representative value (i.e. interpolation). You can explore the Pandas interpolate method here.

Which ever method you decide upon depends on your use-case, and compatibility with the plotting function you use.


I've had a similar problem to putting in x limits to date time data. To combat this, I normally extract the subset of the data which fall in the limits you want to review in further detail.
To do this, you can do something like this:
df_subset = df[(df["Date"] < end_date) & (df["Date"] > start_end)]

And then plot this subset.


In Grid Search, Random Search, we trained different independent model i.e. irrespective of the outcome of previous models and later evaluate all these independent models.

  We have another automatic technique for hyper-parameter optimization,
  known as Bayesian  HyperOpt.

It took the references of previous model and try to identify the best new set of hyper param and trained a new model.
This two step process gets repeated, until we get the desired results.

You can get many online references for complete implementation.

Although I could not find any approach for batch by batch, I thought its worth to mention Bayesian approach.


What are 2D dimensionality reduction algorithms good for?

There are many questions there, I'll try to address each in turn.

How is t-SNE* better than just taking a random (probably stratified) sample of the data?

If your goal is to provide a visual overview of the data then clearly a stratified sample is not going to do that -- the samples each still live in high dimensional space, so are no easier to visualise, and if you are interested in how the data relate to each other then looking through a sample item by item does not necessarily help to build a reasonable mental model of those relationships. So sampling and visualisation with t-SNE (or similar approaches) are both attacking similar underlying problems: "What does my data look like"; they provide very different views on that, highlighting different aspects however, and I would say are entirely complementary. Why not both!

How is t-SNE* better than just fitting a neural network with a 2-neuron bottleneck to the data and then taking the (normalized) value of the 2 neurons for an embedding?

This is harder, because I can't point you to explicit studies demonstrating otherwise, but t-SNE generally does a better job of providing a visual representation that is meaningful to users. It is worth noting that t-SNE was co-created by Geoff Hinton, who I am sure knows a great deal about neural networks and their potential uses and benefits. If Geoff felt t-SNE was worth using then one can reasonably presume it has some merits over other NN approaches. Now, with that in mind one can probably interpret t-SNE as a NN with 2-neuron bottleneck; I haven't tried to write that out in detail, so don't quote me on it. Finally, NNs aren't necessarily the answer to all problems; t-SNE is an algorithm for manifold learning specifically designed for low dimensional embedding, while an autoencoder with suitable bottleneck may happen to give similar results, but certainly isn't necessarily clearly suited to the task.

Does t-SNE* give any guarantees?

Guarantees of what? Providing a visual representation of the data? Yes it will do that. Guarantee that the Kullback-Liebler divergence of inferred distributions for the high dimensional and low dimensional representations has been minimised? Up to the quality of the optimisation, yes. Guarantee that local structure is preserved? Up to a suitable choice of perplexity for the dataset, yes. Guarantee that the low dimensional representation is a fully faithful representation of the global structure of the data? No, no guarantees there. It really depends on what you're looking for.

Is t-SNE* good for the construction of classifiers? I mean: If you already have a classifier which is much better than random / guessing the most frequent class, does t-SNE help you to make better classifiers? How?

Not really -- not any more than what you potentially expect from any dimensionality reduction technique, and as you note forcing things to 2 dimensions constrains that a lot. In principle, since t-SNE focuses on preserving local structure and sacrifices global structure to do so, if your classification relies heavily on local structure then t-SNE could perform better than other dimension reduction techniques that seek to preserve global structure and thus have poorer local structure representation. In the case of unsupervised (density) clustering it is certainly true that local structure is more important than global structure so potentially t-SNE can help. However compression to 2 dimensions may be rather extreme.

There are many dimensionality reduction algorithms. How does one compare the non-linear ones? When is one algorithm better than others? Especially: Are they better than bottleneck features of neural networks?

If your interest is in visualisation (as is the case with t-SNE) then unfortunately comparisons are inevitably fairly subjective. This is much like the case of clustering. Certainly there are many different measures of clustering quality, but often that comes down to "what do you mean by a cluster?", and for each measure there is a clustering algorithm that optimises for that measure. Likewise there are measures for how well an embedding has succeeded, but ultimately it depends on what you mean by a successful embedding (is preserving large scale relationships critical? is preserving local structure more important? How do you weigh one against the other?), and for any given measure there is an algorithm that optimises that measure. For a subjective evaluation the standard approach seems to be to embed labelled data and then view the result colored by label -- no, that is never going to clearly demonstrate superiority, but it can hint at what seems to be working. In practice t-SNE seems to fair well at this (hence its popularity). Are any of these better than bottlenecked autoencoders? I'm not sure why one assumes that a bottlenecked autoencoder is by default the superior option, but certainly in my experience whenever I've tried several options on data I've found t-SNE to provide better intuition about the dataset. Anecdotes are not data of course, so by all means try the options yourself.

I hope this covers most of it, although perhaps these were not quite the sort of answers you wanted.


Coding a Content Addressable Memory on a GPU

I similarity matched a billion 8 character strings at about 1.5fps on a gtx980 - is that fast enough for you?

similarity matching -> do it very block list like, and charge through all the patterns in the whole corpus/memory, match in lots of 32 bits and get the similarity score by getting the pop sum is the fastest way to do it I think. (gpu or cpu)

exact matching -> you get a lot more if you exact match because it needent access the whole memory per access, its approximately a root more optimized, put it in a tree, reading is simple, just go node to node, but writing is a problem, youll get thread conflicts if it tries to append the tree twice at the same place.

Interlocked operations make it alot simpler to do,  but its easier if you read in parallel and just dont bother to parallel the write, unless its a necessity.


Is autocorrelation the same as multicollinearity?

Autocorrelation is a measure of a correlation of a signal with itself, as a function of delay.

Correlation is usually between two different variables (without respect to time). You can take it with the same variable, but that will of course equal 1.

Either way, XGB is robust to multi-colinearity (correlations between features, or explanatory variables as you'd call them in statistics), please have a look at this post for more information.

Lots of things could be wrong with the model, I would start with really understanding the data and making sure there's nothing wrong with it(missing values, unexpected distributions etc...).


Could you clarify what you mean by "However in the last booster all the leaf values changed to around 0.5 probability"?

My understanding is when computing predicted probabilities, you'd need to add base score (default = 0.5) to estimated weight parameter (leaf score), like so:

$\hat{p} = \frac{\text{exp(0.5 + w)}}{\text{1 + exp(0.5 + w)}}$

where $\text{w}$ is the estimated leaf score.

Below, is the link to the default xgboost parameters in python API: https://xgboost.readthedocs.io/en/latest/python/python_api.html

class xgboost.XGBClassifier(max_depth=3,
      learning_rate=0.1, n_estimators=100, silent=True,
      objective='binary:logistic',
      booster='gbtree', n_jobs=1, nthread=None, gamma=0, min_child_weight=1,
      max_delta_step=0, subsample=1, colsample_bytree=1,
      colsample_bylevel=1,
      reg_alpha=0, reg_lambda=1, scale_pos_weight=1, base_score=0.5,
      random_state=0,
      seed=None, missing=None, **kwargs)

base_score:
The initial prediction score of all instances, global bias.

Does this answer your question?


Modern Feature Selection Review/Resources

I looked into it quite recently and found these papers:

Saeys et al. (2007)  (also already a bit dated, but a good overview, and I think the 'key' paper in this area).
Dougherty et al. (2009)  (interesting paper discussing various difficulties such as classifier dependence, label distribution and sample size).
Bolòn-Canedo et al. (2014) .
Dessì & Pes (2015).

However, I do agree with you that there is a need for an updated review regarding this topic.


What Predictions can be done for Restroom/Washroom Data?

Welcome to the site! In order to prevent from falling into common data science traps, I would encourage you to think about the following:

You should never take data at face value. You should be thinking about the transforms that you could be doing to your data to get more insights. For example, from the fields you have, you can also determine the time in between visitors, right? Can you determine the number of visitors per day? Also, what other datasets are out there that you can be tying together with your data?
You shouldn't be thinking about the business value of the data until you figure out what the data is telling you. This is where exploratory data analysis (EDA) comes in. You should explore your data and let the data speak to you (not the other way around!) so that you can be in a better position to think about your entire project.

Remember, the entire point of data science is to enable you to ask smarter questions. So, take the time to explore your data and figure what else you need. That will enable you to ask smarter questions and then declare some use cases for your business/client.


Setting class weights for categorical labels in Keras using generator

For categorical data, it is best to use sample_weight instead of class_weight argument. This can be done by simply giving all samples of a particular class the same weight. sample_weight works for categorical data because it takes a numpy array as its value as opposed to a dictionary (which won't work for categorical class labels) in case of class_weight.

See: Keras sequential model methods

  You can either pass a flat (1D) Numpy array with the same length as
  the input samples (1:1 mapping between weights and samples), or in the
  case of temporal data, you can pass a 2D array with shape (samples,
  sequence_length), to apply a different weight to every timestep of
  every sample. In this case you should make sure to specify
  sample_weight_mode="temporal" in compile().

The other way, as you mentioned is using custom weighted loss function. A detailed discussion on it can be found here.


Linear transformation from one sample to another

Say we want to sample $x$ from $N(a,b)$. We could certainly do this in R using rnorm(a,b). However, we could also sample $z$ from $N(0,1)$ and apply the linear transformation $a + zb$. This transformation would then give us samples from $N(a,b)$.
What I've described is for the univariate case. You can easily extend this to the multivariate case.


How best to show the best model over multiple labels?

You can use some index, such as (possibly weighted) average.

Another way would be to use 3 dimensional graph, with one of the dimensions being the task.


Type of regression with nominal, ordinal, interval and ratio data

I think I'd try to model the rate at which the ambulances are released instead of the time it takes. That would let me use Poisson regression, which is the canonical type of GLM for rates (in R's GLM, set family = "poisson".) However, in order to use PR, the data needs to have its variance equal to its mean, or at least close to it.


Alternative methods for novelty detection and correlations

I'm doing something similar and found that the best approach has to be comprehensive. My process obtains a ranking of signals based on degree and measure of anomaly, both as a function of 'bads' (if available). If information on 'bads' is unavailable then I perform the same study but as a function of the degree of anomaly. In both cases, I perform hierarchical clusters. These are helpful in grouping signals that have similar behavior.


How to create a barplot using multiple column

The function you are looking for is gather from the tidyr package. This function takes a wide data.frame and makes it a long data.frame. gather is easy to use:

library(dplyr)
library(tidyr)

# Building a sample data.frame like your example data.
df <- data.frame(Boardgame = c("Game1", "Game2", "Game3", "Game4", "Game5"),
             categorie1 = c("Deduction", "Deduction", "Card Game", "Horror", "Horror"),
             categorie2 = c("Medieval", "Medieval", "Medieval", NA, "Medieval"),
             categorie3 = c("Word Game", NA, "Zombies", NA, "Zombies"),
             stringsAsFactors = FALSE)

# Using gather from tidyr and some dplyr functions:
df %>%
  gather(key = "Cat_Label",    # Key is the name of the column that will holdthe old column names
         value = "Categorie",  # Value is the name of the column that will hold the data
         -Boardgame,           # Ignore the Boardgame column, use every other column
         na.rm = TRUE) %>%     # Remove NA values
  arrange(Boardgame) %>%       # Sort by Boardgame
  select(-Cat_Label)           # Remove the unneeded Cat_Label column (if you want)

# Results:
   Boardgame Categorie
1      Game1 Deduction
2      Game1  Medieval
3      Game1 Word Game
4      Game2 Deduction
5      Game2  Medieval
6      Game3 Card Game
7      Game3  Medieval
8      Game3   Zombies
9      Game4    Horror
10     Game5    Horror
11     Game5  Medieval
12     Game5   Zombies

The -boardgame notation in the gather function means that every column but Boardgame will be gathered, even if you have 200 categorie columns. Once you have your data.frame, you can use ggplot2 to visualize the Categorie column as you see fit.

It should be noted that development of gather by the Tidyverse team is complete. A new, more general function pivot_longer has been implemented to replace gather. It has similar usage in a simple case, but the arguments are a little different:

df %>%
  pivot_longer(cols = -Boardgame,         # You now explicitly declare the columns
               names_to = "Cat_Label",    # New column containing old column names
               values_to = "Categorie",   # New column containing old column values
               values_drop_na = TRUE)     # Remove NA values


Calculate Q parameter for Deep Q-Learning applied to videogames

The neural network you need to implement for Q-learning must approximate the Q function $Q(s,a)$. There are two ways to do this:

Using $(s,a)$ as input. A lot of the literature will assume this for simplicity, and you will see notations like $\hat{q}(s,a,\theta)$ to show you are approximating the function, and have parameters $\theta$ to learn for the neural network.
Using $(s)$ as input, and have the network output multiple $Q(s,a)$ values - one for each possible action. E.g. if you have four possible actions, then your output might be $[\hat{q}_0, \hat{q}_1, \hat{q}_2, \hat{q}_3]$. This can be more efficient, since in order to find the best, maximising, action, you need all the action values calculated.

Once you have a neural network set up like this, and a table of history (that grows on each action actually taken), this is how you make use of it:

For each sampled [state, action, reward, future_state]:
  Calculate td_target = reward + gamma * max_a'(Q(future_state, a')):
    Run NN(future_state, a') forward for each possible action a' and find max
  Train the NN using the inputs (state, action) and desired output td_target

You would use this variant if the network output multiple Q values at once:

For each sampled [state, action, reward, future_state]:
  Calculate td_target = reward + gamma * max_a'(Q(future_state, a')):
    Run the NN(future_state) forward and take the max output
  Construct the desired output:
    Run the NN(state) forward to get array e.g. [q0, q1, q2, q3]
    Substitute td_target depending on action e.g. [q0, q1, td_target, q3]
  Train the NN using the inputs (state) to learn the desired output

Although this second approach looks more complex, it avoids looping (or mini-batching) over multiple a' values to find a max, so can be more efficient.

If you are using a frozen copy of NN to help with stability (a common feature in DQN), then you should use the frozen copy exclusively to calculate TD targets, and the learning copy to calculate current outputs.

Note it is important that you don't store, but instead re-calculate Q values  at all times. That is both because the initial values will be incorrect (depending on how the NN was initialised), and also that they should improve over time as the agent learns a better policy.

Another important detail is that you should not use the NN to calculate Q value for terminal states. If future_state is terminal, then instead of using the NN, treat max_a'(Q(future_state, a')) as zero. Typically just detect that this is a terminal state and hard-code a 0 somehow for it.


Application of Machine learning in modelling experimental system as a PhD topic

It's very unlikely that anybody here has the expertise needed to assess the validity of a PhD topic, especially about a highly specialized (and unspecified) experimental system. You should discuss this with your potential academic supervisor, and I'd suggest you try to find a co-supervisor or collaborator expert in DL as well.

In general for a decent PhD topic you would need to make sure that it hasn't been done before and that there are reasonable grounds to think it's feasible. Study the state of the art and find out if anything similar has been published:

If something very close already exists, it might be too trivial
If there's nothing even remotely close to it, it's probably not feasible or very hard.

Good luck!


How to draw Deep learning network architecture diagrams?

I recently found this online tool that produces publication-ready NN-architecture schematics. It is called NN-SVG and made by Alex Lenail.

You can easily export these to use in, say, LaTeX for example.

Here are a few examples:

AlexNet style

LeNet style

and the good old Fully Connected style


Could Deep Learning be used to crack encryption?

Probably not. Modern encryption systems are designed around cryptographic random number generators, their output is designed to be statistically indistinguishable from true randomness. Machine learning is generally based on discovering statistical patterns in the data, and with truly random data there is none. Even for flawed crypto where there is some small pattern to be found, the large amount of randomness in the input will overwhelm any direct attempt to decrypt the ciphertext.

In addition, there is no heuristic you can use to tell if you are getting close to a correct decryption - a single bit out in a guess at a key for example will completely scramble the output (blame Hollywood for when it shows decryption on screen like some crossword puzzle where the correct letters drop into place). That all-or-nothingness rules out discovering algorithms via a machine learning process, even when you have the encryption key. The best you can do is brute-force all known algorithms. If you don't have the key, then you have to brute-force all possible keys too.

You could explore how difficult the problem is by attempting to guess the seed value used for a random number generator. Using the Mersenne Twister RNG (the standard one used in e.g. Python), then the input could be the bit pattern for 624 32-bit unsigned integers, and the output could be the 32 bits of the seed used to generate that series. The reason I suggest those specific numbers is because it is in fact possible to crack Mersenne Twister with that much data. However, I still think that ML approaches would be entirely the wrong tool to do so.

Another simple variant would be to see if you can teach a network to either produce or reverse a cryptographic hash. You could start with a known broken one such as MD5. Input and output could be 80 bits, which simplifies the architecture and pipeline enough that you could put together this test in a few hours. Even though MD5 is known to be compromised, I think there is zero chance you could teach a neural network to find any pattern.

One important detail: If you want to "crack" an encryption, you will not be able to use the key as a known value. Chances are though, that even if you provide the key to an ML process, it will be unable to learn how to decrypt.


You can extend the base Keras implementation of callbacks with a custom on_epoch_end method which compares your metric of interest against a threshold for early stopping.
From the linked article they provide a code sample with a custom callback class + calling that class during model.fit:
import tensorflow as tf

# Implement callback function to stop training
# when accuracy reaches ACCURACY_THRESHOLD
ACCURACY_THRESHOLD = 0.95

class myCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs={}):
        if(logs.get('acc') > ACCURACY_THRESHOLD):
            print("\nReached %2.2f%% accuracy, so stopping training!!" %(ACCURACY_THRESHOLD*100))
            self.model.stop_training = True

# Instantiate a callback object
callbacks = myCallback()

# Load fashion mninst dataset
mnist = tf.keras.datasets.fashion_mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
# Scale data
x_train, x_test = x_train / 255.0, x_test / 255.0

# Build a conv dnn model
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(512, activation=tf.nn.relu),
    tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])

model.compile(optimizer='adam', \
                loss='sparse_categorical_crossentropy', \
                metrics=['accuracy'])

model.fit(x_train, y_train, epochs=20, callbacks=[callbacks])

Check the linked article for more details:
https://towardsdatascience.com/neural-network-with-tensorflow-how-to-stop-training-using-callback-5c8d575c18a9


Help deciding the best statistic tool for my data

You could start with plotting the relation between any relevant pair of variables, typically with a simple scatter plot and possibly using color to represent a third variable.
Pearson correlation coefficient is a simple but useful measure of association between two variables. By calculating it between two variables on the whole data and then on subsets based on specific conditions (e.g. third column is A) you can observe the effect of the condition, i.e. see if it increases or decreases the correlation level.


Gensim word2vec training error on tweets

In newer W2V version it is not enough just to write:

model_name.train(sentences)

You have to write inside word_count or total_examples.

For example, I write:

model_name.train(sentences, total_examples = token_count, epochs = model_name.iter )

where token_count = sum([len(sentence) for sentence in sentences]). And here is how I get sentences:

sentences = []
for raw_sentence in raw_sentences:

    if len(raw_sentence) > 0:
        sentences.append(sentence_to_wordlist(raw_sentence))

More documentation here.


Val Accuracy not increasing at all even through training loss is decreasing

You start with a VGG net that is pre-trained on ImageNet - this likely means the weights are not going to change a lot (without further modifications or drastically increasing the learning rate, for example).

If you are expecting the performance to increase on a pre-trained network, you are performing fine-tuning. There is a section on fine-tuning the Keras implementation of the InceptionV3 network, but the principals are the same: you should freeze some of the earlier feature-extraction layers, leaving only some of the final layers marked as trainable. This is the example given in the docs, where they add new layers to the base model, train only those layers for a while, then additionally unfreeze some of the base model.

# create the base pre-trained model
base_model = InceptionV3(weights='imagenet', include_top=False)   # <--- leave out final layers!
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
    layer.trainable = False

-> run training for a few epochs

The some time later, unfreeze the part of the base model

# we chose to train the top 2 inception blocks, i.e. we will freeze
# the first 249 layers and unfreeze the rest:
for layer in model.layers[:249]:
   layer.trainable = False
for layer in model.layers[249:]:
   layer.trainable = True

Please looked at the full documentation for more details.


What is the difference between data leakage and endogeneity?

Endogeneity refers to explanatory variables correlated with error term because of a missing variable or measurement error. Data Leakage is introduction of spurious explainability in the model because of bringing in new data than ground truth like synthetic data (say we used SMOTE). In the former case, we will see that the model is biased too much for some feature coefficients whose explainability seeped into the error term. In the latter case we will see that the model will have high variance and lacks generalization which makes it useless for production/test data. Typical example to introduce data leakage is by SMOTING (yeah using the word as a verb) before you do validation split.


Can Neural Networks be trained to smooth values / output the average?

In general simpler models are more robust to noise in the input.  The strength of neural networks is also their biggest 'gotcha' - they are extremely expressive. This means they easily overfit and can be sensitive to noise in inputs.  The strategy of simplifying a model to make it more robust to noise is called regularization.  There are many types:

use smaller hidden layers (i.e. 20 instead of 200 nodes)
use dropout, where during training inputs are randomly set to 0 during training (makes the network more robust to noise overall)
stop training earlier - 'early stopping'
use L1 or L2 regularization, which imposes a cost on the weights

All of these can be done from with Keras.  You want to make your network more robust to noise without decreasing your validation quality.  To do this, I would try the above suggestions in order.  You can measure overfitting by looking at the difference between predictive accuracy on your training data and validation data.  If they are very different - your model has learned structure in your training data that is not in your validation that is NOT what you want.  Try to fiddle regularization nobs until (1) your train-validation AUC or accuracy is very similar, (2) your validation AUC/accuracy is still sufficiently high.


Classification affected by a lot of outliers in features? How do you deal with outliers?

If the number of outliers is small and you are concerned that they will destabilize your solution, you could attempt a random forest classifier. The RF fits trees to random selections of data and variables, and collects "votes" from each, thus reducing the impact of outlier valuers.

On the other hand, if the number of outliers is fairly substantital, you might want to create a new class called "outlier". In the training set, apply this label to those values you have deemed to be outliers and then fit the model with the augmented class. Check if the model correctly identifies outliers in the test set.

This is equivalent to removing the outliers, only it creates a repeatable and machine learning way of doing so.


Is there a way to rank the Extracted Named Entities based on their importance/occurence in a document?

An easy way would be to use TF-IDF (term frequency–inverse document frequency). It can help you find how much terms stand out in a document (by comparing with your entire corpus) and use it to rank your entities.

TfidfVectorizer from scikit-learn

Just note that the TfidfVectorizer is on a word level. So some processing will be needed if your entities can consist of more than one word.

Alternatively you could use a model that allows you to produce a heatmap of the words. Then you can use that heatmap to look up your NEs in that heatmap. This paper, A Structured Self-Attentive Sentence Embedding, could give you some ideas.


Are there any examples other than anomaly detection where unsupervised deep learning could be useful?

Recommendation Systems

In recommendation systems the idea of extracting signal and using to make recommendations is very common, e.g, Alternating Least Squares and Singular Value Decomposition approaches.

Autoencoder fits quite well, reducing dimensionality (the encoder part) should help extract signal. The weights in the network represent the behavior of all the users that we trained on, but we don't want to capture all of it, we want to capture just the most import parts. Size of output of encoder (or bottleneck) controls the amount of dimensionality reduction.

Using it is quite simple:

1) You train the encoder-decoder model on, say, user movie ratings, where user ratings are input and output.

2) To generate recommendations for a user, just pass through user's current ratings through encoder-decoder, and you have the scores that can be used to make recommendations.


imbalanced dataset in text classififaction

I don't now wether I got your question right. But if you count all words within a class, for example, the word "the" is counted everytime it appears. However, if you count the unique words the word "the" is counted once. This is why your counts differ from your plot. Each class can have a different number of unique words.


How can I save a final model after training it on chunks of data?

One option is Dask framework's incremental learning. That would automate the loading of chunks and applying partial_fit in scikit-learn.
The code would be something like:
from dask_ml.datasets import make_classification
from dask_ml.wrappers import Incremental

X, y = make_classification(chunks=25)
estimator = LinearSVC(random_state=10, max_iter=100)
clf = Incremental(estimator)
clf.fit(X, y, classes=[0, 1])

joblib.dump(clf, filename)


Architecture for Machine Learning resource management

It depends on how the cluster (i.e., compute resources) are being managed. For example, Apache Mesos is an open-source cluster management tool.

One established option is Horovod. It is designed to manage workloads for
TensorFlow and Keras across multiple cores in a single machine or across multiple machines.


How fit pairwise ranking models in xgBoost?

According to the XGBoost documentation, XGboost expects:

the examples of a same group to be consecutive examples,
a list with the size of each group (which you can set with set_group method of DMatrix in Python).


This indicates that model is not generalizing (it is over-fitting). Few options are :

Get more training data
Reduce complexity of model (Number of LSTM layers, complexity of dense layers)

Andrew NG has a good video on this topic :

https://www.youtube.com/watch?v=OSd30QGMl88

A tutorial specific to LSTM :

https://machinelearningmastery.com/diagnose-overfitting-underfitting-lstm-models/


Find-out abnormal behavior over the time

Your problem definition

You have time series data which is used to measure the pressure using your sensor. You wish to identify when the pressure recordings are abnormal. This problem would be best solved using anomaly detection algorithms. But, there are so many ways that you can approach this problem.

I would use a sliding window approach and use that as your feature space to detect the distribution of your recordings. The window length you will select $m$ is the first of your hyper-parameters that you will need to tune.

Using a statistical model

The retained time series $X \in \mathbb{R}^m$, you can treat this signal as a queue, first in first out. When a new recording is defined then discard the oldest datapoint. For a given set $X$ of samples, get the mean and the standard deviation $\sigma$. If the new point exceeds a multiple of $\sigma$, usually this is set to $3\sigma$, but it depends on the expected variation of your sensor, then we flag a state change.

You can further extend this by using the generalized likelihood ratio test (GLRT) to determine when a new sample causes the point to fall significantly outside the distribution of the null hypothesis. In which case this indicates a state change (the new point came from a different distribution than the normal flow through the pipe).

Using machine learning

You can collect multiple instances of your time series and annotate them based on your experience as being nominal or anomalous. Then you will have a supervised 2-class classification problem. First you should attempt some feature extraction using PCA, LDA or similar techniques.. Then you can attempt to use all the fun algorithms available to you through scikit-learn (SVM, Random Forests, K-NN, etc.).

If there is a significantly higher number of nominal instances than anomalous ones, this will introduce bias to your model. Anomaly detection algorithms are better suited for these types of problems. These algorithms learn the distribution within which your nominal set should belong. Then for novel instances it evaluates the probability of it being contained in the learned distribution. If the probability is small, then the algorithm will flag the instance as anomalous.

For more information on anomaly detection for time series refer to:

Using time series data from a sensor for ML

How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries


This would be an example of Recommendation based system.
You can build a Popularity based recommendation system in this scenario. This type of algorithm works with trend, it basically uses the items which are in trend right now.

Else you can go for Collaborative recommendation system or Content based recommendation system


What is the difference between NLP and text mining?

I agree with Sean's answer.
NLP and text mining are usually used for different goals.
Also, there is indeed an overlap and both definitions are vogue.

Other than the difference in goal, there is a difference in methods.
Text mining techniques are usually shallow and do not consider the text structure. Usually, text mining will use bag-of-words, n-grams and possibly stemming over that.

In NLP methods usually involve the text structure. You can find there sentence splitting, part-of-speech tagging and parse tree construction. Also, NLP methods provide several techniques to capture context and meaning from text.

Typical text mining method will consider the following sentences to indicate happiness while typical NLP methods detect that they are not

I am not happy
I will be happy when it will rain
If it will rain, I'll be happy.
She asked whether I am happy
Are you happy?


implementing temporal difference in chess

I recommend anybody interested in the topic take a look at the paper which combines TDL and deep-learning.

Roughly, you'll need to make the engine play games against each other. Record the mini-max evaluation for each position. At the end of the game, you'll get a reward, which is {0,1,-1} for chess. Then you'll need to adjust your parameters with:

This equation tells us we should adjust the weights by temporal differences, weighted by how far you should do it. If you have a perfect evaluation, your temporal difference would always be zero, thus you wouldn't need to make any adjustment.

Next, you'll need use the new parameters to play a new game. Repeats until as many games as you can afford or when you think it's converged.

Few remarks:

The paper I quote applies a discount factor. It's done for the backproportion algorithm for neural network. You don't need it.
You'll need to experiment with the optimal learning rate (alpha in the equation). Too large will make your learning unstable, too little will take longer to converge. I've seen people using 0.70. The paper I quote used 1.0.


Can sampling like SMOTE/UP/DOWN applied on Validation set?

Oversampling on your test set will only artifically improve your performance. What you may want to do instead is changing your objective function to give more importance to you imbalanced class. There are already a lot of question about class imbalance on this website, such as : Classification problem: custom minimization measure


Machine learning techniques for estimating users' age based on Facebook sites they like

One thing to start off with would be k-NN.  The idea here is that you have a user/item matrix and for some of the users you have a reported age.  The age for a person in the user item matrix might be well determined by something like the mean or median age of some nearest neighbors in the item space.

So you have each user expressed as a vector in item space, find the k nearest neighbors and assign the vector in question some summary stat of the nearest neighbor ages.  You can choose k on a distance cutoff or more realistically by iteratively assigning ages to a train hold out and choosing the k that minimizes the error in that assignment.

If the dimensionality is a problem you can easily perform reduction in this setup by single value decomposition choosing the m vectors that capture the most variance across the group.

In all cases since each feature is binary it seems that cosine similarity would be your go to distance metric.

I need to think a bit more about other approaches (regression, rf, etc...) given the narrow focus of your feature space (all variants of the same action, liking) I think the user/item approach might be the best.

One note of caution, if the ages you have for train are self reported you might need to correct some of them.  People on facebook tend to report ages in the decade they were born.  Plot a histogram of the birth dates (derived from ages) and see if you have spikes at decades like 70s, 80s, 90s.


Find columns with numeric values, but stored as string

I think this does what you want.

for column in data_set:
    If isinstace(data_set[column].dtypes,str)
        data_set[column]=data_set[column].astype(int)


Each channel of rgb colors has values between 0 and 255.
There are many normalization methods. For this case I would just divide the images over 255 without subtracting the mean so that I make sure the output is between 0 and 1

Z = X/255

if you are interested in centering the images around zero, you can subtract 127 (as a mean) and then divide over 255

Z = (X-127)/255


Is there any measure to find how much classifiers are confident?

There are many measures which implicitly take into account the confidence of a prediction. One very common one is Log Loss (also called Cross Entropy).

$-log \space P(y_t|y_p) = -(y_t \space log(y_p) + (1-y_t)\space log(1-y_p))$

Using this metric, confident correct classifications are rewarded more than relatively less confident correct classifications, and confident misclassifications are heavily punished.

Any proper scoring rule meets this criteria when applied to a classification problem.

Some others examples are:

Surprisal
Brier score (as mentioned by darXider in his comment)
Spherical Scoring Rule
Logarithmic Scoring Rule

And an infinite number of other, related functions.


How feed a numpy array in batches in Keras

You should implement a generator and feed it to model.fit_generator().

Your generator may look like this:

def batch_generator(X, Y, batch_size = BATCH_SIZE):
    indices = np.arange(len(X))
    batch=[]
    while True:
            # it might be a good idea to shuffle your data before each epoch
            np.random.shuffle(indices)
            for i in indices:
                batch.append(i)
                if len(batch)==batch_size:
                    yield X[batch], Y[batch]
                    batch=[]

And then, somewhere in your code:

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)

UPD.:
I order to avoid placing all your data into memory beforehand, you can modify the generator to consume only the identifiers of your data-set and then load your data on-demand:

def batch_generator(ids, batch_size = BATCH_SIZE):
    batch=[]
    while True:
            np.random.shuffle(ids)
            for i in ids:
                batch.append(i)
                if len(batch)==batch_size:
                    yield load_data(batch)
                    batch=[]

Your loader function may look like this:

def load_data(ids):
   X = []
   Y = []

   for i in ids:
     # read one or more samples from your storage, do pre-processing, etc.
     # for example:
     x = imread(f'image_{i}.jpg')
     ...
     y = targets[i]

     X.append(x)
     Y.append(y)

   return np.array(X), np.array(Y)


Select more or less features if results are almost the same

It is very contextual. There's a rule that feeding all the data will yield better (unbiased) results but not at all times. If you know anything about the data, then it is always advised to remove the most obvious ones that won't matter. Additionally, you can use other metrics like 'P-Value' to check the variable significance.

Since there's more data that will be added in the future, I'd rather wait for it and then remove the insignificant variables.


This is a kind of confirmation bias, it's often found in recommender systems and it's hard to beat: since the system is designed to find similar books, it's normal that it returns the most similar books but it can be disappointing for the user. This bias is especially strong if there are few past books to refer to, i.e. the reference sample used by the model is too small to represent the user's preferences.
As Nikos suggested in a comment, one way to diversify the recommendations is to broaden the set of similarity features. A more advanced way is to use some second-degree approach: the system doesn't only recommend books similar to a specific user's past, but finds other users who liked the same books and recommends books that they liked (similar to "other users who like X also like Y").


To continue training an existing tensorflow model do I need to restore the session?

I managed to get it workng with just load_model and then continue with model.fit(...). My model is quite simple, using just Keras Input, LSTM and Dense layers with no variables or anything complex.


In recommender systems, how to avoid recommending a product that the user has just bought?

Interesting, so yeah I assume you can find out the time between purchases within category from your data. Some random ideas:

The simplest answer I can think of is to find the mean time per category and use that as the threshold. Empirically, it's the point at which customers are more likely than not to have bought again.

That's fairly binary, just removing all recommendations from a category. You could imagine, say, keeping 20% of the top recommendations in category A when the cumulative probability of enough time having elapsed is 20%. Or factoring it into a scoring function, and multiply by this probability and then rerank. (That is, there's an 80% chance not interested at all, and a 20% chance interested as usual.)

To do that you'll probably need to fit a distribution, and Weibull seems attractive. As a generalization of exponential, which can be thought of time between events like purchases where the rate is constant, it can accommodate a rate that increases over time. That is, the rate of repurchase starts off low and increases.

Both of those are too pessimistic as you more want to assess when they're as willing as ever to buy again, not so much buy again, which may be pretty infrequent in your data. Perhaps you model the rate of purchase per category among those who have and have not purchased in the category and empirically figure out when the rate goes back to similar.


Significance of Number of Calls and Reset Call in Ball Tree

Indeed, they don't seem to be used for anything.  Probably they were included for some initial debugging?
An issue (and a linked PR) discussing these methods:
https://github.com/scikit-learn/scikit-learn/issues/13330
(At time of posting,) searching for n_calls only finds results in _binary_tree.pxi, and none of them seem actually used for anything other than tracking the number of distances computed:
https://github.com/scikit-learn/scikit-learn/search?q=n_calls


How to build the following the string in R?

The way R displays the string (using print command) is different from the actual value of this string. Even though you see 2 backslashes for your output, the actual string contains a single backslash. This is because when you type the name of your variable it shows the value within the quotes and within quotes it should use 2 backslashes. When you display the value of the sting using message() or other similar function, when it just prints the value without quotes, then you will see a single back slash only:

#create a vector of names (words)
words <- c("word1","word2","word3")

#add prefix and suffix part to each word
words_vec <- paste0("\\'",words,"\\'")
words_vec
"\\'word1\\'" "\\'word2\\'" "\\'word3\\'"

#Combine the vector into the string
line <-paste(words_vec,collapse='')
line
"\\'word1\\'\\'word2\\'\\'word3\\'"

#Check how the string looks:
message(line)
\'word1\'\'word2\'\'word3\'

By the way, the above (vectorized) method is more efficient than the for-loop method you are using to put all names together.


I am not sure there is a clear answer to this, especially as the problem does not seem to be well-defined right now - your "figure" seems to indicate edge weights but you then mention node weights, something significantly different.

If the question is whether you can find a way to split a graph into two smaller modules, then you might want to look into applying Sparsest Cut techniques - a cut with low cost would imply (?) high modularity. I believe these can be easily modified to account for either unlabeled, edge-labeled or node-labeled graphs.


Is not having overfitting more important than overall score (F1: 80-60-40% or 43-40-40)?

The best model in your case is B not just because it scored higher on the test set, but because it showed very little sign of overfitting.
By not overfitting as much and being more consistent in its scores (train/val/test), you know what you're getting from the model. If you try it tomorrow on a secondary test set, you'd expect similar results.

Model A on the other hand is very inconsistent. If you evaluated this model on the secondary test set, you can't tell how much to expect. It could be higher, it could be lower... Generally speaking, if you've overfit as much on the validation set, it's a good indication to start over (re-split the dataset into train/val/test randomly, preprocess, fit the model). This time try less hyperparameter options though.

From the numbers I see even though model B currently is better than A, A might have the capacity to be better than B, if properly trained. I'd suggest re-doing model A's training from scratch, but with more regularization and less hyperparameter-tuning.


How to use two different datasets as train and test sets?

You may want to use a pipeline to do this operation. Specifically, you do NOT want to train the TFIDFVectorizer the entire corpus- doing so gives your model hints about what features may be in the test set that don't exist in the training set- a concept frequently referred to as "leakage" or "data snooping".

The correct pattern is:

transf = transf.fit(X_train)
X_train = transf.transform(X_train)
X_test = transf.transform(X_test)

Using a pipeline, you would fuse the TFIDFVectorizer with your model into a single object that does the transformation and prediction in a single step. It's easier to maintain a solid methodology within that pattern.

In the example code, you're both fitting and transforming in the same step fit_transform, which is creating different features each time and is the source of your error.


How to build a machine translation system for a new language

I think your case would benefit from tunning a existing language to the new one but that is only a good approach if you plan to use it commercially. Also Google accepts help to improve their translation algorithm and you could petition for them to assemble a team for this and donate data.

Google uses a Neural Network that they call Google's NMT (Neural Machine Translation) for translation and it works as a encoder-decoder pair. You can read more on their paper.

Also, Google's NMT is available in tensorflow and can be trainned and improved. It is license under Apache License 2.0 and has a nice tutorial and explanation available on their GitHub:

Check on this GitHub folder here


Should I initiate size before loop and append in Python

Yes it will be just as slow as in matlab. You’re going to be copying the whole data frame every time you append something.


Using machine learning specifically for feature analysis, not predictions

You don't need the linear regression to understand the effect of features in your random forest, you're better off looking at the partial dependence plots directly, this what you get when you hold all the variables fixed, and you vary one at a time. You can plot these using sklearn.ensemble.partial_depence.plot_partial_dependence. Take a look at the documentation for an example of how to use it.

Another type of model that can be useful for exploratory data analysis is a DecisionTreeClassifier, you can produce a graphical representation of this using export_graphviz


I do not understand the next line, the cost. Shouldn't the cost be equal to the xent, i.e. the loss? What is the cost function representing here?

The cost is the error (xent.mean()) + some regularization (0.01 * (w ** 2).sum())

  Why is the bias initially set to 0 and not a random number like the weights?

It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights.

More details here.


Is it possible to train a machine learning trainer?

I would try using a Genetic Algorithm to estimate optimal parameters for training. You would need to figure out the objective function of parameters of training algorithm $A_2$ which minimalised fitting $A_1(D)$. Those parameters could be regularization coefficients or numer of layers/neurons, or any other parameters describing your training algorithm.

There is also algorithm called bayesian regularization. It is very similiar to Levenberg - Marquardt training, but it is computing optimal regularization parameters $\alpha$ and $\beta$ by computing effective numbers of network's parameters.

I hope I understood your question well.
Regards, Max


NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need)

So the question is concerned about understanding the self-attention mechanism in greater detail, in particular how this idea of multi-head self-attention is used to compute strength of relations between tokens.
I think it's best you look through this great tutorial on self-attention and see if this helps in your understanding of multi-head self-attention: http://www.peterbloem.nl/blog/transformers


Using HashingVectorizer for text vectorization

I was recently checking some things out. Thought would leave a working code here, in-case its helpful.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction import FeatureHasher
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics
import numpy as np

categories = [
    'alt.atheism',
    'talk.religion.misc',
    'comp.graphics',
    'sci.space',
]
newsgroups_train = fetch_20newsgroups(subset='train', shuffle=True,
categories=categories, random_state=91)
newsgroups_test = fetch_20newsgroups(subset='test', shuffle=True,
categories=categories, random_state=91)

vectorizer = FeatureHasher(input_type='string')
X_train = vectorizer.fit_transform(newsgroups_train.data)
X_test = vectorizer.fit_transform(newsgroups_test.data)

Y_train = newsgroups_train.target
Y_test = newsgroups_test.target
print(X_train.shape)
print(Y_train.shape)
print(X_test.shape)
print(Y_test.shape)

rf = RandomForestClassifier(n_jobs=-1, n_estimators=100)
rf.fit(X_train, Y_train)
pred = rf.predict(X_test)

score = metrics.accuracy_score(Y_test, pred)
print("accuracy: {:.3f}".format(score))


How/What to initialize the hidden states in RNN sequence-to-sequence models?

It is important to clear up the difference between hidden state initialization and weight initialization. Glotrot (Xavier), Kaiming etc. are all initialization methods for the weights of neural networks.

Since your question is asking about hidden state initialization:
Hidden states on the other hand can be initialized in a variety of ways, initializing to zero is indeed common. Other methods include sampling from Gaussian or other distributions. In relation to RNN's this defines what a RNN starts with as its 'memory'. Two common approaches seem to be either a noisy initialization (from some sort of distribution or a random number generator), or a learned initialization.

To synthesize the link above; initializing hidden states with zeros can lead to the network learning to adapt from a zero hidden state, rather than minimizing the loss for a long sequence (it follows that this is more of a problem for short sequences). If there are enough sequences it can make sense to have the initial state be a trained variable that is a function of the error during back propagation.


It would appear that you have a data processing task. You could use two columns 'A Occurrence' and 'B Occurrence'. If that value occurs for the record that index would contain a 1. If that value does not occur for that record the index would contain a 0.

Example:

X       | A Occurrence | B Occurrence
A       | 1            | 0
A and B | 1            | 1


why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding

From the blog, one hot encoding resolves the issue by explicitly showing that 1 category is true, while all others are false (1 vs 0). One column is turned into 3 columns which describe all categories as either true or false.

As opposed to letting a model see one column with 0, 1, and 2. One column to show this data with label encoding does make it seem as if the data is numerical so 0 < 1 < 2.


Could data from a test set 'leak' into predictor during PCA?

The author is right.  You are wrong.  There's no such thing as "only metadata leak" or "can't leak really".  It's like saying "not pregnant really" -- either you're pregnant, or you're not.  Same here -- either data can leak, or it can't.  In this case, data can leak.  Maybe only partial data, but it's hard to know just how partial and just how bad the impact of that might be.  It might be very bad, or it might not; but since you can't know, testing with a methodology where you know the results might be meaningless isn't a good idea.


How can I use octave function minimizers in c++?

You can use a native library in C++ for optimization. Check out this one :
dlib


Do I correctly understand, that the test data is the whole dataset, whereas training is only a subset of it? Training and test data must not overlap. The test is a measure of quality on unseen, unfamiliar data.
In the case of inbalanced data and two class classification the naive classifier, predicting always the most probable class has the quality 891 / 1121. Any sensbible model should beat this score.
To handle inbalanced data you can use several approaches - undersampling the majority class, oversampling the minority https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/. Also many classifiers have the attribute weights, which can be set to balanced - this would penalize larger for mistakes on minority class.
In the case of imbalanced data measure not only accuracy, but precision and recall as well https://en.wikipedia.org/wiki/Precision_and_recall


Yes, there are open source examples. Take a look at here and here.
About your second question, yes. There are numerous studies. For instance, take a look at Supervised Representation Learning: Transfer Learning with Deep Autoencoders.


Biasing SVM algorithm towards particular subset of data

I don't think that's a very good idea: the goal is not to make the model predict a more extreme polarity when the tweet relates to the company.
Instead you might want to consider oversampling the few instances of this specific company. For instance if you have 100 company-specific tweets and 1000 general tweets in your training set, you could duplicate the company-specific ones 10 times in order to give the specific tweets have a higher weight in the data. If possible you should tune the parameter of how many times to duplicate in order to obtain the optimal value.


Why to adjust class weights instead of simply finding the best threshold?

If there are enough samples of both classes, I don't think it makes a lot of sense. I've been in kaggle competitions with very imbalanced datasets, like:

Fraud detection
Home default risk
Quora insincere questions

And none of the top solutions used any kind of treatment of the imbalance, as there were enough samples for both classes.
I've also done myself, several fraud-related models with high imbalance, and using no imbalance solution proved better than all the other options.


what is sentence embeding and how to do sentence embedding for a sentence and how to use word embedding to create a sentence embedding?

It could be just a simple average of word vectors; words that are a member of the sentence. However, there are other methods that could work better. Please see this article:

In the paper Unsupervised Learning of Sentence Embeddings using Compositional N-Gram Features, a new model for sentence embeddings called Sent2Vec is introduced. Sent2Vec presents a simple but efficient unsupervised objective to train distributed representations of sentences. It can be thought of as an extension of FastText and word2vec (CBOW) to sentences. The sentence embedding is defined as the average of the source word embeddings of its constituent words. This model is furthermore augmented by also learning source embeddings for not only unigrams but also n-grams of words present in each sentence, and averaging the n-gram embeddings along with the words.


I have just used LSTM to train a model predict time series value and get good result as below:

# reshape input to be [samples, time steps, features]
trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))

model = Sequential()
model.add(LSTM(40, input_shape=(trainX.shape[1], trainX.shape[2])))
model.add(Dense(1))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(trainX, trainY, epochs=10, batch_size=1, verbose=0)

Here is my result: blue is real value, orange is predict.


Hypertune xgboost to dealing with imbalanced dataset

Given an imbalanced dataset and focusing on increasing your true positive rate, it is quite relevant to use the right evaluation metric (the one used to validate the model being trained on each evaluation round).
In this case, I recommend you use Precision-Recall AUC instead of ROC AUC, so you force your model to focus on the minority class.
A nice post about it can be found here
Another points to take into account could be:

increase the range of possible number of tree estimators in your hyperparameter tuning process
set the scale_pos_weight at about your (majority class samples number)/(minority class samples number)


State of the art for Object detection/image recognition

Maybe.

Do they have a unique feature on the item  (e.g. a polo player, a red sole) the model can be trained on? I have seen people try identifying clothing model from images via Google's Vision API and it didn't work well. If you are in a fashion category where the business model is to copy designs like H&M or Zara do, you're likely going to have trouble.


Yes, the loss of a normal autoencoder is simply the difference between the input image and the decoded image.
While encoder and decoder have different names, they are effectively part of the same neural network, so the prediction error is backpropagated through the decoder up to the encoder.
In more sophisticated autoencoders, like variational autoencoders, the loss may present extra terms, e.g. to enforce some structure in the output of the encoder.


What are some of the best practices for sharing data and models with colleagues?

You can try using dvc, which stands for data version control.
https://dvc.org/


This link contains an amazing amount of deep learning literature.
Summarizing it here(going in the order a beginner ideally should)-
NOTE: All these resources mainly use python.

1) First of all, a basic knowledge of machine learning is required. I found Caltech's Learning from data to be ideal of all the machine learning courses available on the net.

Andrew Ng's Coursera course is pretty good too.

2) For Neural networks, nobody explains it better than Dr.Patrick Winston.
The assignments should be tried out for better understanding. They are in python.

3) For a better understanding of Neural Networks, Michael Nielsen's course should be done(as suggested by Alexey). It is pretty basic but it works.

4) For deep neural networks, and implementing them faster on GPUs, there are multiple frameworks available, such as Theano, Caffe, Pybrain, Torch,etc.
Out of these Theano provides a better low level functionality that allows its user to create custom NNs. It is a python library, so being able to use numpy,scikit-learn, matplotlib, scipy along with it is a big plus.
The deep learning tutorial written by Lisa Lab should be tried out for a better understanding of theano.

5) For Convolutional Neural Networks, follow andrej karpathy's tutorial.

6) For unsupervised learning, follow here and here.

7) For an intersection of deep learning and NLP, follow Richard Socher's class.

8) For LSTMs, read Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780 and Graves, Alex. Supervised sequence labelling with recurrent neural networks. Vol. 385. Springer, 2012.

Here is LSTM's Theano code.


You actually convert the output of your algorithm from continuous to categorical.

I see many reasons of why you would want to do that. A simple case for this would be when you have very long time-series data that take a lot of space to be saved. In this case, it is convenient to convert the timeseries in a histogram representation, which actually is a number of predefined bins where all the values fall into. This will greatly reduce the memory requirements of your algorithm because you just increase the count of a bin (integer number) when a measurement is taken that falls in it, instead of storing the measurement itself as a float number.

This of course has pitfalls, such as losing the "time" dependencies between the measurements as well as reduction in resolution of your dataset. BUT, it can be really useful when forecasting measurements because instead of having the continuous R space as output of a predictor, you actually have (e.g.) 10 bins (classes) that the next measurement is predicted to fall into, making it easier for your model to be trained.

FYI, this is exactly what is done in the automotive industry.

EDIT - (Added sources)

Sources:

https://people.dsv.su.se/~tony/papers/dmin_2015.pdf (check the algorithm)

https://www.phmsociety.org/node/2284 (the previous algorithm is applied here for trucks)

https://link.springer.com/article/10.1007/s10618-017-0538-6 (another algorithm based on histograms of truck data)


If you look at the 2nd equation under "Propagation function"

$p_{j}(t)=\sum _{i=1}^No_{i}(t)w_{ij}+w_{0j}$

The $w_{0j}$ is the constant (bias), so it's reasonable to write that the $w_{ij}$ weights are associated with the connections, and the $w_{0j}$ is a weight associated with the nueron itself.

Additionally if you assume $o_{0}(t)=1$ then you can rewrite the equation above as

$p_{j}(t)=\sum _{i=0}^No_{i}(t)w_{ij}$

(i.e. note the sum is from 0 to N, not 1 to N)

When you do this it no longer really makes sense to make a distinction between a "weight" and a "bias".