Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?

credit for this answer goes to @timleathart in the comments above:

You can get the same effect very efficiently in pandas by using df['your column name'].astype('category').cat.codes. Be careful when doing this, because it implies an ordering on the columns. Your learning algorithm might overfit to spurious relationships that appear in the training data


How to get the mu+0.1sigma from a normal distribution?

From your question, I imagine you are interested in:

$\alpha = P[\mu - k \, \sigma \le x \le \mu + k \, \sigma]$

where, in your case, $k = 0.1$. In other words, by "confidence level" you mean alpha.

In R, you can obtain that via function pnorm (empirical cumulative distribution function for the standardized normal distribution):

alpha <- 2 * (pnorm(k) - 0.5)

which, for k = 0.1 produces 8%.


Here in your Scenario you need to select the one with more Information Gain rather than the least one and the process goes on till you reach the last feature/last node.

Go through this Link. I think your doing it vice versa, I agree with Emre.

In the above link it was explained with an example to decide whether to play tennis or not.

Do let me know if you have any issues.


If you still have the question I hope my answer helps you. From what I understand you want to fit a RL model to participant's data who did a decision making task. The whole process of fitting RL models to data is described in great detail in Trial by Trial analysis using RL by Nathaniel Daw.

Briefly, what you observe (your data) is just actions and rewards at every time step. If the participant is using a RL model you should expect to update the values of his actions according to:

$Q_t(a_t)=Q_t(a_{t-1})+\alpha\cdot (r_t-Q_t(a_{t-1}))$

and action selection with the Boltzmann function:

$p(a_{t}|s_{t})=\frac{e^{\beta Q(s_{t},a_{t})}}{\sum_{a'}e^{\beta Q(s_{t},a')}}$

We prefer to use a stochastic policy as it simulates better human choices and it is straightforward to create a likelihood function. The likelihood function would be the product of all the probabilities $p(a_t|s_t)$. The model parameters are $[{\alpha,\beta}]$ and these can be inferred by standard Maximum Likelihood techniques.

Finally, our hypothesis (=model) can be summarized as: A subject generates a sequence of (observed) actions. To generate these actions we assumed that uses a RL model with some parameters. We want to fit the model to the data to infer these parameters and test the suitability of the model and the cognitive/psychological links of the values of the parameters.


How much data should I allocate for my training and and test sets? (in R)

There is no "golden rule" here. Your data set is very handy - neither too large nor too small. Sounds like a very exciting project!

Here is how I often proceed in comparable settings.

Do all splits stratified by response or, if the rows are not independent but rather clustered by some grouping variable (e.g. the family etc), grouped sampling. Important rule: avoid any leakage across splits.
Set aside 10%-15% of rows for testing. Don't touch them until the analysis is complete. Act as you would never utilize this test set.
Select loss function and relevant performance measure.
Fit a random forest without tuning and use its OOB error as benchmark.
Choose parameters of XGB by 5-fold cross-validation iteratively by grid search, first starting with very wide parameter ranges and then making those ranges smaller and smaller. The number of boosting rounds are automatically optimized by early stopping.
Choose model and present cross-validation performance.
At the very last, reveal test performance.


What kind of algorithm should I choose for this music classification system?

There are two high level approaches (Approach 2 was a better fit for a music-classification problem that I worked on) :

Signal processing + CNN  : Output of signal processing is saved as image. Models use the image as input.

This paper is a good intro to the approach : https://arxiv.org/ftp/arxiv/papers/1712/1712.02898.pdf

Couple of articles on this : https://www.codementor.io/vishnu_ks/audio-classification-using-image-classification-techniques-hx63anbx1 , https://medium.com/datadriveninvestor/audio-and-image-features-used-for-cnn-4f307defcc2f

Raw audio  + recurrent networks : https://deepmind.com/blog/wavenet-generative-model-raw-audio/  ,  https://arxiv.org/pdf/1606.04930.pdf , https://arxiv.org/pdf/1612.04928.pdf , https://gist.github.com/naotokui/12df40fa0ea315de53391ddc3e9dc0b9

GPU will make the project easier, but is not a requirement.


For a while I thought you were right, but the equation is correct.

If you think about it conceptually,  then I think you will see that it has to be correct. When updating $w_{ij}$, you have to make the change that improves the total error $E$, and not just the error $E_k$ of output node $k$, right? So you have to be looking for the weights that give you the lowest $E$, that is, where $\partial E / \partial a_k$ is 0.

If you write out the sum over $k$ in the equation for $E$, you get one term for $a_1$, one for $a_2$, and so on. The sum over $k$ in the equation that confuses you, just makes sure to first get the $a_1$ term from $E$, then the $a_2$ term, and so in. It first differentiates $E$ to $a_1$, which is $t_1 + a_1$ if I did that correctly, and gets the correct tion for that. Then for $a_2$, until the end, and finally you have to sum over $k$ to add up the corrections from all terms.


Custom loss for low false positive rate (higher precision)

You have two options here that I can see:

1) quick and very very dirty - rebalance your data so that negatives are favored in the training data. Then your model will prefer to predict negative rather than positive, and false positives will be suppressed. To see this, consider what happens if all training data were negative. Tune the data balance until you get the desired result (I feel like a bad person for saying this...)

2) use a custom loss function such as the $F_\beta$ score (https://en.wikipedia.org/wiki/F1_score). However, this may be numerically tricky to implement for training as the gradient of the naive function generally behaves badly.


Dropout on inputs instead on outputs = DropConnect?

It depends on the type of input pattern but to make a decision, I suggest not to. There are different reasons for that. First of all, you are damaging your input signal. I don't know whether you are familiar with the information theory or not but the signal to noise ratio will be too small and if you do so, you will be left with a signal which is far from your real signal.

Moreover, it is also not good to add dropout in the convolutional layers because they are feature extractors and they are significant features for classification problems. If you miss them, means that you are losing information more than usual. Consider the point that your input to the network is already resized to a smaller shape than its original shape, for instance, the input shape of typical CNNs is 224 * 224 while the original shape may be ten times bigger or even more  for each direction.

You may have seen that in the Lenet-5 the authors have used a data-augmentation technique that changes the colors of the inputs with different distributions. The point there is that they have not changed the locality of the input signal. Moreover, the signal to noise ratio also is not a too small number due to the fact that they have not set the input features to zeros. They just have changed them slightly.

Finally, the last layer should not employ drop-out. Because the output size has to have specified characteristics, sum to one. Dense layers due to having a large number of weights, and consequently a large number of activations, are good points for exploiting drop-out.


How to measure correlation between several categorical features and a numerical label in Python?

If you want to do an ANOVA test, you can do it with scipy and stats package. Link to documentation

You can do it like that

def anova(data):
    if len(data.groupby(level=1)) <= 2:
        raise Exception('ANOVA requires a secondary index with three or more values')

    return pd.DataFrame(
        [f_oneway(*[v for k, v in data[col].groupby(level=1)]) for col in data.columns],
        columns=['f', 'p'],
        index=data.columns)

Source

or just choose the two columns you want to test

statistic, pvalue = stats.f_oneway(data['col1'], data['col2'])

However, I would advise you to take a different path. You need to test how important a feature is in your dataset to predict the lead_time. You can train a simple Decision Tree with the whole dataset and get the feature importance for each of the features. Then you remove those that are below a threshold.

from sklearn.tree import DecisionTreeRegressor

tree = DecisionTreeRegressor().fit(X, y)
tree.feature_importances_


Keras difference beetween val_loss and loss during training

val_loss is the value of cost function for your cross-validation data and loss is the value of cost function for your training data. On validation data, neurons using drop out do not drop random neurons. The reason is that during training we use drop out in order to add some noise for avoiding over-fitting. During calculating cross-validation, we are in the recall phase and not in the training phase. We use all the capabilities of the network.

Thanks to one of our dear friends, I quote and explain the contents from here which are useful.

  validation_split: Float between 0 and 1. The fraction of the training
  data to be used as validation data. The model will set apart this
  fraction of the training data, will not train on it, and will evaluate
  the loss and any model metrics on this data at the end of each epoch.
  The validation data is selected from the last samples in the x and
  y data provided, before shuffling.

  validation_data: tuple (x_val, y_val) or tuple (x_val, y_val,
  val_sample_weights) on which to evaluate the loss and any model
  metrics at the end of each epoch. The model will not be trained on
  this data. This will override  validation_split.

As you can see

fit(self, x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None, validation_split=0.0, validation_data=None, shuffle=True, class_weight=None, sample_weight=None, initial_epoch=0, steps_per_epoch=None, validation_steps=None)

fit method used in Keras has a parameter named validation_split, which specifies the percentage of data used for evaluating the model which is created after each epoch. After evaluating the model using this amount of data, that will be reported by val_loss if you've set verbose to 1; moreover, as the documentation clearly specifies, you can use either validation_data or validation_split. Cross-validation data is used to investigate whether your model over-fits the data or does not. This is what we can understand whether our model has generalization capability or not.


Identify outliers for annotation in text data

The idea is to find the documents which are not well represented in the current labeled data. The first point is indeed a bit vague and can probably be interpreted in different ways. My interpretation would be something like this:

For every document $d_u$ in the unlabeled data, count the number of words in common with every document $d_l$ in the labeled data. This value is the "match score" between $d_u$ and $d_l$.

Note: I think that this value should be normalized, for example using the overlap coefficient. Note that other similarity measures could be used as well, for instance cosine-TFIDF.

As output from the above step, for a single document $d_u$ one obtains a "match score" for every labeled document. The average across the labeled documents gives the "average match" for $d_u$.


Interesting question! I'm assuming you have a bunch of points $(X, Y, Z)$ plotting the lung you want to measure.
So your question is actually the measure of a convex hull.

Checkout scipy.spatial.ConvexHull. The ConvexHull object has a volumn attribute which indicates the column of the convex hull you want. It supports N dimensions, which your question is about three dimensions.


what information can we obtain from t-SNE?

The aim of dimensionality reduction is to preserve as much of the significant structure of the high-dimensional data as possible in the low-dimensional map.

Dimensionality reduction yields a more compact, more easily
interpretable representation of the target concept, focusing the
user’s attention on the most relevant variables when you don't know
anything about data.

Having said that Dimension Reduction will not help if you have a less features  and only some are highly correlated as you can just use that correlated features for model so it all depends on data.
Unlike PCA, t-sne uses the local relationships between points to create a low-dimensional mapping. This allows it to capture non-linear structure.
T-sne also solves the Crowding Problem making the optimization "spread-out" the medium distance points to prevent crowding.
To understand more about t-sne plot refer here


Data science projects explained step by step?

If you want an application-oriented book, consider Christopher Bishop's Model-Based Machine Learning. He has more technical books that are well regarded.
If you are looking for lots of code, Probabilistic Programming & Bayesian Methods for Hackers is an option.
Another introductory book with a more statistical bent is An Introduction to Statistical Learning with Applications in R. Again, the authors have a well-regarded technical version of the book.


Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?

I think the answer might be this:

The main difference between Python -and R- and SQL is that the first are Turing complete.

A given programming language is said to be Turing-complete if it can be shown that it is computationally equivalent to a Turing machine.

That is, any problem that can be solved on a Turing machine using a finite amount of resources (i.e., time and tape), can be solved with the other language using a finite amount of its resources.

Typically, one proves a given language is Turing-complete by providing a recipe for translating any given Turing machine program into an equivalent program in the language in question. Alternately, one can provide a translation scheme from another language, one that has already been proven to be Turing-complete.

Nearly every existing computer language is Turing-complete. About the only computer languages that aren't Turing-complete are a handful of special languages that are capable of LESS than a Turing machine - usually because some limitation is "hard-wired" into the language's structure or definition. (For example, Hofstadter's designed a language called BlooP so that it was impossible for it to have a iteration structure with an arbitrarily high upper bound.)

Turing Completeness definition

The question of yours is: Why we don't use SQL (or another DBMS) to do all the Data Science process? The answer is that because SQL lenguage is limited in certain aspects that Python and R are not. And one of those aspects is key to Data Science: Recursion.

Just imagine Decision Trees and all their variants without recursion! They are impossible!

Is SQL Turing Complete?


Machine Learning with sometimes missing data

There are couple of ways to deal with missing data.

Replace missing values by mean/median. If the missing values is very
less, then this method would be apt. Also depends on how skewed
your data is.
Imputation. Build a linear regression model to predict
the missing values based on other parameters. KNN could also be used
to predict the missing value

Also there are other methods to deal with missing data such as mimicking the parameters, removing missing data, etc.


High dimensional space is dense or sparse?

Data in a high dimensional space tends to be sparser than in lower dimensions.  There are various ways to quantify this, but one way of thinking that may help your intuition is to start by imagining points spread uniformly at random in a three dimensional box.  Now flatten the box into a square, pushing two opposite sides together so all the points lie on a single plane.  Do you see that the average distance between a point and its nearest neighbor is now smaller?  Now flatten the square into a line segment.  Do you see that the average distance between a point and its neighbors is now smaller still?

There is no conflict between this and saying that the average distance between any 2 points in the high dimensional space tends to be similar.  The latter statement doesn't imply density.  The real number line is dense (it has no gaps), and yet the distance between points ranges from 0 to infinity.  The point is that the higher the dimension of your space, the more likely the points are to lie near the edges of the space rather than the center.

Again, consider the dimensions we can actually see.  Consider a circle with radius=1, inscribed in a square with sides of length=2.  The circle occupies $\pi / 4$ of the square's area, about 78.5%.  Now consider a sphere of radius=1 inscribed in a cube with sides of length=2.  The sphere occupies $\pi / 6$ of the cube's volume, about 52.4%.  As you see in this example, the odds of a randomly placed point lying close to the center (with close to the center in this case meaning within the circle or the sphere) is lower as the dimension increases.  Points are more likely to be in the corners.  This is why in high dimensions the distance between the points tends to be similar - because randomly placed points tend to be close to the edges of the region.


What is the most straightforward way to visualize color-coded clusters along with the cluster centers?

There can be multiple ways, one can be -
- Plot the points with hue=cluster_number
- Plot the Centroid with a different markers
Code for 3 Clusters on 2 Iris Features -
from sklearn import datasets
iris = datasets.load_iris()
X = iris.data
y = iris.target
X=X[:,:-2]
X = (X - X.mean())/X.std()

def create_cluster(k=3):
    from sklearn.cluster import KMeans
    kmeans = KMeans(n_clusters=k)

    kmeans.fit(X)
    return kmeans

kmeans = create_cluster()
y_pred = kmeans.predict(X)
centroid = kmeans.cluster_centers_

_, ax = plt.subplots(1,1,figsize=(10,6))
color = ["#e74c3c", "#34495e", "#2ecc71"]
sns.scatterplot(X[:,0],X[:,1], hue=y_pred, palette=sns.color_palette(color),ax=ax)
sns.pointplot(centroid[:,0], centroid[:,1],markers='^',join=False,ax=ax)

$\hspace{2cm}$Output - Ignore the convergence quality
$\hspace{2cm}$


How should I improve my Vectorized Gradient descent linear regression model?

It seems like number of iterations is very small compare to learning rate. (50 vs 0.001). The optimizer can't converge when we look at the fitted line plot.

Try to increase number of iterations, it will help you. You can check this beautiful work and my another answer about the problem.


distinguish users for recommender system

I was interestedin this problem awhile back ago.  I still have this paper and should serve as a good primer.

Guess Who Rated Thos Movie:...


sort occurrence matrix to minimize its spatial variance

In case someone got the same concern, https://gmarti.gitlab.io/ml/2017/09/07/how-to-sort-distance-matrix.html


Why does test data need to be normalized on train data mean and std?

The reason you split your dataset to Training and Test is to simulate real-world cases. What you actually do with the train-split validation is to evaluate your model in unknown data.

Imagine now that you have trained your model and you are on a production where new data keep coming for prediction. You might not get them in mass, but one by one such as in an API call. You don't have the mean and standard deviation of those "new" data. You only have the mean and std during the training process.

To sum up, train-test validation tries to be as close as possible to the real problem. And since you won't know anything about your upcoming data, you should not use any knowledge you get from the test data.


In short: Yes, you can.

You need to first load the vectors using the Gensim module in Python.

# Load Google news vectors
word2vec_path = "path_to_the_vectors/GoogleNews-vectors-negative300.bin"
word2vec = gensim.models.KeyedVectors.load_word2vec_format(word2vec_path, binary=True)

# contains the list of all unique words in pre-trained word2vec vectors
w2v_vocabulary = word2vec.vocab


How to best accommodate snapshots of data over time into a single dataset for training

It would depend on the approach you want to take...
Based on the information you gave, I could imagine turning the data into a classification problem, whereby you cluster in the feature space of the various customer profile features. You could train the classifier on a "did buy"/"didn't buy" column.
The descriptive statistics you mentioned could also be valuable. I would recommend doing a mixed method approach and see which yield the most interesting results


Do neural networks have explainability like decision trees do?

I disagree with the previous answer and with your suggestion for two reasons:

1) Decision trees are based on simple logical decisions which combined together can make more complex decisions. BUT if your input has 1000 dimensions, and the features learned are highly non linear, you get a really big and heavy decision tree which you won't be able to read/understand just by looking at the nodes.

2) Neural networks are similar to that in the sens that the function they learn is understandable only if they are very small. When getting big, you need other tricks to understand them. As @SmallChess suggested, you can read this article called Visualizing and Understanding Convolutional Networks which explains for the particular case of convolutional neural networks, how you can read the weights to understand stuff like "it detected a car in this picture, mainly because of the wheels, not the rest of the components".

These visualizations helped a lot of researchers to actually understand weaknesses in their neural architectures and helped to improve the training algorithms.


Can I deal with a missing not at random column by creating a new column? (Feature engineering)

Creating a new variable as you described would be redundant as it is a function of the other two variables. In other words it is not adding any information.
The below suggestion assumes the model cannot deal with missing values, but a lot of the best models (ex. xgboost is typically one of the best for classification) will deal with this in a smart way for you.
Just fill the missing values via imputation, ex. by using the mean score or training a model using all the other features to predict the scores, and then predicting the missing ones. The model will be able to take into account that some users didn't complete the action, so it won't just treat them as if they got the imputed score, since their value in column A is 0 and that is useful information.


Is there any difference between a weak learner and a weak classifier?

A weak learner can be either a classification or a regression algorithm:

  Boosting (Schapire and Freund 2012) is a greedy algorithm for fitting adaptive basis-function models of the form in Equation 16.3, where the $\phi_m$ are generated by an algorithm called a weak learner or a base learner. The algorithm works by applying the weak learner sequentially to weighted versions of the data, where more weight is given to examples that were misclassified by earlier rounds.
  This weak learner can be any classification or regression algorithm, but it is common to use a CART model.

Source: "Machine Learning - A Probabilistic Perspective"; Murphy; 2012

So a weak classifier is simply a weak learner which is a classifier.


Improve CoreNLP POS tagger and NER tagger?

Your best best is to train your own models on the kind of data you're going to be working with.


Maybe you are not using the attention.py you think you are using. The error states that the problematic line in Attention build is:

shape=tf.TensorShape((input_shape[0][3], input_shape[0][3])),

but the code of attention.py you shared with us does not seem to contain such line.
It is possible you also just forgot to save your changes to the new attention.py file.


Plot 2D function: do I need to pass all the values to the function?

How about defining a plot function like:

function meshfcn(fcn, x, y)
    [X, Y] = meshgrid(x, y);
    Z = arrayfun(@(xx, yy)(fcn([xx, yy])), X, Y);
    mesh(X, Y, Z);

and then use it:

x = -2:0.1:2; % the range of 'x' values to plot. change to your needs
y = -2:0.1:2; % the range of 'y' values to plot. change to your needs
fcn = @powellsumfcn; % the function needs to be on the path
meshfcn(fcn, x, y)


Adaboost doesn't care about the order of the features, so no matter how many dimensions your samples have, you can just flatten them. Flattening them gives you a 2D X with shape (n_samples, n_features) which you can feed to the Adaboost classifier.


Why am I getting the same value for the accuracy and recall when using spark's mllib's MulticlassClassificationEvaluator?

It's correct. You can see comments to this effect in the docs/source. In a multiclass setting, overall precision and recall are the same.

Precision and recall in the "binary" classification case is really reporting precision with respect to one of the two classes: the positive one. There are in fact two classes, but we tend to think of it as class vs not-class.

However in multiclass, there is no positive and negative. All N classes are equal, and precision and recall are computed over all N classes, not N-1. This ends up making their definition the same.

To see it, consider the confusion matrix of a multiclass classifier, with predicted class down the rows and actual classes across the columns. "True positives" are equivalent to all the diagonal entries: these are the cases where predicated and actual classes match. "False positives" occur when the predicted class is wrong: these are all the entries in each row that aren't on the diagonal. "False negatives" occur when the actual class isn't the predicted one: these are all the entries in each column that aren't on the diagonal.

But the sum of rows (less the diagonal) is the same as the sum of columns (less the diagonal). So the false positive and false negative count are the same. And from the definition of precision and recall, this makes their values the same.

In fact, both equal the accuracy of the multiclass classifier.


How to use binary text classifier(built using SVM with TF-IDF) to classify new text document?

What is your model built in?

Most popular libraries have a score function separate from the training part. You should be able to just pass the new document to the score function of the trained model and get back the predicted class.


Loss Function for Probability Regression

Given that you are trying to predict a scalar probability value, the cross-entropy formula you listed in the question is only valid if the target variable is discrete. So if your question is "predicts the odds of drawing a matching card from a deck", it will be fine.

The main difference between cross entropy and MSE will be how they penalize wrong predictions. Let's say given a target of 1 but prediction with 0. The cross-entropy is actually undefined in this case, but as the prediction gets closer to 0, the cross-entropy loss gets exponentially larger. On the other hand, your MSE is only 1. Which one is better, it depends on your application, if you want to avoid large margin of errors, it would seem the cross-entropy is more appropriate and vice versa.

The second and third approach only differs in how they make sure the prediction is within [0, 1], one uses a sigmoid function and another uses a clamp.
Given you are using a neural network, you should avoid using the clamp function. The clamp function is the same as the identity function within the clamped range, but completely flat outside of it. So the gradient of that function is 1 within the clamp range but 0 outside of it. Because of this, you are more likely to run into the dead neural problem in the same way when people talk about "dead relu".


Multiple Output Layers in Neural Networks in Deep Q Learning

You decide what the output layer represents, so it should be OK (and probably easier to implement) to have a single fully connected layer of size $m_x + m_y + m_z$ and to interpret the output in your own code. E.g. $n_x = N_{(x)}, n_y = N_{(m_x + y)}, n_z = N_{(m_x + m_y + z)}$, where $N_{(i)}$ is the output of the $i^{th}$ neuron. In the case of regression, this should have very similar behaviour to 3 separate fully connected layers for each dimension.

However, once you have either structure, it is not clear that you will be estimating reward directly any more. There is definitely no guarantee that the rewards will be the same between $n_x, n_y, n_z$, in fact they will not be in most cases. You will need to define a function $\hat{r}(x, y, z)$ that combined the outputs of the network and have your loss and error gradient calculated from that. The function could just be $\hat{r}(x, y, z) = n_x + n_y + n_z$ or maybe $\hat{r}(x, y, z) = n_x \times n_y \times n_z$

Clearly this approximation has limits built in - it will likely over-estimate and under-estimate rewards due to its structure. If the simplified structure matches your problem, then this will be fine - it may even speed convergence. But if not, it will limit performance.

An alternative, if you don't want to build a network that has a large number of outputs, is to have a single reward estimate as the output, and have the action choice as an input. That has the advantage of leaving the details of the approximation to the neural network. It has the disadvantage that you would need to run the network up to $m_x * m_y * m_z$ times in order to decide the policy in each state.

Because of the large number of possible actions in your case, you may want to look at a policy-based learning method, rather than DQN. There are deep NN-based versions of those algorithms. A recently-published algorithm called Asynchronous Advantage Actor-Critic (A3C) has performed well in computer gaming tasks and might be appropriate for your problem.


how can I visualize the bag-of-words in a scatterplot

I think you should have a look at t-SNE which is a visualization technique based on dimensionality reduction. Wikipedia excerpt:

t-distributed stochastic neighbor embedding (t-SNE) is a machine learning algorithm for dimensionality reduction developed by Geoffrey Hinton and Laurens van der Maaten. It is a nonlinear dimensionality reduction technique that is particularly well-suited for embedding high-dimensional data into a space of two or three dimensions, which can then be visualized in a scatter plot. Specifically, it models each high-dimensional object by a two- or three-dimensional point in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points.

This paper describes a python implementation of the technique on various word2vec datasets.


It seems there does exist a python library for supervised ML to solve this
https://pypi.org/project/dedupe/


In theory, any neural network can be used for feature extraction since neural nets perform it by themselves.

Whether or not using CNNs vs RNNs is a different question. CNNs are typically more computationally efficient whilst RNNs are more flexible but typically slightly harder to train.


Train an LSTM neural network with time series containing seasonal and trend

I don’t know if the seasonality will be achievable in your predictions, due simply to the timeframes you we using. If you see a daily up—down movement, but only provide 20 minutes for a prediction, how will the Model know whether or not it is at a turning point? You would perhaps need to include other features that contain that information - perhaps even the time stamp would suffice.

It is odd, that the model always predicts a downward movement — I would have expected it to simply continue on the current path (up or down), assuming you have both directoins in your training data...?

Perhaps you could look into some ideas used in common timeseries analysis methods, like separating the seasonality, trend and noise and feeding them separately to the model. Search for terms SARIMA, ARIMA, seasonality and cycles. (S)ARIMA stands for “seasonal autoregressive integrated moving average”, and represents a common way to look at data over time using previous values (autoregressivej, the differences of current value to previous values (integrated) and a moving average of past time steps (moving average).

The terms generated would of course catch the phases where the value changes direction and so be able to model seasonality fairly well.


Which Loss cross-entropy do I've to use?

which kind of this classification is multi-label or multi-class?

If we take a look at the data panel at the competition's page in kaggle:

  Category - category of the crime incident (only in train.csv). This is the target variable you are going to predict.

The target we are trying to predict is the category of the crime (which can take multiple values) so this answers your question: the problem is multi-class.

  So which crossentropy is accurate or better use for this scenario?

There are three main types of cross-entopy in keras:

categorical cross-entropy is used for multi-class classification, where the target variable is one-hot encoded.
sparse categorical cross-entropy is used for multi-class classification, where the target variable is not one-hot encoded.
binary cross-entropy is used for binary classification problems.

From your question I assume that you think the different types of crossentropy are interchangeable. That is not the case. You should select the cross-entropy based on how your problem and how you encoded your data.

In your case, since you have a multi-class problem and you state that you have used pandas.get_dummies(), which ont-hot encodes your data you should use categorical cross-entropy and nothing else!


Which ML classifier is appropriate for me if all of my features are categorical?

What does performance is not up to the mark mean?

To optimize performance you first need to understand it.

A typical workflow is this:

1. Define a baseline performanc

E.g. predict values with a constant (average, mode) or pick randomly from a distribution and measure the accuracy of that

2. Compare your 1st draft model (in your case your RandomForest) to that baseline

If your 1st draft model isn't better than the baseline, especially a random baseline, you have some errors in your code, data, etc. Try to find and eliminate those first.

If it is better but not by much, you have top optimize the model (see next step). If it is better by much you are either a) done or b) not satisfied with the absolute performance in which case you have to optimize.

3. Optimize your model parameters

Now you start grid searching your model parameters to get every last bit of performance out of it to make sure the problem is with the model and not the parameters.

4. Beauty contest

If your performance is still subpar now you can try other algorithms but do not expect wonders from this step. If you already tried RandomForest maybe go for:

Boosted models like XGBoost
Naive Gauss models
...

Fit them on the same data, do parameter optimization and compare results.

5. Back to the data

Most likely just picking another model did not help if the original performance was too far away from acceptable. Then you have to go back to the data, collect more, feature engineer, etc.


Test data significantly different from training data

The main risk is underfitting, a model trained on a significantly different dataset will poorly predict the test set


Intuition behind the number of output neurons for a neural network

The reason 10 neurons work better than 4 is because it allows the network to encode all possible answers independently.

You could have 4 neurons and train the first one to encode the least significant bit, the second neuron the second-least significant bit, and so on. It would work.

Images of a 1 (bits: 0001) are quite like images of a 7 (bits 0111). Imagine an image where we know it’s a 1 or a 7, but we have no idea which one. With 4 outputs (as described) you would output (0, 0.5, 0.5, 1), meaning the correct answer is either 1, 3, 5, or 7. With 10 outputs, all classes would have 0 probability and 1 and 7 would have 50% each, which conveys more accurately what can be interpreted from the image.

You could have a single output that just outputs the value of the digit directly. But a 1 and 7 are now far apart in the output node (the whole range is from 0 to 9), so it’s impossible for this network to say “1 or 7”. It will probably say: 4 (midpoint between 1 and 7). That’s why it doesn’t work well.

Very poorly explained by “Nielsen’s book”, IMHO.


There are a few models that are trained to analyse a sentence and classify each token (or recognise dependencies between words).

Part of speech tagging (POS) models assign to each word its function (noun, verb, ...) - have a look at this link

Dependency parsing (DP) models will recognize which words go together (in this case Angela and Merkel for instance) - check this out

Named entity recognition (NER) models will for instance say that "Angela Merkel" is a person, "Germany" is a country ... - another link


How to interpret Hashingvectorizer representation?

Basic Background

Imagine the process of count vectorizer: you first create a vocabulary which maps each word (or n-gram) to an integer index (index in the document term matrix). Then, for each document, you count number of times a word appears and set that value at appropriate index to build vector representation for the document.
This can potentially create a very large number of features since each n-gram/token is one feature.
Even if you want to limit the total number of features by using some trick like top-N words by occurrence, you still need to calculate and hold in memory the map of all word-counts. This can be potentially prohibitive in some applications.
Similar problem happens for TfIDf, where you additionally store the mapping of word to document occurance for calculating the IDf part.
Either way, you are doing multiple passes over the data and/or potentially large amount of memory consumption.
The problem is also with bounds or predictability: you do not know the potential memory usage upfront in first phase.
Hashing vectorizer can build document representation for all documents in one single pass over the data and still keep memory bounded (not necessarily small, the size depends on size of hash-table).
In a single pass, you calculate hash of a token. Based on the hash value, you increment the count of particular index in the hash-table (the array underlying the hash table implementation). You get representation of current document without looking at every other document in the corpus.
This gives rise to problem with representation accuracy. Two different tokens may have hash collision.

So you are in effect trading [representation accuracy and explanatory power] Vs. [space (bounded predictable memory usage) and time (no multiple passes on the data)].

Answers to your specific questions

There is a (sort) correlation between input words and features: through the hash function. But this correlation is potentially defective (hash-collisions) and there's no inverse transformation (you can't say what word is represented by feature number 207).
There's no fit and transform. For a fixed hash-function, no dataset specific learning is happening (ala word2vec).
There's no semantic interpretation of the distance. Two words semantically similar words may not be close to each other in the representation. As long as two almost (syntactically, based on tokens) similar documents are close enough, it will work on text classification.

Why Would It Work?

Given these information, you are right in being skeptical: why on earth this should work? The answer is empirical: a randomized representation like hashing works reasonably well in practice (the benefits from exact count based representation are not that great). There might be some theoretical explanation too but I don't know it enough. If curious, you can probably read up this paper.


Algorithms and techniques for spell checking

Here is what I built...

Step 1: Store all the words in a Trie data structure. Wiki about
trie.
Step 2: Train an RNN or RNTN to get seq2seq mapping for words and
store the model
Step 3: Retrieve top n words with  levenshtein
distance with the.  Wiki about LD with the word that you are trying to
correct.

Naive Approach: Calculating the edit distance between the query term and every dictionary term. Very expensive.
Peter Norvig's Approach: Deriving all possible terms with an edit distance<=2 from the query term, and looking them up in the dictionary. Better, but still expensive (114,324 terms for word length=9 and edit distance=2) Check this.
Faroo's Approach: Deriving deletes only with an edit distance<=2 both from the query term and each dictionary term. Three orders of magnitudes faster. http://blog.faroo.com/2012/06/07...

Step 4: Based on the previous 2 or 3 or 4 words, predict the words
that are retrieved from Step 3 above. Select the number of words
depending on how much accuracy you want (Of course we want 100%
accuracy), and the processing power of the machine running the code,
you can select the number of words you wanna consider.

Check out this approach too. It's an acl paper from 2009. They call it - language independent auto correction.


How to use boolean data in DecisionTreeClassifier in sklearn?

Why would this not make sense? It tries to create a split with maximal separation between signal and background. Any float >0 and < 1 will result in the same split.


What's the best strategy to train a CNN with images that only have labels for positive characteristics?

I think your best bet would be transfer learning. Start with a model that has already been trained with a wider dataset such as the ones presented here. From there you can train the model with your specific dataset. You can then use output nodes for the labels which you have available to you, and you can get the predictions for the other images from the pre-trained model which are usually trained for thousands of different classes.

Alternatively, you can train a model with all the output classes you have in your label set and another output for "other". Then when an output node is selected you can pass that same input to the pre-trained model which was trained with your data as well.


Is it advisable to merge similar datasets to improve model accuracy?

Ideally if you mix data from different sources (different distributions), the mix should be uniform for each class.   Otherwise if for example class A gets more of source S and class B gets more of source T, there is a risk that the model will learn to associate class A with source S and B with source T and that it won't generalize across sources.
To validate such an approach, keep a separate test set with both sources and compute several accuracies, one for each (class, source).


grid search - optimal weighting of classifiers

GridSearch finds those optimals weights for you.

You can access these weights through the attribute best_params_ of the GridSearch object, which will return all the optimal parameters (including the weights):

optimal_weights = grid_Search.best_params_


The actual results and results from pickle files are not matching in pandas for DBSCAN clustering

It appears your pickle file isn't being loaded as a pandas dataframe. Why not just use df_pickle = pd.read_pickle('/MC.pkl') – the rest should fall into place after.


Intuitive explanation for representing gradient in higher dimensions

Yes, you are thinking of multiple dimensions in the wrong way, but I'm not sure anybody knows the right way to think about them.
You don't represent complex networks with many parameters in 3D space. We start in low dimensional 2D and 3D space to develop an intuition about what happens with the gradient, how to move along it, and so on. Then, we go into thousands dimensional spaces, were we don't know what happens, and we apply our intuition - those spaces work the same way, just that we cannot visualize or comprehend them.
Your cost function will have thousands of dimensions, and your gradient will be the same. But we rely on our intuition from low dimensional spaces to manipulate them.


How to clusterize points lying on the sphere

You have circular variables.
One way to deal with them is to create two variables for each:
sin(alpha_1), cos(alpha_1), sin(alpha_2), cos(alpha_2), ...
Then you can use regular clustering algorithms.

There exists custom algorithms for circular data.
Examples: Unsupervised clustering of multivariate circular data


Tensorflow.js error: must be either an HTMLVideoElement but was HTMLVideoElement

I finally managed to get it to work by opening a issue in the tfjs repo
I copy here the solution provided by the github user tafsiri

After digging around a bit, it looks like your project is running in
separate context mode and there are two JS contexts in which code is
run. Your code is being run in the browser context and the tfjs
library code is being run in the node context. In the node context
video instanceof HTMLVideoElement returns false. The nw.js docs
suggest that browser api's aren't usable in the node context so there
may be incompatibility there.
Their docs suggest you can switch to mixed context mode, I tried that
by adding the following entry to the package.json and then it worked.
"chromium-args": "--mixed-context", If you want to stay in separate
context mode (and I don't know enough about nw.js to know the
advantages and disadvantages of each) you would need to not use any of
the browser specific apis of tfjs.


How to discretize a numerival value with predefined ranges in Weka?

Discretising or binning, very common. There is a filter for that in Weka. You'll find it under weka.filters.unsupervised.attribute.Discretise. It is in the GUI too.

You can find the documentation under: https://weka.sourceforge.io/doc.dev/weka/filters/unsupervised/attribute/Discretize.html


They are almost identical:

  The second key reason that maxout performs well is
  that it improves the bagging style training phase of
  dropout. Note that the arguments in section 7 motivating
  the use of maxout also apply equally to rectified
  linear units (Salinas & Abbott, 1996; Hahnloser, 1998;
  Glorot et al., 2011). The only difference between maxout
  and max pooling over a set of rectified linear units
  is that maxout does not include a 0 in the max.

Source: Maxout Networks.


Regression: What defines Linear and non-linear models or functions

I think it is helpful to distinguish between linear functions (representing the relationship between independent and dependent variables) and linear models (representing the relationship between the model parameters and the outcome).

A linear model can be represented by a non-linear function. A linear regression model is any model that is represented by a linear function in the parameters. A polynomial can be represented as such.

Likewise a nonlinear regression model is any model that is represented by a nonlinear combination of parameters.

To quote from Wikipedia:

  Although polynomial regression fits a nonlinear model to the data, as
  a statistical estimation problem it is linear, in the sense that the
  regression function E(y | x) is linear in the unknown parameters that
  are estimated from the data.

Let the model be represented by the function $f(x,w)$ with inputs $x$ and parameter vector $w$. Polynomial regression has the form $f(x,w) = w_0 + w_1*x + w_2*x^2 \ldots$ It is linear in the parameters because $f(x,a+b) = (a_0+b_0) + (a_1+b_1)*x + (a_2+b_2)*x^2 \ldots = a_0 + a_1*x + a_2*x^2 \ldots + b_0 + b_1*x + b_2*x^2 \ldots = f(x,a) + f(x,b)$. But it is not linear in the inputs because $f(x+y,w) \neq f(x,w)+f(y,w)$.

Often people think about the relationship $y=f(x)$ between $x$ and $y$, but linear in linear regression refers to linearity in $w$.


Hierarchical Clustering: Extract observations from large heatmap

A similar question was asked at stackoverflow. There it was proposed to use the option criterion='maxclust' for the flcuster function:

from scipy.cluster.hierarchy import fcluster
clust = fcluster(Z, t=k, criterion='maxclust')

The description of this option in the flcuster documentation is a bit confusing, but that's how you get a clustering with t=k clusters.

You should be able to retrieve the answer to your second question with the resulting array.


import pickle

knn = NearestNeighbors(10)
knn.fit(my_data)

# Its important to use binary mode
knnPickle = open('knnpickle_file', 'wb')

# source, destination
pickle.dump(knn, knnPickle)

# load the model from disk
loaded_model = pickle.load(open('knnpickle_file', 'rb'))
result = loaded_model.predict(X_test)

refer: https://www.geeksforgeeks.org/saving-a-machine-learning-model/


Predictive modeling when output affects future input

I am afraid that such situations are fundamentally inherent in predicting/forecasting contexts; quoting from the very recent paper by Taleb et al., On single point forecasts for fat-tailed variables (open access, para 3.7):

3.7. Forecasts can result in adjustments that make forecasts less accurate
It is obvious that if forecasts lead to adjustments, and responses
that affect the studied phenomenon, then one can no longer judge these
forecasts on their subsequent accuracy.

So, other than  communicating this clearly beforehand and reaching an agreement on how the predictions will be assessed in the presence of such adjustments, there is not much else you can do from a modelling or methodology perspective. The advice suggested further in the same paragraph quoted above:

In that sense a forecast can be a warning of the style “if you do not act, these are the costs”.

can form the basis of such a communication and agreement.


Sequence prediction with unlimited predictions

Your biggest issue with the evaluation scheme you have - "success" means within tolerance, "failure" means outside tolerance, plus your constraint on model outputs needing to vary per time step - is that it will be hard to extract gradients in order to train the prediction model directly. This rules out many simple and direct regression models, at least if you want to use "maximise number of scores within tolerance" as your objective function. The constraints on sequential predictions and allowing re-tries are also non-differentiable if taken as-is.

I think you have two top level choices:

1. Soften the loss function, and add the hard function as a metric

Use a differentiable loss function that has best score when predictions are accurate and constraints are met. For example your loss function for a single predicted value could be

$$L(\hat{x}_n, \hat{x}_{n+1}, x_{n+1}) = (\hat{x}_{n+1} - x_{n+1})^2 +  \frac{a}{1+e^{s(|\hat{x}_n - \hat{x}_{n+1}| - \epsilon)}}$$

the second constraint part is essentially sigmoid with $a$ controlling the relative weight of meeting constraints with accuracy of the prediction and $s$ controlling the steepness of cutoff around the constraint.

a. The weighting between prediction loss and constraint loss will be a hyper-parameter of the model. So you would need to include $a$ and $s$ amongst parameters to search if you used my suggested loss function.

b. You can use your scoring system, not as an objective function, but as a metric to select the best model on a hyper-parameter search.

c. With this approach you can use many standard sequence learning models, such as LSTM (if you have enough data). Or you could just use a single step prediction model that you feed current prediction plus any other features of the sequence that is allowed to know, and generate sequences from it by calling it repeatedly.

This system should encourage re-tries that get closer to the true value.

2. Use your scoring system directly as a learning goal

This will require some alternative optimising framework to gradient descent around the prediction model (although some frameworks can generate gradients internally). Genetic algorithms or other optimisers could be used to manage parameters of your model, and can attempt to change model parameters to improve results.

For this second case, assuming you have some good reason to want to avoid constructing a differentiable loss function at all, then this problem can be framed as Reinforcement Learning (RL):

State: Current sequence item prediction (or a null entry), as well as any known information such as tolerance, length of sequence, current sequence item value (which may be different from current prediction) $\epsilon$, $d$, $\mu$ or $\sigma$ can be part of the current state.
The action is to select next sequence value prediction, or probably more usefully, the offset for the next sequence item value. Using offsets allows you easily add constraint for minimum $\epsilon$
The reward is +1 for being within tolerance or 0 otherwise.
Time steps match the time steps within a current sequence.

You can use this to build a RL environment and train an agent that will include your prediction/generator model inside it. There are a lot of options within RL for how to manage that. But what RL gives you here is a way to define your goal formally using non-differentiable rewards, whilst internally the model can still be trained using gradient based methods.

The main reason to not use RL here is if the prediction model must be assessed at the end of generating the sequence. In which case the "action" might as well be the whole sequence, and becomes much harder to optimise. It is not 100% clear to me from the question whether this is the case.

Caveat: RL is a large and complex field of study. If you don't already know at least some RL, you can expect to spend several weeks getting to grips with it before starting to make progress on your original problem.

There are alternatives to RL that could equally apply, such as NEAT - deciding which could be best involves knowing far more about the project (e.g. the complexity of the sequences you wish to predict) and practical aspects such as how much time you have available to devote to learning, testing and implementing new techniques.

Have you forgotten something?

If you allow infinite re-tries, then an obvious strategy is to generate a very large sequence moving up and down using different step sizes (all greater than $\epsilon$). This doesn't require any learning model, just a bit of smart coding to cover all integers eventually. Chances are this model is only a few lines of code in most languages.

If this is to be ruled out, then some other rule or constraint is required:

Perhaps only positive increments are allowed in the predicted sequence (so we cannot re-try by subtracting and trying again)? This conflicts with your "unlimited predictions" statement.
Perhaps a sub-goal here is to make the guessing efficient? In which case RL could be useful, as you can add a a discount factor to reward processing in order make the model prefer to get predictions correct sooner.


Clustering with k-means for text classification based on similarity

So, the problem is how to cluster texts.
Firstly, an alternative approach to representing document meaning, you can use Doc2Vec and compare similarities between document embeddings (https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e).
Secondly, if you are unsure about the ideal number of clusters, instead of using k-means, you can use agglomerative clustering, which is essentially bottom-up method which clusters individual document embeddings by a distance metric to eventually merge all of the clusters to a "mega-cluster", containing all documents.
You can then see how the documents cluster together using a dendrogram.
Here is a good resource on agglomerative clustering : https://stackabuse.com/hierarchical-clustering-with-python-and-scikit-learn/


Advice and Ideas appreciated - Machine Learning one man project

train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples.
--> can't that be solved by using stratify=True?
--> yes, that's what stratify=True is for. However you still only train on the data of your training set and test with the data from the test set
train_test_split doesn't split the sets the same way all the time so the results aren't comparable
--> setting random_state=0 solves the problem?
--> it does...keep in mind that random_state= will work fine as long as you keep the seed aka random-state the same

for 3. and 4. let me make sure you understand the difference between CV and TTS. TTS splits your data once, trains on the now "fixed" training set and tests on the "fixed" test set. However this introduces a kind of bias to your evaluation because you are not training or testing on all observations. By going for a CV you make sure that all observations are used for testing and training. This reduces the bias of setting a "fixed" training and test set.

Now for 3.:
When in doubt use CV. HOWEVER...sometimes this doesn't make sense. Think about time series data and how cv would work then and what the result would be. That being said...now think about how TTS with stratify or random would work and what those would mean. CV usually is a good way to go however you always have to think about if the evaluation method fits your problem. TTS has its merit for time series as well as for performance reasons using large datasets.

And 4.:
The picture visualizes TTS and CV. First you have CV which splits your training data so that each small data set is used as test data. The "Testing" set is then used to measure generalization of your model. This is called the holdout method

The diagram that says "Data Permitting" demonstrates TTS with an additional "Testing" set for measuring generalization. You train on "Training", validate the model on "Validation" and as with the holdout method you keep a "Testing" set for measuring how well your model generalizes on unknown data.

Last but not least...5.:
If you get good results and generalization by using TTS...go for it. If you get problems with over-/underfitting or generalization try CV and see if it helps. Of course as with everything in Data Science and Machine Learning there are multiple levels of complexity that you can add on top of TTS and CV as well. For example you could read about K-Fold or LOOCV etc. I don't think that is necessary though.


Splitting training and test set with financial data

I think what you're doing is correct, in fact it would be even more correct to introduce a gap between your test and train set, i.e.

train <-head(stock_indicators,round(0.65*nrow(stock_indicators)))
test <- tail(stock_indicators,round(0.25*nrow(stock_indicators)))

The reason for this is time series data (usually) exhibits strong serial auto-correlation, so if you put the price for one day in your training set, and the next days price your test set they're a long way from independent and your test error is biased.

The reason you're not getting a good fit is that the data is probably not stationary, i.e. what's driving stock prices is changing over time and it's hard to forecast - if it wasn't everyone would do it perfectly.

Another way of doing the split is to split the data into blocks (say weeks) and then take 80% of the blocks for the training set and 20% for the test set. This is useful if say your data is seasonal (i.e. electricity demand), in which case using the last 20% isn't really representative of the full future you want to forecast. I'd argue stock prices aren't overly seasonal.


Implementation of LDA (Latent Dirichlet Allocation) for classification tasks

You can use LDA on your training data to build the topic representation of it for example:

(document)entry[1] Label A: (topic 1 has 4 words in document 1)T[1]=4, T[3]=7, T[4]=5..
entry[2] Label C: T[1]=3,T[2]=2...
entry[3] Label A: T[1]=2,T[2]=2,T[3]=5...
.
.
.
 -

Using this you can build a simple decision tree: T[1]>1 AND T[3]>4 AND (T[2]>1 OR T[4] > 3) ---> Label A

Another approach would be to use: https://en.wikipedia.org/wiki/Dirichlet-multinomial_distribution#A_second_example:_Naive_Bayes_document_clustering


why we have run make.sh file initially in darknet and YOLO object detection?

I know what is cython and make (but I never use YOLO!)

Cython is a C-extension for python. It allows you to write code C/C++ in a python script. (use for very fast program execution)

Make is command which executes your makefile. You can consider makefile is a build script to create/tune the necessary things like environment/folders/.. etc.


What are the key differences between FPGA and GPUs for Deep Learning?

I'll address the example you linked to first.

The link in your answer redirects, it seems that an equivalent product from the same company is Myrtle. The only product on that webpage is Nervana. All of Intel's AI products (including their Xenon processors, containing no AI acceleration) are supported by their libraries. That permits running your project on a range (most, but not all) of their hardware.

Attempting to use Wayback or search for the old webpage turns up nothing useful.

  Myrtle has significantly accelerated the time-to-market for FPGA designs by automating the translation of AI network designs using compiler technology targeting the Intel® Arria® 10 FPGA. Trained, high level algorithms, specified in a wide range of network descriptor formats - Torch, Caffe etc. are compiled and mathematically optimised, directly to VHDL to produce efficient, low latency designs. In cases where proprietary non-standard, deep learning layers are used these first need to be implemented.

  Myrtle.ai sets new deep learning standard; trains ResNet network 4.5x faster, using 8x fewer GPUs and at a quarter of the cost than previous benchmark winner.

Currently it's being implemented on the Intel Arria 10 GX FPGA:

  The Intel Nervana NNP is a purpose built architecture, an application specific integrated circuit (ASIC) that is custom-designed and optimized for deep learning.

  There are two different versions:

  The Intel® Nervana™ Neural Network Processor-T 1000 put a large amount of HBM memory and local SRAM much closer to where compute actually happens. This means more of the model parameters can be stored on-die to save significant power for an increase in performance. The Intel® Nervana™ Neural Network Processor features high-speed on- and off-chip interconnects enabling multiple processors to connect card to card and chassis to chassis, acting almost as one efficient chip and scale to accommodate larger models for deeper insights.
  The Intel® Nervana™ Neural Network Processor-I 1000 is a discrete accelerator designed specifically for the growing complexity and scale of inference applications. It is built on Intel’s 10nm process technology with Ice Lake cores to support general operations as well as neural network acceleration.

This is the Nervana:

The FPGA has optimized libraries available that compile direct to VHDL, if you want your own libraries you'll be writing your own code. It is somewhat analogous to wanting a custom BIOS for a GPU, you can wait for the manufacturer, look for a third party offering, or write your own - not simple, not extremely difficult. The FPGA is highly configurable, though that one isn't particularly powerful.

The ASIC also has optimized libraries, the same ones (with a compiler switch) so there's no difficulty switching from one to another; or running your code slower on a GPU or CPU. The ASIC isn't configurable, after years of development on the FPGA they decided to implement the results in dedicated hardware; gone is the reconfiguration but there's an increase in speed and a decrease in price (and TDP).

  ... would need someone who is able to configure the FPGA according to the type of project they want through the use of languages such as Verilog or VDHL.

You could, much as you could write your own BIOS for your GPU, like some cryptominers do. It's not something you have to do, but there are easy tools and brain surgery tools. The manufacturer offers downloadable updates from time to time.

  Furthermore, if they want to change the type of DL project they want to do, they have to reconfigure the FPGA to follow suit.

Maybe, probably not. Just change the program and recompile is probably good enough, unless you want to squeeze another few percent more performance out of it; and you know more than the manufacturer about the device and optimized AI architecture.

Now to your main question, which seems to be the first thing that you asked:

  I'm trying to investigate the ways in which FPGAs differ to GPUs for the purpose of deep learning. I understand this is a complex question and ...

First, a similarity: They are a means to solve AI problems, they have a certain amount of potential (both capability and processing speed); better ones cost more but because GPUs are mass produced (comparatively, FPGAs are mass produced also) they will be cheaper and there will be more free tech support (Forums).

A cheap FPGA (or ASIC) won't be fast, and a fast one won't be cheap - if you think GPUs are expensive then that will be your cheaper route, unless you have a new algorithm, a breakthrough, then make your own ASIC and sell your own card for the greatest profit. Assuming that you're the average person an off the shelf solution is likely better priced.

The "biggest challenge" is Bayesian optimization, recurrent network and reinforcement learning or genetic algorithms for parameter tweaking; something that GPUs don't do well.

Intel has its Loihi neuromorphic research chip to represent the next direction in AI.

While training is done the conventional way the inference is ran on that tiny device, much faster than previous generation devices.

Neuromorphic computing is one of the newest directions for AI, an FPGA or GPU is a stone-age tool. Here's what BrainChip says:

  BrainChip is the leading provider of neuromorphic computing solutions, a type of artificial intelligence that is inspired by the biology of the human neuron - spiking neural networks. Our spiking neural network technology can learn autonomously, evolve and associate information just like the human brain. We have recently unveiled the Akida Neuromorphic System-on-Chip.

  The Akida™ NSoC is both low cost and low power, making it ideal for edge applications such as advanced driver assistance systems (ADAS), autonomous vehicles, drones, vision-guided robotics, surveillance and machine vision systems. Its scalability allows users to connect many Akida devices together to perform complex neural network training and inference for cybersecurity and financial technology (FinTech) applications.

Notice that it also comes full circle back to FPGA. If you could write VHDL and have considerable experience you could have developed that but the FPGA probably isn't an Arria 10. A GPU is the least expensive and quite a flexible solution for the average person, it's also not a one trick pony. Something a few years old, like the Intel Deep Learning Inference Accelerator shown below, costs U$10K.


There are methods like the Kullback-Leibler divergence model, the Wald-Wolfowitz test for detecting non-randomness and covariance shift.

A simple test for quick analysis of covariance test would be to build a machine learning model, where the model is repeatedly tested with inputting training data and the production data.

In case, the model can make out the difference between the training and production datasets then it can be a sign of covariance shift.


Clustering with categorical as well as numerical features

thank you for your question, which asks about how to represent categorical variables in clustering.
The main way that we represent any categorical variable is to represent them as one-hot encodings (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/).
After converting our categorical variables to one-hot encodings, we simply concatenate these vectors with our numerical values (which should be normalised!) to make a long / high-dimensional vector for a particular example in your dataset.


Relative merits of different open source natural language generators

In terms of open source NLG components, I'm most familiar with Mumble and FUF/SURGE. They've got both similarities and differences, so it's hard to say which is better...

Mumble:

written in Lisp
EPL license
based on tree-adjoining grammar
focuses on linguistic message planning

FUF/SURGE:

written in Lisp
GPL license
based on functional unification grammar
focuses on syntactic realization

Since it sounds like you're interested in abstractive summarization (which is much harder than traditional extractive summarization), I'd recommend the following academic papers:

Text Generation for Abstractive Summarization
Framework for Abstractive Summarization using Text-to-Text Generation
Towards a Framework for Abstractive Summarization of Multimodal Documents -- full disclosure: I'm the author of this one

Also, consider checking out this textbook to get started: Building Natural Language Generation Systems


Predicting outcome of games - how to make team 1/2 interchangable

Home field advantage is incredibly important in sports. If you always put the home team in the first column, your model will adjust the bias term accordingly. I've been doing this successfully for NBA and MLB models for several years now. Making the first team the home team also eliminates the need for an explicit home indicator variable and you'll be able to use "half" of the data like you've described.

For rare neutral games like the Superbowl, you would need to create a mirror image of the data. In other words, present two rows for each game (team-A,team-B;team-B,team-A). When it's time to score you model, you can just arbitrarily pick one of the two records because they represent mirror opposite of the same thing. I've done this before for March Madness basketball games (all neutral courts), and this year (2016) my March Madness model was in the top 10% in Kaggle, so the technique works.

Home field advantage is just one of the variables you'll need to make a good predictive model, but that's an answer for another question.


How to measure the similarity between two text documents?

In general,there are two ways for finding document-document similarity

TF-IDF approach

Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . NLTK library provides all .
Convert the documents into tf-idf vectors .
Find the cosine-similarity between them or any new document for similarity measure.

You can use libraries like NLTK , Scikit learn ,Gensim for Tf-Idf implementation . Gensim provides many additional functionality .
See :  https://www2.cs.duke.edu/courses/spring14/compsci290/assignments/lab02.html

Word Embedding

Google's Doc2Vec ,which is available in Gensim library can be used for document similarity .Additonaly,teh Doc2Vec model itself can compute the similarity score ( no cosine or anything needed her ) . You just need the vectorise the docs by tokenizing ( use NLTK ) and make a Doc2vec model using gensim and fins similarity and many using Gensim inbuilt methods like model.n_similarity for similarity between two documents .

Other

Additionally,since your aim is to cluster documents,you can try Topic Modelling using LDA ( Latent Dirichet Allocation) or LSI ( Latent Semantic Indexing )  .


Should I include all dummy variables or N-1 dummy variables (keep one as reference) in neural networks

I will answer the question in the Neural Network context (i.e. I won't talk about regularization in regression algorithms handles this problem).
The problem of not dropping one of the encoded variables will cause multicollinearity. That is, one of the variables can be estimated using others. e.g. If you have a variable that says whether a person is female or not, why would you need another variable which says whether a person is male or not. However, the main thing that people forgot to say while answering this question is that multicollinearity is not actually a big problem unless you need the interpretation of your variables. Multicollinearity will cause your coefficients to be false, but it won't affect your predictions. Thus, it should not be a big problem unless you need to explain your model in terms of your variables. Another possibility is that, if the covariance between your variables in the training set and test set would be different, then your predictions would be affected, and you would have incorrect results. However, if you shuffle your dataset good enough (which is your assumption) and split the train and test set (also validation) correctly then covariance should be the same in both. In other words, the relation between those correlated variables can be assumed to stay the same in the train and test set. Thus, you can safely assume that your predictions are correct.


How to understand the weights and biases for beginners?

Y = Wx + b

Here, I am assuming that the h is in subscript :

Y is the value that we want to predict
x is the input
b is the bias
W is the weight

Now, the question that you asked If I understood clearly :

  You want to know, How can you get the weight and Bias of your model ? And are the W input to your machine learning model or is it the neural networks task to find the weight for you ?

Weight W is the coefficient of the input x which when combined with bias b returns the predicted value Y. Note that weight W is the coefficient of the feature input x.

The sole aim to run a machine / deep learning algorithm is to find the best set of weights corresponding to each feature and the bias. The bias does not correspond to every weight w, It corresponds to each layer in the network.


How to decide the processing power required based on the dataset?

Dataset (number of samples, number of features) is one variable. Algo/model complexity is another.

For example, linear regression will be much faster as compared to 4 layer neural network (and will require much lesser compute power).

So, before deciding need for compute powers, you can :

Try few models with hardware (or AWS instances) you already have
Estimate need for better hardware (CPU / GPU) based on the performance and what is the bottleneck

For very large data sets (say 10 TB+), I/O can become the bottleneck.


Obtain a model where each feature vector is past few samples and labels are future few samples?

This looks like almost an ideal use case for a Recurrent Neural Network (RNN) model. I say almost, as your output variable is 2 separate class vectors, that part is slightly fiddly.

The main advantage of using a RNN is that the model would be built on the assumption that s1_var1 is of the same type as s2_var1 etc. Whilst with other models you could flatten the time series into a 10 x 20 = 200 long vector, and the output likewise into a 50 long vector, and treat the whole problem as a single step supervised learning, such a model would not include the same assumption.

For your problem, the RNN would have 10 inputs (corresponding to the 5 classes of each of 2 variables) and 10 outputs. You would train it by presenting your input sequence one sample at a time in the correct order 1 to 20, then continue running the network 5 more steps, comparing the outputs against expected. When predicting, you do similar, but then just read off the outputs as predictions, one at a time.

There are a few different Python libraries that support RNN models, and there are different internal choices (such as using LSTM layer versus GRU) which you might want to explore. I am currently learning the Keras framework, which supports some options for RNNs. Probably the example code which learns to predict an exponentially-decaying sine wave would be a good place to start trying to understand the Keras model - although that's a regression, it should give you some insight into how RNNs can be built for this kind of series-based data.

An alternative to RNN here might be a hidden markov model - if you have some insight into the internal state of the system, that could be a better option.


well there are two things.

You write your learning algorithm from scratch. Then you define your cost function yourself. No one usually does that unless they are asked for.
Using Machine Learning libraries in Python (Most common is SKlearn unless you want to use Neural Networks in e.g. Keras). They have their own cost functions designed. You just need to define your problem properly then it does the thing for you.

In (2) just define your problem properly. The question is fuzzy (what does "not generating traffic good traffic" mean?) but you input your data to your algorithm and it optimize the cost function according to the chosen algorithm. If you need more input, please update the question and give more details so I can help.


The National Institute of Standards and Technology (NIST) has ideal gas datasets here


question about splitting data set by attributes and labels

You could indeed use an X to have it all. However your downstream models expect the features and labels (I.e. Xs and ys) to be referenced via different object pointers in order to train and evaluate the model.
E.g
model.fit(X,y)


Improving Accuracy of the Deep Learning Model

You can try bunch of things like:

Data augmentation.
Transfer learning on model trained on similar dataset.
Train multiple neural network with different intialisation of weights and then take majority voting during inference.

The last point matters if you're training from scratch because weights intialisation direct the model towards different minima. So with average of different networks you're essentially taking average of different minimas which might increase overall accuracy.


What is the best way to read SQL dataset in to Tensorflow?

You use the methods on SqlDataset to manipulate the data. For example, create a train/test split with:

test_dataset = dataset.take(1000)
train_dataset = dataset.skip(1000)

I would get rid of unneeded columns in the SELECT statement to reduce the size of the data as early as possible.


How do you distinguish between conversational text and possible news article?

News sentences will have more unique tokens than normal conversations.
Conversations have more stop words than news articles.

I think you can use bert or normal wordvect classification to train a baseline model here.
I would play aroud the pipeline  of fake news classifier and news-conversation classifier. like passing the text to news classifier first and then passing it to news-conversation classifier. Try to mix and match to get the best results. Set some thresholds.


Cross-Validation: Repeated K-Fold/Group K-Fold

Group k-fold is sufficiently specialized that the comparisons you ask for don't really make sense.  "Repeated" really does just mean to remake the splits multiple times; you could easily make a "repeated group k-fold" splitter.

You use group k-fold when you have groups you don't want split across the training and test sets.  For example, if your data includes multiple rows for each customer (but it still makes sense to train on individual transactions/rows), and your production use-case involves making predictions for new customers, then testing on rows from customers that also have rows in your training set may be optimistically biased.

You use repeated XYZ when the data and models have high variability and you want more performance datapoints with which to make statistically sound statements.  (Likely this is when your dataset is small, in which case happily your model building should be faster so that you can afford the computational cost of repeating the cross-validation procedure multiple times.)

https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html
esp. the image:

(but now I'd like to see one with groups not lining up with classes like that...)


Alignment of square nonorientable images/data

Fun with Group Theory!

There are only 8 unique rotation-inversion operations for a square matrix.

The four rotation operators are (0,90,180,270).  Further rotation or rotation in the reverse direction is the same as these four.  Two successive rotations just yields one of the rotation operators, so we will only consider these four rotations applied one time.

The five inversion operators are (0,/,\,|,-).  Two successive inversions just yields a rotation, so we will only allow for a single inversion.

We can thus derive all operators by combining these two vectors, which yields 4*5=20 possible states.

(0,90,180,270,0/,90/,180/,270/,0\,90\,180\,270\,0|,90|,180|,270|,0-,90-,180-,270-)

But there is still symmetry to be exploited in the inversion operators.  You can probably intuit that the 4x4 matrix only has 8 final states: inverted or not and rotated by (0,90,180,270).  It turns out that you can arrive at any possible state involving an inversion using any of the other inversion operators followed by one of the rotations.  So we only need to retain a single inversion operator!  So the final set of 8 orthogonal operations are:

(0,90,180,270,0|,90|,180|,270|)

If there is any symmetry in the matrix's members then some of the resulting states may be degenerate.

In terms of mapping possible states into a ground state, it makes sense to apply a set of successive deterministic rules to determine the ground state orientation.  I suggest finding the largest corner square and locating it in the lower right corner. If there are multiple candidates with equally large values in the corner square then use the next closest square as a tie breaker.  There are 16 squares, so you can eventually break all ties or declare degeneracy. There is one remaining \ operation that you can decide to apply in order to locate the larger of the two squares adjacent to the lower right corner at the bottom.  Again, you can use squares adjacent to these as tie breakers.


Is there away to change the metric used by the Early Stopping callback in Keras?

You can use any metric function that you specified when compiling the model.

Let's say you have the following metric function:

def my_metric(y_true, y_pred):
     return some_metric_computation(y_true, y_pred)

The only requirement to this function is that it takes accepts the true y and the predicted y.

When you compile the model, you specify this metric, similarly to how you specify build in metrics like 'accuracy':

model.compile(metrics=['accuracy', my_metric], ...)

Notice that we are using the function name my_metric without ' ' (in contrast to the build in 'accuracy').

Then, if you define your EarlyStopping, just use the name of the function (this time with ' '):

EarlyStopping(monitor='my_metric', mode='min')

Make sure to specify the mode (min if lower is better, max if higher is better).

You can use it just like any build-in metric. This probably also works with other Callbacks like ModelCheckpoint (but I have not tested that). Internally, Keras just adds the new metric to the list of metrics available for this model using the function name.

If you specify data for validation in your model.fit(...), then you can also use it for EarlyStopping by using 'val_my_metric'.


What is the vector value of [CLS] [SEP] tokens in BERT

First a clarification: there is no masking at all in the [CLS] and [SEP] tokens. These are artificial tokens that are respectively inserted before the first sequence of tokens and between the first and second sequences.

About the value of the embedded vectors of [CLS] and [SEP]: they are not filled with 0's but contain numerical representations like any of the other vectors in the embedding table. From the point of view of the model, there is nothing special about their embedded vectors, so they are trained normally like the others.

About masking proper names: you should take into account that BERT is a subword-based model. This means that words get split into subwords that are part of the vocabulary. Sometimes a word maps to only one token, but other times a single word maps to a sequence of several tokens. Infrequent words tend to be sliced almost at the character level. Depending on the data the model is trained with, the proper nouns you are referring to may be split into multiple subwords. You should take this into account when designing your masking procedure by either:

Before tokenizing, you replace the proper nouns with a text sequence that is included in the vocabulary and that is exclusive to them.
After tokenizing the original sentence, you check if the text contained a proper noun and if so, you locate the sequence of subwords associated to them and replace each subsequence with a special proper noun token ID.

Take into account that this is not similar to BERT's inherent masking, where the input token is masked but the loss forces the model to predict the original token. In your case, the original proper noun would be masked to never return.


Recurrent neural network (LSTM) dimensions error

EDIT after comments

  Also are arguments of Embedding layer OK?

Yes. But You need to pass return_sequences = True in the first LSTM layer so that it will pass sequences to the next LSTM layer.

From the docs

return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence.

  How are both these adjusted?

From the docs.

input_dim:
Size of the vocabulary, i.e. maximum integer index + 1. This determines the largest integer in the input data. The largest integer in the input should be no larger than the vocabulary size. This should be the

output_dim:  Dimension of the dense embedding. int >= 0

input_length: Length of input sequences, when it is constant. This argument is required if you are going to connect Flatten then Dense layers upstream (without it, the shape of the dense outputs cannot be computed).

I am posting code below for dummy data. The size of vocabulary has been taken 100.

from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
input_array = np.random.randint(100, size=(250, 2000))
input_y = np.random.randint(7, size = (250))

Y_dumy = pd.get_dummies(input_y)
X_train, X_test, Y_train, Y_test = train_test_split(input_array, Y_dumy,
test_size=0.1, random_state=0)

model = Sequential()
model.add(Embedding(input_dim = 100, output_dim = 64, input_length=2000))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2, return_sequences
=True))
model.output_shape

#Output shape should be:
#model.output_shape = (None, 2000, 64)
#3D tensor with shape: (batch_size, sequence_length, output_dim)

model.add(LSTM(200, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(Y_dumy.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=
['accuracy'])
model.fit(X_train, Y_train, epochs=50, verbose=True, validation_data=
(X_test, Y_test))`

  Where is the error coming from and how can it be solved?

I believe the error is coming because of absence of input_length. For similar errors please have a look at this post.

After the comments
The error is coming because of return_sequences =False in the first LSTM layer.


fast ML algorithms for binary classification with (large+sparse) binary input data

I mentioned in a comment that boosting might be a option. However, after second thoughts, I guess (as far as I can tell from the description of your data)  that you might be better off starting with logistic regression with regularization (lasso, elastic net, ridge). Why?

You want to learn „fast“ (not much tuning etc)
Your data tends to be of high dimension
You don‘t know what feature(s) are good predictors

This together makes me think, that 1) trying lasso would be the thing to start. If this fails, 2) go on with ridge.

Lasso can shrink the impact of features to zero (good in high dimensional data). However, this may lead to a situation in which „too much“ regularization happens. So you can try ridge, in which case features are „shrunken“, but will never become zero.

I don‘t know if you work with R or Python, but lasso/ridge is available in both. Make sure you find the right lambda (tuning parameter for regularization) by cross validation (this is not too expensive).

Here is a good R tutorial: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

For Python, there are also good tutorials around.

Some background can be found in „Introduction to statistical learning“, for which R and Python code is online. http://www-bcf.usc.edu/~gareth/ISL/


Why is my LSTM is working best with batch size of 2 and no hidden layers?

There are some papers which tell us that lower batch size may generalize better than large batch size. and large batch size may cause regularization in the model too. maybe that is the reason Bayesian optimization is suggesting a lower batch size for your dataset.
Please check below papers,

https://openreview.net/pdf?id=B1eyO1BFPr

https://openreview.net/pdf?id=B1Yy1BxCZ


Linear regression model with (categorical) predictor variables

Your interpretation is correct. I would also add that if Panelist1 increases by 1 unit and the other predictors are held constant, then the average increase in the response variable is 17.557 minutes.


WEKA Random Forest J48 Attribute Importance

If you must use WEKA, I would suggest looking in the Attribute Selection panel. In this panel, you can rank attributes by information gain, as well as look at which subsets of attributes perform best. To my knowledge, you can't get this information directly from the classifier in the WEKA Explorer, but it is quite easy to get in other frameworks such as scikit-learn: see this example in the docs.


Plotting an empty bin in a Seaborn histogram

Plotting empty bins can happen in Seaborn when plotting histograms with discrete data.
Plotting empty bins can be prevented by adding discrete=True.
sns.histplot(values, bins=dice_max, discrete=True)


A situation that validation accuracy is higher than test accuracy at its minimum loss region

Assuming the goal is to give the true performance of the model, then it should be the performance obtained by applying the final model on the test set. The fact that the performance on the validation set can be observed during training is irrelevant.
So the question becomes: which model should you select based on observing the performance on the validation set during training? You can select the model which gives the maximum accuracy, but the real performance is obtained when you evaluate it on a fresh test set.
The second question depends on which performance measure you want to optimize: if you want to optimize accuracy, then you should pick the model which obtains the highest accuracy.


The actual code is included in the notebook, if you click on the 'Code' button just above the cell with the code you provide you will see the code used to define the train_model function. If you look in that function you will see that this function can use different models based on the model_type argument, and in the line of code you linked 'lgb' means that a gradient boosting model from the lgb library is used.


Problem in Recommendation for categorical data?

Welcome to the site, I'll propose few alternatives below in increasing order of complexities.

Simple sorting criteria: Think what the user wants to see and create a score for sorting questions. Ordering of question based on some criteria like number of answers or number of views it has already received. Show top N based on score.
Derived sorting criteria: Use a combination of factors like number of answers, freshness of question, popularity of question based on number of views to create a derived score. Show top N based on score.
Adding Discovery to (1-2): The options above will penalize new questions and run into typical explore Vs. exploit trade-off in any recommendation system. You can alleviate that by adding a random bump to score of new questions so that they get a chance to participate.
Jointly learning categories and questions: You can setup your recommendation algorithm to actually work at question ID level and use categories as side information. Check this worked out example of LightFM library in Python. Coincidentally, it uses stackoverflow questions and categories as examples.

The first 3 options are easy to implement but biased. Fourth one will need some data transformations and I'm not sure if your objective is to model directly at question ID level.


How to set a newtwork with two objectives?

So this problem is less of a Deep learning problem and more of a logical problem. Your y1_train and y2_train can be modelled along with a pointer label that points what output to be considered in the output. Lets say we create the concatenated output as follows:

[[0/1],[y1_train], [y2_train]]

Where 0 could represent weather the label to be selected is y1 or y2 and so on.

But if you are planning to create a little more complex output and train different outputs on different loss functions, here is an article you should refer to Multiple output tutorial/examples and Custom Loss Function for Unequal Weighted Multiple-Output Node Regression


How many Layers in the original Yoon Kim CNN implementation?

Based on the model architecture from original paper below, it looks like there is just one dense layer at the end. Conv layers --> Max Pool over time --> Dense


How to approach for predicting semantic similarity between two phrases

What you are describing is one of the "standard" NLP problems faced in NLP and it usually referred to as "natural language inference" (NLI), or sometimes also as "textual entailment".
There is plenty of research in this kind of task, and its variants, like cross-lingual NLI (XNLI). I suggest you have a look at nlpprogress (link) or paperswithcode (link) to understand the landscape of benchmark datasets and SoTA models in NLI.


Most Efficient Post Processing with Python and Pandas

I'd do it this way:

helper dictionary:

In [79]: d = {'SD':[3], 'ND':[1,2,4]}

let's convert it to a Pandas Series:

In [80]: lkp = pd.Series({el:key for key,lst in d.items() for el in lst})

In [81]: lkp
Out[81]:
1    ND
2    ND
3    SD
4    ND
dtype: object

now we can map Location into State and group by it:

In [82]: df.assign(Location=df['Location'].map(lkp)).groupby('Location')['Value'].sum()
Out[82]:
Location
ND    254008
SD       200
Name: Value, dtype: int64


How to add Earlystopper in Classifier Model

Early stopping can be achieved with Keras callbacks.
early_stop= tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3)

history = model.fit(X, y, callbacks=[early_stop])

patience: Number of epochs with no improvement after which training will be stopped.
To define the meaning of "significant learning stopping", you will have to try with the parameters of Callback. e.g.
monitor="val_loss",  min_delta=0, patience=0, mode="auto", baseline=None, etc.
Keras_API


splitting mechanism with one hot encoded variables (tree based/boosting)

i have read online that if we have a categorical variable with q levels, the tree has to choose from ((2^q/2)-1) splits. For a dummy variable, there is only one possible split and this induces sparsity
i am not sure i understand this, say i have a column called color: red, green, blue,yellow , and i implement one hot encoding so is the number of splits that happen is 2^4/2 -1 = 3?...

You have the order of the operations wrong there (probably because the "((2^q/2)-1)" you quote above is misleading, but compare to the ESL quote in your image):  it's
$$ \frac{2^4}{2}-1 = 2^{4-1}-1 = 7$$
possible splits, namely:
(red green blue) vs (yellow)
(red green yellow) vs (blue)
(red blue yellow) vs (green)
(green blue yellow) vs (red)
(red green) vs (blue yellow)
(red blue) vs (green yellow)
(red yellow) vs (green blue)

if this increases as i have e.g. 2^25/2 -1, more splits means the tree is more likely to find a 'good split' for data at hand and lead to overfitting?...

It is certainly true there are more possible splits, and that increases the model's capacity and hence perhaps increases overfitting.  This is particularly worrisome if some of the levels are quite rare, and less so if you have a lot of data.

But what i don't understand is how this splitting chages with dummy variables.. does that equation hold or not for one hot endoded variables.

No, when you one-hot encode such a feature, the tree now must split on only one of those dummy variables (at a time).  So, considering all the $q$ levels' new indicator variables, you have exactly $q$ splits to consider.  In your example, they are the first four splits given above:
is_yellow=0 vs is_yellow=1
is_blue=0 vs is_blue=1
is_green=0 vs is_green=1
is_red=0 vs is_red=1


The main difference between supervised and unsupervised learning is the following:

In supervised learning you have a set of labelled data, meaning that you have the values of the inputs and the outputs. What you try to achieve with machine learning is to find the true relationship between them, what we usually call the model in math. There are many different algorithms in machine learning that allow you to obtain a model of the data. The objective that you seek, and how you can use machine learning, is to predict the output given a new input, once you know the model.

In unsupervised learning you don't have the data labelled. You can say that you have the inputs but not the outputs. And the objective is to find some kind of pattern in your data. You can find groups or clusters that you think that belong to the same group or output. Here you also have to obtain a model. And again, the objective you seek is to be able to predict the output given a new input.

Finally, going back to your question, if you don't have labels you can not use supervised learning, you have to use unsupervised learning.


How to extract crucial features to create an image

This is a bit similar, merge an image with style of another https://github.com/jcjohnson/neural-style

Maybe with the GAN you can choose cases where the discriminator predicts 50/50 to be cat or dog


machine learning on athlete performances to predict the time in a future race

I think the key here is what you believe to be dependent variables. You mentioned, for instance, a three month rest. Encoding the rest between races is likely to be a better idea than simply encoding the date of the rest as a lot of the date is redundant to what is actually being said. As with any machine learning algorithm, representation of the data is key and in many ways, the actual algorithm applied is less important.