Scaling the data iteratively one by one vs batch scaling

With the first approach, you are completely disregarding the global scale of the signal and only focusing on the relative scale. This will most probably hurt the performance of whatever system you train on that data (or any analysis you perform on it), compared to training on the globally scaled data, unless the relative values are the only important pieces of information in your signals.
I noticed that you already know the range in which the data is defined: (-127, 128). If you want to scale your data, why don't you scale the data with a fixed computation like $((x + 127) \cdot 2 / 255) - 1$ ?
Anyway, I don't see how a mere linear change in scale would save memory or help visualization, which seem to be your final goal.


The output layer is usually the same size as the last dense layer because we apply a loss function to train the model by comparing the last layer to what the output should be. If your output layer was bigger, it's less intuitive what your loss function should be, but the interpretation of your output would likely come from how you define this loss.


Why convolute if Max Pooling is just going to downsample the image anyway?

Max pooling doesn't down-sample the image. It down-samples the features (such as edges) that you have just extracted. Which means you get more approximately where those edges or other features are. Often this is just what the network needs for generalisation - in order to classify it doesn't need to know there is a vertical edge running from 10,5 to 10,20, but that there is an approximately vertical edge about 1/3 from left edge about 2/3 height of the image.

These rougher categories of features inherently cover more variations in the input image for very little cost, and the reduction in size of the feature map is a nice side effect too, making the network faster.

For this to work well, you still need to extract features to start with, which max pooling does not do, so the convolutional layer is necessary. You should find you can down-sample the original image (to 14x14) instead of using the first max-pooling layer, and you will still get pretty reasonable accuracy. How much pooling to do, and where to add those layers is yet another hyper-parameter problem when building a deep neural network.


Develop a “Credit Risk Scoring System” with R

First of all, you need to decide how you evaluate "good" vs "bad", which means picking a metric. From the top of my head you have three choices:

accuracy: simplest, but potentially misleading (if you have few defaults and predict all 0s, you still get high accuracy)
logloss: penalize assigning a high def probability to good quality credit and vice versa
ROC: sort the credit from best to worst.

Once you've made your pick, make sure your model can generate predictions in the right format (i.e. output probabilities and not just 0/1) and the go through the training / validation / test paradigm. This should get you started.


How to create pipeline for Neural Network models?

I am able to achieve this with below code

Thanks  @JahKnows

scaler = StandardScaler()
model = Sequential()

model.add(Dense(120, input_dim=46,activation='relu'))
model.add(Dropout(0.1, noise_shape=None, seed=None))

model.add(Dense(80, activation='relu'))
model.add(Dropout(0.1, noise_shape=None, seed=None))

model.add(Dense(40, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

pipeline = Pipeline([("scaler",scaler),("model",model)])
pipeline.fit(X_train,y_train, model__epochs=50, model__batch_size=20, model__validation_data = (X_test,y_test))

# evaluate the model
nn_pred = grid_search.predict(X_test)

print('Accuracy of NN on test is:',accuracy_score(y_test,nn_pred))
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))


Non-Real Time Data Augmentation for CNN Classification. What are the drawbacks?

Yes, they are mostly referring to real-time data augmentation.

It is not said that raw images are never used. Actually, it depends on what kind of data augmentation function you are using. Original images can either be kept intact or distorted.

Data augmentation si done real time pretty much 100% of the times for the following reason: data would be too big to be stored on your hard drive. Even the smallest image datasets could take hundreds of GBs after augmentation.


I need help on how I would go about solving this problem, my apologies for the vague title

I agree that it looks like an optimization problem, but the parameters are not completely clear. Here are some vague ideas I have about the design of such a system:

Let's assume a unit of time for which you have data and by which you attempt to predict things, for instance by the hour.
I think the first part needs to be about predicting energy needs by unit of time, based on past data and using any relevant factors as features (e.g. temperature, week day or not, etc.). I think this would be a reasonably standard time series regression problem.
The second part would be the optimization of energy consumption or carbon footprint: assuming we know the energy needs for a given period of time by unit of time, what are the optimal times for storing/discharging the battery? I don't know enough about the technical factors here, but I guess it's possible to simulate the target (say carbon footprint) based on different scenarios, and then select the optimal scenario. It seems to me that the problem at this stage could be simple enough to simulate all the scenarios and find the exact optimal times, that would probably correspond to a grid search. but in case this is too complex there are ML techniques which could estimate them, for instance genetic algorithms.


How to tune learning rate with HParams Dashboard on Tensorflow?

I guess the error in the example is that, instead of numbers objects were passed to HParams. You should handle the learning rate just as any other float parameter and just pass it to the optimizer. Actually hard to tell what you want exactly without a code example.


Is Annoy a machine learning algorithm to find nearest neighbor ? and is it similar to K nearest neighbor algorithm?

Don't have enough reputation to comment to a resource, so answering this myself.

About Annoy

Annoy is a library being used here for finding approximate nearest neighbours, approximate being the key word here.

Understanding K-NN and Approximate NN

Now, let's see what is the difference with example of a problem.

Say you have 10 entities (words / sentences / objects / anything) for which you have vectors.

1 vector for each entity.

Let's consider that the metric we're interested in finding for finding neighbours is cosine similarity / cosine distance

How many pairs would this form?
Roughly (10 * 9) / 2 = 45.

So, you can find cosine similarity between the 45 pairs and then find K-nearest neighbours you want. Pretty simple. This is your K-Nearest Neighbours algorithm.

Also given you are storing these 45 cosine similarities somewhere, you basically have to deal with storing 45 float numbers (cosine similarities can be between -1 and 1).

Now, let's scale up the problem. What if instead of having 10 entities, you have 10,000,000, i.e. 10M entities. Pretty easy to have that number of entities exist in real world.

Now, you'll need to find cosine similarity between (10M * (10M-1) / 2) =  4,99,99,99,50,00,000 pairs. Which is an enormous number! Basically, the complexity grows with O(n^2), where n is number of entities.

Now, think about finding K-nearest neighbours for any given entity. You'll also be running sorting algorithm for 10M similarities for it. Also, let's just think about storage again. Say 1 float number takes 64 bits (or 8bytes). How much storage do you need to store those many pairs?

4,99,99,99,50,00,000 * 8 bytes ~ 364 TeraBytes!

Getting a memory (RAM) of that size is near to impossible.

So, you understand the problem we're having with regular K-NN approach. We don't wish to calculate all pairs, and it's very difficult to store distances for those many pairs and calculate true K-nearest neighbours at runtime for any entity.

This is where Approximate Nearest Neighbours comes into picture. These say, that given your need of the problem is almost always going to be to get only the K Nearest neighbours, why bother calculating similarities for any given entity to those entities that have zero chances to appear in neighbourhood?

Think about a space where you have clusters of points. Say you need to only find the 3 most nearest neighbours for every point for your problem, then you would never need to find similarities between points of different clusters. Right?

This is the idea that Approximate Nearest Neighbour algorithm runs on.

  For any given entity, instead of calculating similarities/distances to each of the other point, it chooses X other points and only calculates distances to those.

This reduces the complexity from O(n2) to just O(n*k). Storage becomes easier, Calculation becomes faster.

A hint of Annoy's implementation

While I'm myself not familiar with all implementation details of Annoy's ANN approach. Usually, most ANN approaches require building a Cluster Index first -- which basically decides the X leaders with which you'll find similarities. Then, the cluster index is used to find Approximate K-nearest neighbours for all given points.

The reason why this is called 'approximate' is that the cluster building process is not perfect. If it were, it'll take days or years of computation and defeat the purpose of reducing complexity. It is approximate cluster indices. Which means, sometimes you can miss out on one of the most similar entity appearing in neighbourhood of given entity. Usually, however this doesn't happen so much that it affects practically. Spotify, Facebook, Google and many large companies are using approximate nearest neighbours approaches.

Annoy is one of the most widely used library for doing approximate nearest neighbours.


Inconsistency on A. Graves' original Connectionist Temporal Classification (CTC) paper?

The lecture I take from the paper is not that equation (6) is a recursive form of (5). In (5), the equation refers to label sequence $l$, while (6) refers to $l^\prime$, which is a modified version of $l$ with blanks added to the
beginning and the end and inserted between every pair of labels. See the paragraph between equations (5) and (6):

  To allow for blanks in the output paths, we consider a modified label sequence l, with blanks added to the beginning and the end and inserted between every pair
  of labels. The length of l is therefore 2|l| + 1. In calculating the probabilities of prefixes of l we allow all transitions between blank and non-blank labels, and also those between any pair of distinct non-blank labels.


I'm not an expert in this field but you should take a look at the work of Bhargav Srinivasa Desikan, a gensim contributor, who works a lot with topic modelling.

He has a couple of notebooks on his github account which could interest you, especially this one (should be pretty much your use case if I understand your problem correctly).

The aforementioned notebook evaluates Latent Semantic Indexing, the Hierarchical Dirichlet Process as well as the Latent Dirichlet Allocation for identifying topics.


Support vector machines in R: Finding the equation of a hyper plane (in 6 dimensions) and showing it's correct

I over complicated the answer I was looking for. If you have the equation of the hyperplane, you can test it against each point in the dataset. If all points of class 1 are ALL (above or below) the intercept and the converse is true for all points of class -1. Then the hyperplane separates the two classes


Implementing RMSProp, but finding differences between reference versions

RMSProp is indeed an unpublished method, and in the lecture Geoffrey Hinton gives just the general idea behind RMSProp - to divide the gradient by a moving average of the gradient magnitude. The lecture has disappeared from YouTube but you can find the slides in the end of this PDF:

https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

When this principle is applied to Stochastic Gradient Descent, the update rule you showed is obtained. Since Hinton did not propose an exact algorithm, this principle has been applied to different optimization methods. I agree it's confusing all these methods call themselves RMSProp.

climin implements RMSProp with Nesterov momentum. Momentum methods try to avoid the oscillation that often happens with SGD by slowly changing the current direction of updates. The algorithm given in climin documentation introduces the $\beta$ parameter that controls how much of the previous update direction is retained. Nesterov momentum is implemented by first taking a step towards the previous update direction $v_t$, calculating gradient at that position, using the gradient to obtain the new update direction $v_{t+1}$, and finally updating the parameters. The climin implementation also includes the smoothing term $\iota$ inside the square root for stability (1e-8), even though it's not mentioned in the documentation.

The implementation in the Downhill library is based on the algorithm described in the paper by A. Graves. In the article (equations 38–40) the square of the average gradient is subtracted from the average square gradient. Apparently the idea is to approximate the variance of the gradient instead of its magnitude (recall that the variance of a random variable is equal to the mean of the square minus the square of the mean).


does entity recognition comes under classification problem?

Simply put, Named Entity Recognition (NER) is a multi-class structured prediction (classification) problem, so you have a sequence of words and you want to label each one most of the time with these labels ( start-of-a-person-name, continue-of-a-person-name, start-of-an-org-name, continue-of-an-org-name, start-of-a-location-name, continue-of-a-location-name, other). Note that these are not the only classes, classes with different granularity can also be used.

If you want to solve this problem with SVM you can use StructSVM or other  variations of SVM for structure prediction. Though the common baseline for this task uses Maximum Entropy (Maxent)(log-linear) models.


Acceptable variation in accuracy of each k fold when using K-Fold Cross Validation?

It's indeed a quite large variation, but nothing alarming since 9 out of 10 folds are within the range 0.8 to 0.9.

There are many possible factors: yes a fold can be easier than another one by chance, but it also indicates that the training process is a bit unstable. Increasing the number of instances and/or reducing the number of features often helps reducing the variation.


These are not (direct) assumptions for linear regression.
But rather for OLS-Ordinary Least Squares which is widely used to estimate the parameter of a linear regression model. OLS estimators minimize the sum of the squared errors (a difference between observed values and predicted values).

In order to guarantee finding best possible parameters, we make these assumptions (as with every optimisation)

But just because they are not present that does not mean that linear regression wont work.

Regarding your other models: If they use the same optimisation, than they have same assumptions.


How to predict next year's gross revenue given this year's data?

Predicting next year's revenue from previous years' revenues is Time Series Forecasting.
You will need data from many previous years to do this. One year is clearly not enough.
Just think about it this way: Can you predict tomorrow stock's price given only today price?


Given one language ngram model, how do I compare likelihoods of two texts of different length?

I didn't find any good explanations or papers on this topic, other than things about category based models and multigrams (parts of words). So, I came up with one myself. I'm using Java, but here's the code for the length normalization translated to Python. Also, this is assuming that you're adding the log probabilities for each word together, like KenLM does for the total sentence (or phrase) score.

# Get likelihood normalized by length and number of words.
# Since the likelihoods of each word are added together, phrases with more words tend to
# have a lower probability.
def getNormLikelihood(phrase):
    score = getLikelihood(phrase) # The normal score
    numWords = phrase.count(' ')+1 # The number of words in the phrase.
    # Get the absolute difference between the
    #   average word length of all words and the average word length of the phrase.
    # Then, weight that weight by the reciprocal of the normal score, such that the
    #   lenWeight is less if the overall score is less.
    # Intuitively, this means that we care less about normalizing by word length the more
    #   common the phrase is.
    # This effectively solves the problem of giving a greater likelihood for long words
    #   that aren't even in the language model (and are given the default "<unk>" score).
    lenWeight = abs(len(phrase) / numWords - avgLen) * (-1/score)
    return score / (1/lenWeight)

How you want to use this normalized score, and how effective it is, depends on your application. In my case (splitting words), I used this normalized score to score each splitting option, but then also added (logarithm) the normal score of the option with context, which you could consider the prior for the option (in Naive Bayes terms). I then added an overall prior (manually picked, but could be tuned) to the original, unsplit option, for the probability of needing to split a word. The normalized score greatly helped, and with these probabilities together, I was able to get 100% accuracy for my test cases.


What is the most appropriate machine learning approach for this scenario?

This certainly looks like a basic optimization problem. However, looking from the Machine Learning perspective, what you describe can be framed as a multilabel classification. In that case, you treat the values of A and B as your input, and try to predict the values of those five parameters, which you mentioned:

A, B -> x1, x2, x3, x4, x5

The only pitfall is that 100 records can be insufficient to train a proper classification model, but if you can obtain more training data, this approach seems solid to me.


Your algorithm is very similar to a Genetic Algorithm, which has almost the same 3 iterative steps.

You would be missing a Termination condition to end the search for an answer, once your reach an acceptable one.

You can probably adapt your algorithm to fit under a formal GA search heuristic.


Visualization with many lines, colors, and markers

You might want to look at these kinds of error band line plots in seaborn:

https://seaborn.pydata.org/examples/errorband_lineplots.html
You could compute this for each colour/ day of the week. The central line would be a mean value of each colour/day, the shadow being the distribution.
Then it's probably best to look at days individually to compare the distributions of functions. A box-whisker plot might be good to compare distributions, although this is typically only for 1-D data.
EDIT
You can limit the y axis with plt.ylim(0.25,1) but the previous might look more appealing


ADAM algorithm for multilayer neural network

It is written right in the algorithm description that $g_t^2$ is elementwise square.


Predicting financial data (choosing a model)

Well, evaluating your model just by comparing matrixes can be pretty hard.
In your case, your matrixes are not done with the same number of row classified 1 or 0, so comparing them is really hard. Let's take an example : your logistic regression classifies around 64000 as 1, while your RandomForest only classifies around 4500, so comparing these data by this matrix is pretty hard.
I'd suggest you to use the ROC AUC metric, that is really useful to compare models. You might find many info on this subject in the Internet. The closer your AUC is than 1, the better your model is. If it's 0.5 or less, your model is less efficient than a random classifier.


Any good hash will be uniformly distributed, which means that you can assume a uniform distribution when you apply modulo n, as long as $n < 2^{M/2}$, where M is the number of bits in your hash, see here. So for SHA1-32 you would at most modulo by $2^{16}$.

There is no approach to calculating an integer value; what you have there is an hexadecimal representation of a hash, you just need to convert it to a numeric type if you obtained it as a string. XXH32() and XXH64() both already produce an unsigned int output.


Is it possible to share and merge classification models?

I believe that I've found an answer here and here.


Seaborn Violin plot error No loop for unfunc add

I had tried the suggestion from @foxthatruns's answer to no avail. I found that changing my numeric column to float64 solved the problem (reference).

df['my_column']=df['my_column'].astype('float64')

This was done with Python 3.7, seaborn 0.9.0, numpy 1.16.4.


AttributeError: 'DataFrame' object has no attribute 'as_matrix' is the error

It could be that your code used to work and now it doesn't because you updated your pandas package.

If you check pandas.DataFrame.as_matrix's documentation you will see that this method has been deprecated.

Instead, you should use:

X_mat = X[['height', 'width']].values()


way to add target delay on TimeseriesGenerator from keras.preprocessing.sequence

You can use

  pandas shift

By default the TimeseriesGenerator performs a lag of 1 timestep. Then, a workaraound is to create a pandas Series, and use the shift function.
Let train be your sequence, and let forecast be an int representing the "size" of your lag.

Eg:

label_train = pd.Series(train)
train_generator = TimeseriesGenerator(train, label_train.shift(-forecast+1).values)

Obs: Be careful to pad your sequence after performing the lag.


How are per-layer-detected-patterns in a trained CNN plotted?

The image you posted is derived from figures which depict the basis functions of a Convolutional Restricted Boltzmann Machine.

I found them in this ICML paper (behind a paywall, sorry) and this PhD thesis, both are by Honglak Lee, who was advised by Andrew Ng.

Strictly speaking they are not visualizations of a feed-forward Convolutional Neural Network, which might be why you are having a hard time interpreting what they depict. I don't really know much about RBM's myself so I'm afraid I can't help with their interpretation.

As far as visualizing CNN's, there are several common methods. You can visualize filter weights directly, find input images which maximally activate a filter, plot activation as a function of image occlusion, etc.
This page has a good summary of techniques, also this paper may be useful.


how to create input file for collaborative filtering

This data set is in a sparse format. Most ratings are unknown, which is a common scenario in recommender systems. This data format is much smaller due to leaving out all the unknowns. You can interpret this data as follows:

user, item, rating

What this means is that user U has rated item I with rating R. Every UxI combination not in your dataset is unknown and has to be predicted using collaborative filtering.


tf.trainable_variables() returns blank list outside model_fn

This problem is resolved... tensorflow.python.framework.errors_impl.NotFoundError: Key output_bias not found in checkpoint – Actually there were model.ckpt-10949 files.. and tensorflow was picking up this checkpoint automatically... these files seem to be incomplete or corrupt. After deleting 1model.ckpt-10949* files key NotFoundError went away


You can replace the third line with the following import
from tensorflow.keras.wrappers.scikit_learn import KerasRegressor, KerasClassifier
Hope this helps.


Are there studies which examine dropout vs other regularizations?

Two points:

Dropout is also usually compared with neural networks ensembles. It seems it has some of the performance benefits of training and averaging several neural networks.
Dropout is easier to calibrate than regularization. There is only one hyperparameter which is the dropout rate and people widely use 0.5 while training (and then 1.0 on evaluation of course :)), see e.g. this TensorFlow example.

Anyhow, I am a little skeptical of neural networks empirical studies. There are just too many hyperparameters to fine tune, from the topology of the network to the gradient descent optimization procedure to activation functions and whatever it is you are testing like regularization. Then, the entire thing is stochastic and usually performance gains are so small that you can hardly statistical test for differences. Many authors do not even bother doing statistical testing. They just average cross-validation and declare whatever model had the highest decimal point gain to be the winner.

You may find a study promoting dropout only to be contradicted by another promoting regularization.

I think it all boils down to aesthetics preferences. Dropout IMHO sounds more biological plausible than regularization. It also seems easier to calibrate. So, I personally prefer it when using a framework like TensorFlow. If we have to use our own neural network, which we often do, we will use regularization because it was easier to implement.


emphasise some observation weights more than the others

If you have a date variable (or something similar), you can create a weight using this.

If you're using XGBoost, there is an option to specify a weight for each instance when creating the DMatrix - feed your observation weighting in here.


User-product positive (click data) available. How to generate negative (no-click data)?

So, there are two problems.

Recording impressions (shows)
How to deal with non-impressions

For (1) you should be recording this information. If it is not currently being recorded you should start recording this information. Given that you do not have this information you want to provide recommendations. Fortunately, with just click data you can still create a utility matrix, see 9.1.1.
You could then use user- or item-based collaborative filtering as described in the paper. This is basically an exercise in populating the utility matrix and trying to find "scores" for unclicked items. Your recommendation would be an unclicked item with the highest score.
For (2) you will still make recommendations on unclicked items. So, that alone is not an issue. You will want to optimize your impressions however. You also cannot have full knowledge where a user can see all possible options. You need to record impressions and understand a number of things.

show rate of an item
click rate of an item
how to incorporate new items
how to optimize which items to show

This is a huge topic and basically this is the problem domain of online advertising. However, a recommendation engine tries to find items of interest in the long tail, which is a bit different from ad optimization. This is a feedback loop to evaluate your recommendation. A/B tests are common. You will want to test click rates and recommendation errors between your current system and new system.
Also see here and here.


Text Classifier with multiple bag-of-words

I cannot comprehend the question completely but let me see if I got this right.

You want to train a classifier. You have different columns for subject and body. You have now pre-processed it in a manner such that subject and other meta information have been removed.

Now I shall try to answer your question. Keep in mind that, this is how I would approach the problem (based on experience with similar task in the past) but there could be better ways or approaches.

I would not remove the subject column in the first place. I would also not train the classifier on two separate bag-of-words. I would append both, the subject and the content, into one single document and then classify them accordingly with one unit of bag-of-words instead of two.

Why would I do this? There could be times when the subject could have keywords or something that could lead to better classification than the body of the email. The vice versa could also be true. Hence, without having seen the entire dataset, and to be on the safer side, I would also give this approach a shot. Compare both of your approaches and then see where you have better scores of accuracy, precision, recall and other metrics that aids you in your conclusions.

Ways to accomplish this? Yes, NLTK is one way to get the bag of words. However, I would go ahead with scikit-learn package to train my classifier. I would also run LSA/LDA/HDP (Topic Modeling Algorithms) to see what constitutes my data. To do this, I would use Gensim.

http://scikit-learn.org/stable/

https://radimrehurek.com/gensim/

Additional tutorials to help you get started with text classification if you are new to it (that is what it seems like, pardon me if I am wrong). These will also help you in achieving better results to begin with:

https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a

http://zacstewart.com/2015/04/28/document-classification-with-scikit-learn.html

Although both the tutorials I mentioned primarily concentrate on binary classification, you can also achieve fine-grained (multi-class) classification in similar manner.


What should I study to find optimal value of best feature combinations in machine learning?

Welcome to DS.SE!

As you already mentioned, machine learning is not about finding the best values of features regarding a function. Also, loss functions usually measure the errors of predictions and not the quality of the feature values (e.g. " the best product y").

What you are looking for is called optimization or search algorithms but not learning or prediction. If the search space (i.e. the combination of all possible feature values) is not that large, you can just perform an exhaustive search (trying all combination). If the search space is large, then smarter approaches might be useful. These techniques need a function (usually called objective function) to optimize (finding the combination of input values that results in a min/max outcome).

In general, there are two approaches: mathematical modeling (i.e. linear programming) and metaheuristics (e.g., evolutionary algorithms and swarm intelligence). Mathematical models can be fast and guarantee to find the optimum when the objective function is well-defined and the underlying assumption are met (you may need to simplify your constraints sometimes). If these conditions are not met, then metaheuristics usually can find a set of approximate solutions.


why this naming convention for padding as "Same" and "Valid" in keras

Using valid will essentially use as much of your input as possible, such that the dimensions continue to work. This means there is a chance some input will be trimmed (removed).

same on the other hand, will add padding to allow e.g. the number of filters/convolutions you specify to be applied.

The reason not to simply have padding=True as an option, is that there are more than two options (see snippet from the docs below). Also, that would not be very explicit: using same or valid makes it crystal clear, what the result should be using the given approach.

This is what the documentation says:

  padding: One of "valid", "causal" or "same" (case-insensitive).  "valid" means "no padding".  "same" results in padding the input such that the output has the same length as the original input

Take a look at this nice answer with a simple example.


how do I find out if my random forresrt is overfitting?

The standard method to detect overfitting is to evaluate the model on both the training set and the test set. If the performance obtained on the training set is very high but much lower on the test set, then it's likely that the model is overfit.
Naturally this method works only with a proper test set: if there has been some data leakage from the test set to the training set, then the evaluation on the test is biased (the performance value is unreliable).


generalized likelihood ratio test (GLRT)

Likelihood-ratio tests are a mainstay of classical hypothesis testing. The idea is to form the likelihoods of the two hypotheses under consideration, and choose the one with the highest likelihood if their ratio is sufficiently large. Hypotheses come in two flavors: simple, and composite. Simple tests are those for which the hypothesis uniquely defines the distribution; e.g., the mean is that, or the variance is that. If it is not simple, it is composite; e.g., the mean is not equal to something, or the variance is less than something. Generalized likelihood ratio tests apply to composite hypotheses, and the goal is to find the distributions out of all possible options in the hypothesis space that maximize the likelihoods, and consider their ratio.

In Bayesian statistics one considers the posterior probabilities instead of the likehoods, and the corresponding ratio is termed the Bayes factor.

Supervised/unsupervised learning is something else.

I suggest you consult a textbook for worked examples; I think my explanation does not do the subject justice.


Assuming 2 partitions since, for index in range(0,2) and a fixed bunch size of 3 since, start_range + 3
The shape of your y_true should be (nb_samples, 2), where the first vector of 2nd dimension is the ground_truth and the other is partition_index
import numpy as np
import tensorflow as tf
import tensorflow.keras.backend as K
from tensorflow.keras.losses import mean_squared_error

y_true = tf.Variable(np.array([[1.5, 0],[1.2, 0],[1.3, 0],[1.6, 1],[3.0, 1],[2.25, 1]]), dtype=tf.float32)
y_pred = tf.Variable(np.array([[1.35],[1.24],[1.69],[1.55],[1.24],[1.69]]), dtype=tf.float32)

def get_two_splits(y_true):
    child_y = y_true[:, 1]
    child_y = tf.expand_dims(child_y, 1)
    y_true = y_true[:, 0]
    y_true = tf.expand_dims(y_true, 1)
    return y_true, child_y

def some_processing_on_bunches(y_all):
    y_true, y_pred = get_two_splits(y_all)
    return mean_squared_error(y_true, y_pred)

def custom_loss(y_true, y_pred):
    n = K.shape(y_true)[0]
    y_true, partitions = get_two_splits(y_true)
    partitions = tf.reshape(partitions ,[n, ])
    partitions = tf.cast(partitions, tf.int32)

    y_elements = tf.dynamic_partition(tf.concat([y_true, y_pred], 1), partitions, 2)
    losses = tf.vectorized_map(some_processing_on_bunches, tf.stack(y_elements))
    return tf.reduce_mean(losses, axis=1)

K.eval(custom_loss(y_true, y_pred))
>>>array([0.05873336, 1.1379    ], dtype=float32)


How to see actual numbers in a hist on R?

Hope this may help you

hist(iris$Petal.Width,labels=T)


Algorithm selection rationale (Random Forest vs Logistic Regression vs SVM)

I suppose I will suggest as a starting point and expand on what you suggested by just adding the following

Knowing the type of data you are working with and it's characteristics, (categorical, supervised/unsupervised, data size etc.).
Knowing what accuracy requirements you need, timeframe and computational power you have at your disposal vs accuracy and really answering "why, am I trying to solve this problem?"

After answering these questions you can at least narrow down slightly what you may use (and eliminate those you clearly don't believe fit).  After that, I suppose it's trial and error, experience and comparing to others who dealt with similar datasets and problems.

I have this crude flow chart I found in my favourites from the scikitlearn website. Not sure where I found it to be honest.  Take it for what you will, hopefully it helps somewhat:

https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html


R On Cloud provides a browser-embedded R-console.

Jupyter.org evolved from the IPython Project (the language-agnostic parts of IPython); supports Python 3, Julia, R, Haskell, Ruby, etc.


What is a distribution-wise asymmetric measure?

What is the meaning of distribution-wise asymmetric measure?

The (forward) KL-divergence is distribution-wise asymmetric because if you calculate it as $$D_{KL} \langle P(X) \Vert P(Y) \rangle$$ where $P(X)$ and $P(Y)$ are two different probability distributions with the latter being the reference distribution, then $$D_{KL} \langle P(Y)\Vert P(X)\rangle \neq D_{KL}\langle P(X)\Vert P(Y)\rangle.$$ In other words, the reverse KL-divergence is not equal to the forward KLD. If forward KLD were symmetric, then the above would be an equality, not an inequality.

Is there a symmetric measure?

A distribution-wise symmetric measure would, for example, be mutual information:
$$I(X;Y) = H(X)+H(Y)-H(X,Y) = D_{KL} \langle P(X,Y) || P(X) \cdot P(Y) \rangle,$$
where $H(X)$ is the entropy of a variable $X$'s probability distribution, since $I(Y;X) = I(X;Y)$. Mutual information is a special case of the KLD in which the joint distribution is measured against the product of marginal distributions.

What are the rules that a quantity should follow to be qualified as a statistical metric of spread?

The three axioms that a distance metric should meet are:

Identity of indiscernibles
Symmetry
Sub-additivity or triangle inequality

Since mutual information does not obey the triangle of inequality, it does not fit the full criteria for being a distance metric. Instead, variation of information does meet all of the above requirements and is a true metric:
$$VI(X;Y) = H(X,Y) - I(X;Y)$$
where $H(X,Y)$ is the joint entropy.


After the training phase, is it better to run neural networks on a GPU or CPU?

That would depend entirely on what software platform you're running on, not on any particular characteristic of neural nets or their constituent objects. For example, Theano leverages GPUs because they're optimized for matrix math, which neural net weights and activations are often expressed as. On the other hand, they can also be expressed as sets, which makes them ideal for set-based languages like SQL; I prefer this approach, since sets are more flexible (you can, for example, operate easily on ragged dimensional sets, unlike with matrices) and SQL is a somewhat portable standard that makes it easier to think about neural net layers conceptually. As my screen name suggests, I use SQL Server, which is one of many database server platforms that can do some serious number-crunching on very large neural nets, all without the use of any GPU power. Long ago I also programmed neural nets in VB.Net, VB 6 and C# that efficiently made use of arrays, without leveraging GPUs at all.

I think the choice may come down to a combination of your existing skill set, what architectures you have access to, your portability requirements, code maintenance issues and all that, not to any explicit linkage between specific hardware components and the inner workings of neural nets. There may be slight advantages to modeling ragged matrices in SQL or to the matrix-crunching power of GPUs, but I don't think there's much practical difference in most scenarios, nor can I think of a way of testing such a comparison without getting into an apples vs. oranges conundrum. That being said, within a specific platform, it may be true that "GPUs are only needed for the training phase" and "once trained, it's actually more efficient to run them on CPUs." That I can't comment on, since I still with SQL and the .Net stack and avoid Theano etc. altogether (though I respect their capabilities). I hope that helps.


how to compare different sets of time series data

Pretty interesting question!

First of all have a look at my edit as your question was a bit unclear according to the standard terminology. you have a set of Time-Series and you want to detect the outliers (abnormalities).

Your approach is pretty clear and logic and shows understanding of the problem and solution. The point is about the way you chose to apply it.
K-means is not the best way. I would like to point out that choice of 2 clusters are very smart here as you hope the clusters get formed based on a normal/abnormal structure. It just does not work well in practice if the features you extracted from your time-series are not inhibiting the abnormal behavior.
I assume Embedding algorithms are the right way to do so. Most probably if you apply a simple PCA you will see abnormal time-series somewhere further than others. Bellow I write the code. Try it and drop me a line if it didn't work so I go for more sophisticated solutions (e.g. you may construct a phase-space and see your data there or monitor the recurrence of time-series, etc.)

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

pca = PCA(n_components=2)
X_new = pca.fit_transform(X)
plt.figure(figsize=(10,10))
plt.plot(X_new[:,0],X_new[:,1],"*")
plt.show()

where X is a 10xN matrix in which each row is one time-series.

You may choose more components for PCA and check different PCs against each other.

Anyways the problem is not that difficult and if it did not work I will update my answer with another solution.

Hope it helps and good luck!


I am a programmer, how do I get into field of Data Science?

Focus less on gaining skills and more on gaining experience. Try to actually solve some problems and post your work on github. You'll learn more in the process and be able to demonstrate knowledge and experience to employers, which is much more valuable than having a supposedly deep understanding of a topic or theory.

Data Science is a pretty loaded field these days so I'm not sure what kind of work you specifically want to do, but assuming that machine learning is a component of it then kaggle.com is a good place to start. In terms of goals, if you're able to work with the data in pandas/numpy/scipy, build models in sci-kit learn and make some pretty graphs in seaborn, ggplot or even matplotlib then you won't have a problem getting a job from a skills perspective -- especially if you have code samples and examples to demonstrate your abilities. If you get stuck then stackexchange will either have the answer or you can post a questions and you'll have an answer shortly. Once you're doing the work for a living then you'll learn even more, likely from a senior team member who mentors you.

Best of luck.


What I would suggest is to have put the results as a Data frame with
pd.DataFrame(cv.cv_results_)

And then you have the data in a dataframe which is easier to handle.
For the other question (in your comments), once you have the data in a nice dataframe is just about data visualization. How can you put it in a nice visualization?
The easier are either 2d plots or 3d or even contour plots. By googling them and python you will get nice tutorials on how to do it


Spatially constrained geospatial similarity

I assume you are trying to find a suitable distance metric based on features of different areas (although spatial distances might also easily be plugged in). In that case, I would first try to make sure the different features are correctly scaled, for example, to zero mean and unit variance.

If the result does not seem right, I would also try looking at different distance metrics. A simple alternative example is the L1 norm:

L1(a, b) = sum_x |x_a - x_b|


Regions /patterns in images that creates classification

I think visualizing convolutional neural networks is what you want.

CS231n is a good place to start, here is a step-by-step guide about how to visualize the activations, first-layer weights, and shows how to retrieve images which maximally activate a neuron.

Also there's a demo which visualized multiple convnets (including ResNet50, Inception v3, etc). You may learn a lot from this demo (at least I do).

If you're interesting in details, this well-known paper may help you.


Is a new datapoint similar to my reference set?

If you have many features, likely meaning many columns in your table of data, then you could try clustering. Something as simple as k-Nearest Neighbours could work nicely.

You would first fit a model using your available data, then look at the resulting clusters. Each cluster will represent combinations of your features.

Next you can put new data points into the model and it will tell you to which cluster it would best fit.

This would actually be a fully fledged predictive model! It is unsupervised learning, because you are not using and pre-existing labels (for each cluster in this case).

If you are working with Python, the Sci-kit Learn documentation is a good place to start. There are many other clustering algorithms, which might work better for you, depending on your type of data (if it is spatial, DBSCAN is good, for example).


Check your version of JPS and make sure it's the same as the version of java that you are running.  Sometimes you start out with an out of the box jdk install, upgrade, set alternatives for the java bin, but still have the original jps binary being referenced.

Run ps -ef |grep java and look for the resource manager threads.  Maybe it's actually running.  If it is, try update-alternatives --config jps to see what binary jps is pointing at and compare it with the java binary that you are using.

If the resource manager is not running, it's time to do some basic linux troubleshooting.  Check log files and barring that check actual command output.

On the system I'm looking at now, the log files for resource manager are placed in the hadoop-install/logs directory in yarn-username-resourcemanager-hostname.log and yarn-user-resourcemanager-hostname.out.  Your configuration may place them in /var/logs or what have you.  Also, have a look at the syslog.

If the logs don't yield any good information, which can happen, my process is to generally try to figure out the command line from the startup script (usually by prefixing the command line with echo), and then trying to run the command directly to see the output as it comes out.

I have actually run into this problem before, but I can't remember the specific issue.  I'm sure the same result can manifest itself from a variety of problems.  Considering that you are as far as you are in the process of getting set up, I believe it's likely to be a minor configuration issue.


Reason behind choosing Neural Network for classification

I wouldn't give any reason to make an choice a priori, based on such a broad description.

Instead I would build models for NN and SVM (or any other that you were considering), train them on the data then compare using cross-validation and a suitable metric for your problem - such as accuracy, F1 score, AUROC or log loss. You may also want to note characteristics other than performance metric if you care about costs such as time to train, amount of computing resource required etc.

Experience may tell you likely best choices from the problem domain. For instance, if you are classifying photographic or scanned images, then currently a convolutional neural network would be a very strong candidate, as these have repeatedly shown themselves as best performers with real-world image data.


The number of people in each category is not defined by the number of rows but by the number in the Freq column. You therefore have to take the sum of those values and plot the result, this can be done by supplying the column which holds the value you want to use on the y-axis to aes() and use stat="identity" for geom_bar:
library(tidyverse)
library(titanic)

titanic <- as.data.frame(Titanic)
titanic$Survived <- as.factor(titanic$Survived)

ggplot(titanic, aes(Survived, Freq, fill = Survived))+
  geom_bar(stat="identity")+
  ggtitle("Barplot to represent Passenger Count who Survived vs who Died")


Creating an Object Detection model with images and coordinates of bounding boxes

So, I found the answer myself. Basically, if the grid cell contains the centre of the bounding box then the grid cell will have a $p_c$ of 1 otherwise 0 if there is no object.


def __find_best_split(X, Y, i):
best_split = {}
for column in X:
    information_gain,split = (2,3)#self.__find_best_split_for_column(column)
    if not split: continue
    if not best_split or best_split["information_gain"] <information_gain:
        best_split = {"split":split,"col":column,"information_gain":information_gain}
C= (best_split)
return C

I ran this code in python 3.7 and it gave me the correct output: {'split': 3, 'col': 'a', 'information_gain': 2}. Also as pointed out by Ben Reiniger, it should be if not split: continue rather than if split: continue. In your case based on the indentation, C should never get defined or if indented properly, it should give empty dictionary. In both the scenario, getting error assert type(C) == dict does not makes sense.


Implement the below mentioned techniques and check

Add Batch Normalization
Increase the learning rate
Standard/Normalize the inputs if you have not done it already


What does 'energy' in image processing mean?

I quote from here which our friend has provided.

  Energy in this context is not, strictly speaking, the same as the conventional notion of energy in physics and the other sciences. The two concepts are, however, closely related, and it is possible to convert from one to the other:

  where Z represents the magnitude, in appropriate units of measure, of the load driven by the signal.

Also for figuring out the interpretation of energy in this context you can see here.

After a while, I found a bit better explanation for energy as you can read the answers from here.

  Energy is used to describe a measure of "information" when formulating an operation under a probability framework such as MAP (maximum a priori) estimation in conjunction with Markov Random Fields. Sometimes the energy can be a negative measure to be minimised and sometimes it is a positive measure to be maximized.

or you can read:

  Energy is a fairly loose term used to describe any user defined function (in the image domain).


Have you tried using the mean value theorem for integrals? You can relate $f(v)-f(w)$ to $\nabla f$ and use some simple estimates to get something like the above.


The term is Oracle.
Some references:

SO question describing the term
Scientific articles related to machine learning using the term


Help with algorithm approach for computer vision

this problem set is in the realm of object detection and image classification,you need have need a good dataset from your client or scrape one yourself,you will need to perform real time object detection in the assembly line so for that case you will need a have a really good convnet that has high accuracy.One way to have such a high accurate model is to perform transfer learning using your preferred models such as inception,caffe,lenet etc. you will need to delve more into learning about anchor boxes,for this kind of problem look at the keras implementation of Yolo algorithm https://towardsdatascience.com/object-detection-using-yolov3-using-keras-80bf35e61ce1


Feature selection: Information leaking if done before CV-split?

If you do not use target for feature selection, then there is no leakage and you can apply this on the whole dataset. The methods that you mention: checking correlation between features and checking variance of each feature belong here.

If you do use target for feature selection, then you should use 'train' subset only. Otherwise there is a leakage. The examples here would be (i) you check correlation between each feature and target (ii) use build Random Forest model and take best features


Some of the GREC shared task challenges included a named entity recognition & coreference resolution component (i.e., disambiguation), but I don't think they've run GREC since 2010...?

https://sites.google.com/site/genchalrepository/reg-in-context/grec-ner


How to do k-folds in python whilst splitting into 3 sets?

I don't think there's a builtin way to do it, but the two methods you've mentioned combine pretty nicely to do the job:

kf = KFold(n_splits = 5, shuffle = True, random_state = 2)

for train_index, test_index in kf.split(X):
    X_tr_va, X_test = X.iloc[train_index], X.iloc[test_index]
    y_tr_va, y_test = y[train_index], y[test_index]
    X_train, X_val, y_train, y_val = train_test_split(X_tr_va, y_tr_va, test_size=0.25, random_state=1)
    print("TRAIN:", list(X_train.index), "VALIDATION:", list(X_val.index), "TEST:", test_index)

(I also opted to keep things as dataframes.  You could even use "nested cross-validation," using another CV instead of the train_test_split inside the loop, depending on your needs and computational budget.)

For the question of normalizing data, you don't want to let information from the testing fold affect the training, so normalize within the loop, using only the training set;
https://stats.stackexchange.com/questions/77350/perform-feature-normalization-before-or-within-model-validation


You can follow the below steps to load a json file,

First check whether the file is json or not using the following; https://jsonlint.com/. Once you are confirmed the file is a json, use the below code to read it.

with open("training_dataset.json") as datafile:
    data = json.load(datafile)
dataframe = pd.DataFrame(data)

I hope the above will help you.


Scaling thousands of automated forecasts in R

If you're working with R language, I would suggest first to try use R ecosystem's abilities to parallelize the processing, if possible. For example, take a look at packages, mentioned in this CRAN Task View.

Alternatively, if you're not comfortable or satisfied with the approaches, implemented by the above-referred packages, you can try some other approaches, such as Hadoop or something else. I think that a Hadoop solution would be an overkill for such problem, considering the learning curve, associated with it, as well as the fact that, as far as I understand, Hadoop or other MapReduce frameworks/architectures target long-running processes (an average task is ~ 2 hours, I read somewhere recently). Hope this helps.


I would say that lemmatization is generally the preferred way of reducing related words to a common base.

This Quora question is a good resource on the subject: Is it advisable to choose lemmatization over stemming in NLP? The top answer quotes another good resource that motivates why lemmatization is usually better, Stemming and lemmatization, from Stanford NLP:

Why lemmatization is better

  Stemming usually refers to a crude heuristic process that chops off
  the ends of words in the hope of achieving this goal correctly most of
  the time, and often includes the removal of derivational affixes.

  Lemmatization usually refers to doing things properly with the use of
  a vocabulary and morphological analysis of words, normally aiming to
  remove inflectional endings only and to return the base or dictionary
  form of a word, which is known as the lemma.

But that is just generally, it is not always better. Stemming still has some advantages and it will depend on the use case. Some reasons you would use stemming over lemmatization could be:

Some possible exceptions when stemming can be better

Simplicity
Speed
Memory constraints


Difference between a parameter and a weight?

What you are referring to as parameter is rather called hyper-parameter. I also assume that you are talking about weights as in a neural network, so that would be your model's parameters.

Basically, the difference is that hyper-parameters are chosen by the user prior to learning (and affect the learning phase), while the parameters are computed by the algorithm during the learning process. Thus, the model's parameters depend on training data and the model's hyperparameters.

Note: Things are not always that simple, because optimizing hyper-parameters is often mandatory, and in many cases you could say that they are learnt from the validation phase (model assessment over the validation dataset), so they may also depend on the dataset.


I totally agree with Esmailian.

Naive Bayes is Naive - Assumes Independence.

Steps:

Calculate Independently
Smooth using Laplacian smoothing (to avoid zeroing the whole value)

Additional Tip:

Use log instead of multiplying the probabilities. (this will make sure that your values are not closing to zero, keeping some context.)

Example:

$$p(a) = p(x1)\cdot p(x2)$$

Applying logarithm,

$$\log(p(a)) = \log(p(x_1)) + \log(p(x_2))$$

As you need to finally classify, taking the log won't hurt.


Product Manager from Deepnote here.
I'm sorry it doesn't perform well when the notebook is too big. We're aware of the issue, and it's quite hard to solve, but I'll stress its importance to our engineers.
I'm not sure how Datalore handles large notebooks. When you compare them, they both have some nice features (collaboration, secure integrations, reactivity). Deepnote has a few extra ones (automatic versioning, diffing, comments...), and some great ones coming soon (scheduled reports, published applications). On the other hand, JetBrains is a company with decades of experience in the IDE business.
Some other ones you can check which have an open free tier and real-time collaboration: Nextjournal, CoCalc, Databricks (community edition). There are a few more similar products listed here.


Confused about how to graph my high dimensional dataset with Kmeans

Your k-means should be applied in your high dimensional space. It does not need to be applied in 2D and will give you poorer results if you do this. Once you obtain the cluster label for each instance then you can plot it in 2D. However, we live in a 3D world thus we can only visualize 3D, 2D and 1D spatial dimensions. This means you can at most plot 3 variables in a spatial context, then you can maybe use the color of your points as a fourth dimension. If you really want to stretch it you can use the size of your points for a 5th dimension. But these plots will quickly get very convoluted.

You can use dimensionality reduction techniques to project your high dimensional data onto 2 dimensions. What you are looking to do is perform some projection or feature compression (both of those terms mean the same thing in this context) onto a 2D plane while maintaining relative similarity. Many of these techniques exist each optimizing a different aspect of relative "closeness".

The rest of this answer is taken from here.

The following code will show you 4 different algorithms which exist which can be used to plot high dimensional data in 2D. Although these algorithms are quite powerful you must remember that through any sort of projection a loss of information will result. Thus you will likely have to tune the parameters of these algorithms in order to best suit it for your data. In essence a good projection maintains relative distances between the in-groups and the out-groups.

The Boston dataset has 13 features and a continuous label $Y$ representing a housing price. We have 339 instances.

from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.manifold import TSNE, SpectralEmbedding, Isomap, MDS

bostonboston  ==  load_bostonload_bo ()
X = boston.data
Y = boston.target

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

# Embed the features into 2 features using TSNE# Embed
X_embedded_iso  = Isomap(n_components=2).fit_transform(X)
X_embedded_mds  = MDS(n_components=2, max_iter=100, n_init=1).fit_transform(X)
X_embedded_tsne = TSNE(n_components=2).fit_transform(X)
X_embedded_spec = SpectralEmbedding(n_components=2).fit_transform(X)

print('Description of the dataset: \n')

print('Input shape : ', X_train.shape)
print('Target shape: ', y_train.shape)

plt.plot(Y)
plt.title('Distribution of the prices of the homes in the Boston area')
plt.xlabel('Instance')
plt.ylabel('Price')
plt.show()

print('Embed the features into 2 features using Spectral Embedding: ', X_embedded_spec.shape)
print('Embed the features into 2 features using TSNE: ', X_embedded_tsne.shape)

fig = plt.figure(figsize=(12,5),facecolor='w')
plt.subplot(1, 2, 1)
plt.scatter(X_embedded_iso[:,0], X_embedded_iso[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using Isomap \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.tight_layout()

plt.subplot(1, 2, 2)
plt.scatter(X_embedded_mds[:,0], X_embedded_mds[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using MDS \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.show()
plt.tight_layout()

fig = plt.figure(figsize=(12,5),facecolor='w')
plt.subplot(1, 2, 1)
plt.scatter(X_embedded_spec[:,0], X_embedded_spec[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using Spectral Embedding \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.tight_layout()

plt.subplot(1, 2, 2)
plt.scatter(X_embedded_tsne[:,0], X_embedded_tsne[:,1], c = Y, cmap = 'hot')
plt.title('2D embedding using TSNE \n The color of the points is the price')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.colorbar()
plt.show()
plt.tight_layout()

The target $Y$ looks like:

The projected data using the 4 techniques is shown below. The color of the points represents the housing price.

You can see that these 4 algorithms resulted in vastly different plots, but they all seemed to maintain the similarity between the targets. There are more options than these 4 algorithms of course. Another useful term for these techniques is called manifolds, embeddings, etc.

Check out the sklearn page: http://scikit-learn.org/stable/modules/classes.html#module-sklearn.manifold.


How to find the dependent variables from a dataset?

in Statistics, the independent variables are inputs over which you have control. The dependent variables are the outcome observed by altering the values of the independent variables. Therefore, the answer is "it depends".

Therefore, if you are studying how alterations of left values influence the values of satisfaction level. Then, left values is your independent variables and satisfaction is your independent variables

Instead, if your friend was concluding that variations in left values causes changes in Satisfaction Level, Average Monthly Hours and Promotion Last 5 Years, then, it is a very biased conclusion based on no significant evidence. It would be more appropriate to talk about correlations between variables for which you only know that some behavior appear together (correlated) without identifying any causal effect.


Using NLP to automate the categorization of user description

One way to handle this is to use 'supervised classification'. In this model, you manually classify a subset of the data and use it to train your algorithm. Then, you feed the remaining data into your software to classify it.
This is accomplished with NLTK for Python (nltk.org).
If you are simply looking for strings like "hardware" and "software", this is a simple use case, and you will likely get decent results using a 'feature extractor', which informs your classifier which phrases in the document are relevant.
While it's possible to implement an automated method for finding the keywords, it sounds like you have a list in mind already, so you can skip that step and just use the tags you are aware of. (If your results aren't satisfactory the first time, this is something you might try later on).
That's an overview for getting started. If you are unhappy with the initial results, you can refine your classifier by introducing more complex methods, such as sentence segmentation, identification of dialogue act types, and decision trees. The sky is the limit (or more likely, your time is the limit)!
More info here.


Predicting high frequency sparse time series data in python

Here is what you could try doing:

Get a feel of the shape of your data.
Split your time-series by day. Compare the time-series across multiple days.
Plot time in X-axis and the target variable (for forecast) on the Y-axis. Do the shapes look similar or different ? Can you eyeball N-Distinct shapes that the curve takes ?

Find out the "daily mean" of the target variable to forecast. Plot the mean over the course of the year. Do you see a trend or seasonality ? Are you able to get SARIMA or other related techniques to model the changing mean. If so that by itself may be an achievement. Repeat the same exercise for the "daily variance". Forecasting the daily mean / daily variance "may" be a simpler problem to solve - than forecasting the entire time-series.

Normalize your daily data. Subtract the daily series by the "daily mean" and divide by the "daily standard deviation". Run a clustering algorithm (perhaps K-Means) on your daily timeseries. Use the elbow to identify the best number of clusters.

Plot the centroids of your clusters. If you are lucky - you may be able to see distinct shapes that your forecast curve takes.

Use the Cluster Number to label each daily time series. Then use a classification model to predict the correct cluster. The features for the classification model could be "day of week", "is_holiday", "expected_average_temperature_for_the_day" etc etc.

Check if your classification model does a reasonably good job. If it does you are probably in luck. Your classification model assigns probabilities for each cluster. Combine the cluster centroids weighted by the predicted probabilities - to arrive at a predicted curve.

The predicted curve from the previous step is probably normalized - since the data-prep step (Step 3) normalized the data. You now have the task to rescale back to original data. From Step 2 - if you were able to construct a model that does a reasonable job at predicting the "mean" and "variance" for the target day - Then you could do something as simple as:
Final_Curve = Normalized_Curve * sqrt(Forecasted_Variance) + Forecasted_Mean

Let me know if this worked for you.


How can I find out what class each of the columns in the probabilities output correspond to using Keras for a multi-class classification problem?

The class_indices attribute in Keras’ flow_from_directory(directory) creates a dictionary of the classes and their index in the output array:

classes: optional list of class subdirectories (e.g. [‘dogs’, ‘cats’]). Default: None. If not provided, the list of classes will be
automatically inferred from the subdirectory names/structure under
directory, where each subdirectory will be treated as a different
class (and the order of the classes, which will map to the label
indices, will be alphanumeric). The dictionary containing the mapping
from class names to class indices can be obtained via the attribute
class_indices.

Source
I would be interested to see how people have incorporated this into their workflow/scripts though...


To make our life easier, let's omit optimization methods here (e.g. Learning Rate, Momentum) and focus on vanilla Backpropagation. As you know, backprop consists of computing the partial derivative with respect to each weight in order to penalize each weight for its role in the global output error; such that:

\begin{equation}
    \frac{\partial E}{\partial W_{ij}} = e_{i}y_{j}
\end{equation}

With $E$ the global output error to minimize, $W$ the weight to update, $y$ the output value, and $e$ the local error. The local error $e$ is the unknown we want to compute! :)

Using the chain rule, the gradient can be expressed as:
\begin{equation}
    \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}}
\end{equation}

With $x_{i}$ the input value to neuron $i$.

First, the partial derivative with respect to $W_{ij}$ can be computed as follows:

\begin{equation}
    \frac{\partial x_{i}}{\partial W_{ij}} = y_{j}
\end{equation}

Which is simply equal to the output value $y_{j}$!

Secondly, the partial derivative with respect to $x_{i}$ is:

\begin{equation}
    \frac{\partial y_{i}}{\partial x_{i}} = \frac{\partial \phi(x_i)}{\partial x_{i}}
\end{equation}

Which is simply $\frac{\partial \phi(x_i)}{\partial x_{i}}$, namely the derivative of the activation function (i.e. Sigmoid, Tanh, ReLU...).

Thirdly, the partial derivative with respect to $y_{i}$ can be computed as follows:

\begin{equation}
    \frac{\partial E}{\partial y_{i}} =
    \begin{cases}
        \begin{aligned}
            \frac{\partial}{\partial y_{i}} (T_i - y_i)
        \end{aligned} &
        \text{if $i \in$ output layer,}
        \\
        \begin{aligned}
            \frac{\partial}{\partial y_{i}} \left(\sum\limits^{n}_{j=1}W_{ij}\frac{\partial E}{\partial y_{j}}\right)
        \end{aligned} &
        \text{otherwise;}
    \end{cases}
\end{equation}

With $T$ the target expected output (i.e. the label). The local error $e$ is computed differently depending on the layer position. For the output layer, the error is proportional to the difference between the predicted value (i.e. $y_{i}$) and the expected output (i.e. $T_{i}$). Otherwise, the error in hidden layers is proportional to the weighted sum of errors from connected layers.

Fourthly, combining the partial derivatives allow the computation of the error at a given neuron $i$ such as:

\begin{equation}
    \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} = e_{i} =
    \begin{cases}
        \begin{aligned}
            \frac{\partial \phi(x_i)}{\partial x_{i}} (T_i - y_i)
        \end{aligned} &
        \text{if $i \in$ output layer,}
        \\
        \begin{aligned}
            \frac{\partial \phi(x_i)}{\partial x_{i}} \left(\sum\limits^{n}_{j=1}W_{ij} e_{j}\right)
        \end{aligned} &
        \text{otherwise;}
    \end{cases}
\end{equation}

As you can see here, you need to recursively compute the local error $e$ all the way to the input layer, that's why it's called backprop! ;) I think that what you missed!

If we now simplify:

\begin{equation}
    \frac{\partial E}{\partial W_{ij}} = \frac{\partial E}{\partial y_{i}} \frac{\partial y_{i}}{\partial x_{i}} \frac{\partial x_{i}}{\partial W_{ij}} = e_{i}y_{j}
\end{equation}

We already know $y_{j}$, and we now know how to compute $e_{i}$ all the way to the input layer. So the weight can finally be updated from the gradient as follows:

\begin{equation}
    W_{ij} = W_{ij} - \,e_{i}y_{j}
\end{equation}

Sources:

https://www.youtube.com/watch?v=i94OvYb6noo
Backpropagation, Intuitions: http://cs231n.github.io/optimization-2/
Chapter 9.2 Classification by Backpropagation in Data mining: concepts and techniques: concepts and techniques. Jiawei Han, Micheline Kamber, and Jian Pei. 2011.


Problems finding an LSTM model for classification

Since you are preprocessing the data using Librosa ( maybe extracting MFCCs ), I assume that the preprocessing is correct and needs no intervention.

On viewing the first two layers,

model.add (LSTM (units = 512, return_sequences = True, input_shape = (X_train.shape [1], X_train.shape [2])))
model.add (LSTM (units = 32, return_sequences = False))

You can notice that the difference between the units present in the layers is quite huge ( 512 and 32 ). Hence the 512-dimensional data is packed in a 32-dimensional vector immediately in the next layer.

  You can try reducing this difference. You can use 64 - 32 units in both the layers.

Also, most sources and blogs, say that for recurrent neural networks, RMS optimizer is the best choice. Since we use the hyperbolic tangent activation function, there is a problem with it called exploding gradients. RMSProp optimizer can handle such problems efficiently since the square root of gradients is utilised here.

  Try using the RMSProp optimizer with a low learning rate.

I haven't tried this myself, but you can use recurrent or input dropout in LSTMs. They may help you to overcome the overfitting problem.

  Also, you can try adding some more Dense layers with Dropout layers in between them.

Tip:

LSTMs commonly face the problem of overfitting, so you can implement any of the other methods which are meant to eliminate overfitting. Also, as a common step you can try reducing the batch sizd during training.


Merge two or more ML models with different size of output togetger

Maybe you can use the Concatenate layer
outputs = tf.keras.Concatenate()([model1, model2])
full_model = tf.keras.Model(inputs=inputs, outputs=outputs, name='full_model')

This will simply concatenate the two softmax output into one.


Subset of training set produces good results while full training set produces poor results

For the 200 positive samples, are there many different "types"? In other words, will future positive samples look nothing like the current positive samples? If this is the case, you may try anomaly detection in machine learning.

From a lecture by Andrew Ng in Coursera, if you have small number of positive samples (e.g., 20), large number of negative samples (e.g., 10000), and it's hard for any algorithm to learn from the positive samples, you should use anomaly detection, otherwise, you can use supervised learning.


Anomaly detection in multiple parameters

One way to use both metric1 and metric2 in order to find anomalies in metric3 is to consider residual analysis.

In your case, this would require, creating a predictive model with metric1 and metric2 as the predictors and metric3 as the response variable.

Then, calculate the residuals for metric3 as its predicted value subtracted from its true value. Now, you can report the all members of the lowest decile [or any other percentile] as one kind of an anomaly and all the members of the highest decile [or any other percentile]   as another kind of an anomaly.


How to perform polynomial landmark detection with deep learning

Landmarks are nice when you have a fixed amount for every image but I don't think it is the right approach for your problem. Instead I think you should look into models that segment images by applying a mask. A place to start could be to look into Mask R-CNN.

Here is the paper: Mask R-CNN

Here is how to train a ready made Mask R-CNN implementation with your own dataset using Keras: Tutorial


How to apply a trained Random Forest model to a new data set in R?

You need to setup another data frame that has the unlabeled 2019 observations. Assuming you have multiple predictors, your new data would have the exact same columns as your 2018 data but with no 2019_td column (since you presumably don't know anything about the 2019 season). That is, the players in 2019 that you wish to predict on, along with their predictors, but no target column.

Once you have the data frame, you can use predict(trained_model, newdata) to score your new predictions.

Edit: As an aside, you have data that is influenced by time. What happens if a player gets injured next season? What about trends in the players performance?


Correlation threshold for Neural Network features selection

Given non-linearity of neural networks, I believe correlation analysis isn't a good way to estimate importance of variables. For example, imagine that you have 2 input variables - x1 and x2 - and following conditions hold:

cor(x2, y) = 1 if x1 = 1
cor(x2, y) = 0 otherwise
x1 = 1 in 10% of cases

That is, x2 is a very good predictor for y, but only given that x1 = 1, which is the case only in 10% of data. Taking into account correlations of x1 and x2 separately won't expose this dependency, and you will most likely drop out both variables.

There are other ways to perform feature selection, however. Simplest one is to train your model with all possible sets of variables and check the best subset. This is pretty inefficient with many variables, though, so many ways to improve it exist. For a good introduction in best subset selection see chapter 6.1 of Introduction to Statistical Learning.


Yes, multiple different ways.

First, we could consider (id-artist,id-track) items as the elements of our set, and compute the Jaccard similarity by comparing those sets. Note that if the artist's id gives us no additional information beyond the track id, this will give the same result, whereas it will give different results if a particular track id could be associated with multiple artists.

Second, we could compute the Jaccard similarity on tracks, and then the Jaccard similarity on artists, and then add the two (probably with some constant coefficient scaling the two). This way two users who listen to the same artists, but none of the same songs by those artists, will be rated as more similar than users who listen to different songs by different artists.

What coefficient makes sense? Well, you could start with 0.5 for each (i.e. just .5*similarity_artist+.5*similarity_track), see if that's reasonable, and adjust if it's not.


Machine Learning methods for finding outliers

You should plot the data with matplotlib and seaborn to get a visual view of the data.  That is where I would start.  Especially with large datasets, this is a quick way to visually see any outliers.


Attention to multiple areas of same sentence

For the sentences you provided the nltk sentence tokeniser would work just fine.

from nltk.tokenize import sent_tokenize

sentences = ['Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too','Can company1 buy company2? What will be their total turnover then?','Coronavirus was originated in china. After that it is spreading all over the world. To prevent it everyone has to take care of cleanliness and prefer vegetarians.']

for sent in sentences:
    print(sent_tokenize(sent))

out:
['Datascience exchange is a wonderful platform to get answers to datascience related queries and it helps to learn various concepts too']
['Can company1 buy company2?', 'What will be their total turnover then?']
['Coronavirus was originated in china.', 'After that it is spreading all over the world.', 'To prevent it everyone has to take care of cleanliness and prefer vegetarians.']

It is true that these rule based tokeniser do suffer from several issues (see: https://github.com/nltk/nltk/issues/494 ). If you are specifically working on a sequential model for sentence slitting then an LSTM with attention mechanism would surely be a good choice, you can find many repositories on GitHub which implement the code of the original paper 'attention is all you need' (among others: https://github.com/kaushalshetty/Positional-Encoding )


Looks 'correct' to me, in the sense that it satisfies the requirements for being an MDP. Whether it models the underlying real-world problem correctly cannot be validated with the information given here.


Should I relabel this data or remove the potentially leaky feature?

Consider your problem as a binary classification. We have two kinds of prediction: raise an alarm (state is going to change), do not raise an alarm (state is going to stay the same).

Actual / Predicted   Alarm goes off      Alarm did not go off
State changed        True positive       False positive
State stayed         False negative      True negative

Note the true negatives are useless - we don't care if the state stays in the same spot (below/above threshold). Therefore, the accuracy is NOT important for this system. Do not try to minimize the wrong metric. This extends to "do not use ROC/AUC" (in your problem). You can use a PR curve, but be careful with those (no interpolation, no AUC PR, these are wrong/useless). Basic metrics that you could use are F-score, precision and recall, or a re-weighting of those (if the false alarm is less important than a missed alarm, for example).

We also would like to deal with a huge "lag", trying to predict many time periods before (predicting just before the change is less useful). For this reason, I'd suggest investigating models other than MLP, notably those based on recurrent neural networks, such as LSTM. Also consider using time series prediction instead of classification - the literature on the subject is extensive and matches your problem really well.


sagemath: compared to r.quantile, what is a faster way to find boundaries for a boxplot?

The following (crude) code is at least 50 times faster than r.quantile:

def findFences (orderedList, outlierC = 1.5, farOutlierC = 3.0):
   """findFences: an ordered list of ints or floats -> tuple of 9 floats
   (aMin, outerLoFence, innerLoFence, Q1, Q2, Q3, innerHiFence, outerHiFence, aMax)
   keys: float for outlier constant [outlierC] and far outlier constant [farOutlierC]"""

   lenMod4, half, quarter = len(orderedList) % 4, int(len(orderedList)/2), int(len(orderedList)/4)

   aMin, aMax = orderedList[0], orderedList[-1]

   # find quartiles
   if not lenMod4: Q1, Q2, Q3 = (orderedList[half-quarter] + orderedList[half-quarter-1])/2.0, (orderedList[half] + orderedList[half-1])/2.0, (orderedList[half+quarter] + orderedList[half+quarter-1])/2.0
   elif lenMod4 == 1: Q1, Q2, Q3 = float(orderedList[half-quarter]), float(orderedList[half]), float(orderedList[half+quarter])
   elif lenMod4 == 2: Q1, Q2, Q3 = float(orderedList[half-quarter-1]), (orderedList[half] + orderedList[half-1])/2.0, float(orderedList[half+quarter])
   else: Q1, Q2, Q3 = (orderedList[half-quarter] + orderedList[half-quarter-1])/2.0, float(orderedList[half]), (orderedList[half+quarter] + orderedList[half+quarter+1])/2.0

   IQR = Q3 - Q1
   outDist = IQR * outlierC
   farOutDist = IQR * farOutlierC
   innerLoFence, innerHiFence = Q1 - outDist, Q3 + outDist
   outerLoFence, outerHiFence = Q1 - farOutDist, Q3 + farOutDist

   return aMin, outerLoFence, innerLoFence, Q1, Q2, Q3, innerHiFence, outerHiFence, aMax


This article on Medium introduces GPT-3 makes some comparisons with BERT.
Specifically, section 4 examines how GPT-3 and BERT differ and mentions that: "On the Architecture dimension, BERT still holds the edge. It’ s trained-on challenges which are better able to capture the latent relationship between text in different problem contexts."
Also, in section 6 from the article, author lists areas where GPT-3 struggles. It may be that BERT and other bi-directional encoder/transformers may do better, although I have no data/references to support this yet.


Machine learning goal: given a population of 100,000 students, predict a group of 3,000, and minimize the median grade of that group

Try writing a custom loss function for a regression model!
Keras' neural networks support this, for example. See https://stackoverflow.com/q/43818584/745868
(But many other libraries give support for this as well)
The only special thing about your custom loss function is that it doesn't add up the error of a datapoint if min(pred_y, actual_y) >= THRESHOLD


Difference between mathematical and Tensorflow implementation of Softmax Crossentropy with logit

As I understand it, the softmax function for $z_i$ is given by $a_i$. Then just taking the loss you've defined you get back exactly the formula that is implemented. The way it is written down however is, as you mentioned, to avoid underflow/overflow.

For instance, suppose you want to compute the following:

$A=\log(\sum_{i=1}^{4}\exp(z_i))$, with $z_i=(-1000.5,-2000.5,-3000.5,-4000.5)$

Clearly, if you just type in the formula directly, you will get an underflow error. Instead if you isolate the main contribution in the exponential by taking the $\max(z_i)$, the same formula can be written as:

$A=\max_i(z_i)+\log(\sum_{i=1}^{4}\exp(z_i-\max_i(z_i)))$

The difference now is that the expression is "numerically stable" and we see that $A\approx -1000.5$.

Thus, let's make the softmax numerically stable:
\begin{align}
\log(a_i)&=z_i-\log(\sum_j e^{z_j})\\
&=z_i-\max_j(z_j)-\log(\sum_je^{z_j-\max_j(z_j)})
\end{align}
which is the expression that is implemented for the loss (just multiply by $y_i$ and sum over $i$).


Are there any python based data visualization toolkits?

Bokeh is an excellent data visualization library for python.
NodeBox is another that comes to mind.


If your network outputs a vector  $x \in \{0,1\}^N$ with $N=100$ and $\sum_{i = 1}^{N} x_{i}= 1$, you could consider weights $\mathbf{W}=(w_{i}) \in \mathbb{R}^{N}$ with $w_{i} :=i$ for $i \in \{1,\ldots,100\}$.
Then, for prediction vector $x$ and ground-truth vector $y$, you could use the loss function $L(x,y):= || \mathbf{W}^{T}x - \mathbf{W}^{T}y || = || \mathbf{W}^{T}(x-y)||$


The word "prediction" does not belong to any specific type of machine learning. There is nothing wrong with "predicting" new data to the cluster it belongs to; (e.g. there are many applications that place new customers into pre-discovered market segments). A conditional probability, like that used in classification, is not "stronger" than an unsupervised approach, as it rests its assumption on properly labelled classes; something that is not guaranteed.

This is why there are packages that provide a predict function to clustering algorithms. Here is an example using the flexclust package with the kcaa function. That being said, the prediction step is usually handled by a supervised classifier, so the approach would be to sit a classifier on top of your learned clusters (treating cluster assignments as "labels").

You just have to reason about your weaknesses. As stated above, the weakness in classification is the assumption that labelled data is tagged correctly, whereas the weakness in clustering is that your discovered clusters are assumed to be valid. Unsupervised approached cannot be validated the same way it is done with classification. Clustering requires a variety of cluster validity techniques along with domain experience (e.g. show campaign managers your market segments to validate customer types).

Ultimately, you are just matching an incoming vector (new data) to the cluster most similar. For example, in k-means this could be accomplished by finding the smallest distance between the incoming vector and all the centroids of your clusters. This kind of pattern matching depends on the data you are using.

This works best for clustering techniques that have well-defined cluster objects with exemplars in the center, like k-means. Using hierarchical techniques means you would need to cut the tree to obtain flat clusters, then use the "label" assignment to run a classifier on top. This comes baked with a lot of assumptions, so you need to make sure you understand your data very well, and validate any clusters with non-technical users that have deep domain experience.

POSSIBLE APPROACH
If you're bent on using hierarchical clustering, then here is the general approach. Note I am not suggesting this is the best way. Every approach comes baked with a number of assumptions. You will need to work to understand your data, attempt many models, validate with stakeholders, etc.

Readers can use the tutorial by Jörn Hees to get started in hierarchical clustering if needed:

Create some example data:

from matplotlib import pyplot as plt
from scipy.cluster.hierarchy import dendrogram, linkage
import numpy as np

np.random.seed(42)
a = np.random.multivariate_normal([10, 0], [[3, 1], [1, 4]], size=[100,])
b = np.random.multivariate_normal([0, 20], [[3, 1], [1, 4]], size=[50,])
X = np.concatenate((a, b),)

Confirm clusters exist in synthetic data:

plt.scatter(X[:,0], X[:,1])
plt.show()

Generate the linkage matrix using the Ward variance minimization algorithm:
(This assumes your data should be be clustered to minimize the overall intra-cluster variance in euclidean space. If not, try Manhattan, cosine or hamming. You can also try different linking options).

Z = linkage(X, 'ward')

Check the Cophenetic Correlation Coefficient to assess quality of clusters:

from scipy.cluster.hierarchy import cophenet
from scipy.spatial.distance import pdist

c, coph_dists = cophenet(Z, pdist(X))

0.98001483875742679

Calculate full dendrogram:

plt.figure(figsize=(25, 10))
plt.title('Hierarchical Clustering Dendrogram')
plt.xlabel('sample index')
plt.ylabel('distance')
dendrogram(
    Z,
    leaf_rotation=90.,  # rotates the x axis labels
    leaf_font_size=8.,  # font size for the x axis labels
)
plt.show()

Determine the number of clusters (e.g. can be done manually by looking for any large jumps in the dendrogram...see Jörn's blog for plotting function):

Retrieve clusters: (using our max distance determined from reading dendrogram)

from scipy.cluster.hierarchy import fcluster

max_d = 50
clusters = fcluster(Z, max_d, criterion='distance')

Map cluster assignments back to original frame:

import pandas as pd

def add_clusters_to_frame(or_data, clusters):
    or_frame = pd.DataFrame(data=or_data)
    or_frame_labelled = pd.concat([or_frame, pd.DataFrame(clusters)], axis=1)
    return(or_frame_labelled)

df = add_clusters_to_frame(X, clusters)
df.columns = ['A', 'B', 'cluster']

df.head()

Build a classifier using this "labelled" data:

Here, I'll just use the original data and the assigned clusters along with a knn classifier:

np.random.seed(42)
indices = np.random.permutation(len(X))
X_train = X[indices[:-10]]
y_train = clusters[indices[:-10]]
X_test  = X[indices[-10:]]
y_test  = clusters[indices[-10:]]

# Create and fit a nearest-neighbor classifier
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier()
knn.fit(X_train, y_train)

res = knn.predict(X_test)
print(res)
print(y_test)

predicted labels: [2 2 1 1 2 2 1 2 2 1]

test labels: [2 2 1 1 2 2 1 2 2 1]

As with any classifier, your incoming data needs to be in the same representation as your training data. As new data arrives you run it against the predict function provided by your classifier (here we use sci-kit learn's knn.predict). This effectively assign new data to the cluster it belongs.

Ongoing cluster validation would be required in the model monitoring step of the machine learning workflow. New data can change the distribution and results of your approach. BUT, this isn't unique to unsupervised as all machine learning approaches will suffer from this (all models eventually go stale). As argued by Jörn in the reference above, manual inspection typically trumps automated approaches, so regular visual/manual inspection of the flat clusters is recommended.


Training And Testing Error Curves caret package in r

Here we are creating data to for a given model.

The main advantage with this code snippet is you are able to see how the model performers over various sizes of the training set. The learing_curve_data function can be found in the [6.0-7.1 version of caret][1]. Of course, learning curves are useful in machine learning for several reason which include comparison of algorithms, choosing model parameters, optimization, right data size to use for training.

set.seed(1412)
class_dat <- twoClassSim(1000)

set.seed(29510)
lda_data <- learing_curve_dat(dat = class_dat,
                               outcome = "Class",
                               test_prop = 1/4,
                               ## `train` arguments:
                               method = "lda",
                               metric = "ROC",
                               trControl = trainControl(classProbs = TRUE,
                                                        summaryFunction = twoClassSummary))

ggplot(lda_data, aes(x = Training_Size, y = ROC, color = Data)) +
geom_smooth(method = loess, span = .8) +
theme_bw()


Tensorflow.Keras: How to get gradient for an output class w.r.t a given input?

You can get that from the weight updates, not sure if it is the best approach
-Save the model
-Save the weights of the first layer
-Load the model and Compile the model with SGD w/o momentum
-Set all the weights = that of the previous model
-Train with the input and output i.e. the Array for epoch=1 and batch_size=1
-Get the weights again
-Calculate the respective gradient using two weights of 1st layer
-If Gradient is very small, it might become zero due to matching digits of both the weights
w0 = model.get_weights().copy()
w0_0 = w0[0]

optimizer = tf.keras.optimizers.SGD( learning_rate=0.01, momentum=0.0, nesterov=False, name="SGD")
model.compile(optimizer=optimizer, loss='mse')
model.fit(input, output, epochs=1, verbose=1, batch_size=1)

w0_new = model.get_weights().copy()
w0_0_new = w0_new[0]

'''
w_new = w0 - learning_rate * g
g = (w0 - w_new)/learning_rate
'''
grad = (w0_0 - w0_0_new)/0.01


scale_pos_weight is used for binary classification as you stated. It is a more generalized solution to handle imbalanced classes. A good approach when assigning a value to scale_pos_weight is:

sum(negative instances) / sum(positive instances)

For your specific case, there is another option in order to weight individual data points and take their weights into account while working with the booster, and let the optimization happen regarding their weights so that each point is represented equally. You just need to simply use:

xgboost.DMatrix(..., weight = *weight array for individual weights*)

You can define the weights as you like and by doing so, you can even handle imbalances within classes as well as imbalances across different classes.


Standardising may not be the best option.

Because they will still not be bounded (like when normalised) between -1 and 1 but be distribution dependent. What I mean is if they are outliers their standard deviation will be big for these values. In any case its not that you should rescale the values to combat these outliers. Outliers are not in the first place problematic because they are big in values, but mostly because these values lie in the cluster thats not possible/representative of future data and you will learn on it.

One solution for without dropping them is for example binning, where you say all outliers that have standard deviation bigger than two should have value "X" where X is some 99,5% quantile for example.


What is coadaptation of neurons in neural networks?

Co-adaptions in simple English term would mean co-operation. If you think nodes of a NN as workers it would mean missing even a few workers would result in failure of the NN to do something substantial. This happens mainly due to few nodes of NN outputting values which get cancelled by other nodes (so we can remove those nodes altogether -  essentially redundant units), thus resulting in inefficient use of the NN. Thus instead of say 2 nodes, 8 nodes are performing the same job with 6 nodes cancelling each other out. Here is the best illustration of the same.

In case of XOR approximation you need minimum (I'm not sure) 2 nodes to approximate the function correctly, but you can use 100 nodes and most of them will be redundant.

Also this will clearly cause over-fitting as the nodes instead of generalising on the training data are micro-managing the training data.

Now we do not want our NN's to behave in such a manner, we  want the other nodes to take over jobs of missing nodes (if any are missing) just like our brain takes over function when one centre (Say speech, hearing is damaged) and the state of art mechanism currently is dropout in layers.


The input dimensionality is the embedding size, which is the same as the model dimensionality, as explained in section 3.4 of the article:

3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{model}$.

Therefore, the input dimensionality and the model dimensionality are the same, which makes them suitable for the residual connection.


Not sure how to visual customer retention vs product line count

There are probably a good few options, as usual I would start with something simple (possibly similar to what you have in mind).

Simple year1 vs year2 or year2 vs year3 global comparison: proportion of customer who go from number X in year N to number Y in year N+1. A heatmap would probably work well with this kind of data, and if you want some fancy visualization to impress your boss you could try a chord diagram or a Sankey diagram. Of course you can vary the input with average of the past 2 years, etc.
Since you have data by territory and it appears to show different patterns, it might be interesting to look at something by territory. I'm not sure which value should be used as a "summary" of the trend for a territory, I can think of a few options:

simple difference of the mean number of products between year N and year N+1 (i.e. shows positive or negative trend)
some measure of how different the distributions are, i.e. how much variation from year N to year N+1
Simple Pearson correlation over all the customers between number of products in year N vs year N+1.


How to replace a part string value of a column using another column

data['Product Name'] = data['Product Name'].str.replace('\d+','')

This should get rid of the number if that's what you are looking for. I am not sure what you mean by 'chomped.'


Sample Importance (Training Weights) in Keras

I'm not sure I completely understand what you want but it looks like you're trying to find an implementation for sample_weight. Well, I had tried something similar. Before I go into it, I want to mention that my error actually went up - I haven't had the time to go into why that happened.

For simplicity, lets say you know the weights you need for each class and can pass it as a dictionary. In my example I have my y_train as a one hot encoded vector. I'm using that fact to reverse engineer which class each row is pointing to and adding a weight for it. You essentially need to pass an array of weights mapping to each label (so the same length as your training data) when you fit the model.

def generate_sample_weights(training_data, class_weight_dictionary):
    sample_weights = [class_weight_dictionary[np.where(one_hot_row==1)[0][0]] for one_hot_row in training_data]
    return np.asarray(sample_weights)

And this is passed when you call fit on the model.

model.fit(x=X_train,
    y=y_train,
    batch_size = 64,
    validation_data=(X_val, y_val),
    shuffle=True,
    epochs=20,
    sample_weight = generate_sample_weights(y_train, class_weights_dict)
)


Can't load custom Keras metrics using mlflow.pyfunc

Long story short is to use cloudpickle instead of joblib or pickle to dump thing to disk, and this all works much more cleanly.


Error 'Expected 2D array, got 1D array instead:'

This is a bit tricky. Using pandas data, sklearn only accepts input variables (features) with type pandas.Dataframe. In your code variable sex_train in pandas.Series type. Try the following code :

sex_train = data['Sex'].map({'male':0,'female':1}).to_frame()


Yes, it appears to be so. Look here for more details.
Here is a small, but important, extract:

"Just like other Amazon SageMaker algorithms, it [deepAR] relies on Amazon
Simple Storage Service (Amazon S3) to store the training data and the
resulting model."

Hope this helps.