In general, you need to wrap the feature selection filter method within a cross validation scheme. Here is an example of such an algorithm, assuming you are also tuning hyperparameters.

Split the data into training and testing.
Split the training set only into k folds.
For each iteration of hyperparameter tuning, use cross validation, i.e. do:

  For each of the k folds, for iterations i = 1 to k, do:

  1) Let the validation set be fold i, and the "inner training" set be all folds not equal to fold i.

  2) Using your filtering method, select some subset of your features using some cutoff p on the inner training set. For example, I might filter out variables that have absolute correlation with the target variable y less than p = 0.3.

  3) Fit a model.

  4) Predict the validation set = fold i. Score the model, i.e. evaluate your function you are trying to optimizer like log loss, mean squared/absolute error, Brier, etc. Save the score.

  end

After you are done with your cross validation for one set of hyperparameters, take the mean score over all of the k folds. If the score is better than a previous set of hyperparameters found in a prior iteration, save these hyperparameters.

Note: you can choose to have p, your filter cutoff, also be a hyperparameter here. For simplicity I kept it constant but in general you will probably do better if you tune the filtering cutoff as well.

end

Using the same (or best, if you let it be a hyperparameter and optimized over it in step 3) filter cutoff p, and the best hyperparameters that you found in the cross validation process in step 3, use the training set created in step 1, and fit your "final" model. That is, filter your training set with the cutoff value p using the exact same filtering method you did in step 3. Then, fit a model to this data set using the optimal hyper parameters, also found in step 3.
Predict the test set using your fitted model in step 4. Score the model.

Finally, repeat the entire process again but this time, don't do any filter selection method (omit the filtering process). Use the exact same seeds in your cross validation and at the initial split in step 1 to make sure you are splitting your data the same way from when you used filtering. Now, compare test set scores and see for yourself if there is any benefit to using your filtering method.

In general, I am not a fan of filter based methods because they all introduce some arbitrary cutoff = p and also are all univariate (they don't take into consideration other variables in the model, just in isolation). That being said, they are fast and they might work for your problem.

EDIT: Also, depending on how much data you have and how variable your validation scores are, multiple repeats of the entire algorithm above (steps 1 through 5) might be necessary in order to obtain valid comparisons between using filtering and not using filtering. You would repeat the algorithm above say, N times, once  with filtering and once without for a total of 2N times. You would then calculate an average of averages (find the average score over N repeats using filtering, and find the average score over the same N repeats with no filtering). Then, you could compare scores, form confidence intervals, etc. for a more trustworthy (but also more time consuming) comparison.

Hope this helps.


I agree with Dawny33, choosing learning rate only scales w.

While training of Perceptron we are trying to determine minima and choosing of learning rate helps us determine how fast we can reach that minima. If we choose larger value of learning rate then we might overshoot that minima and smaller values of learning rate might take long time for convergence.

It is okay in case of Perceptron to neglect learning rate because Perceptron algorithm guarantees to find a solution (if one exists) in an upperbound number of steps, in other implementations it is not the case so learning rate becomes a necessity in them.

It might be useful in Perceptron algorithm to have learning rate but it's not a necessity.


Measuring the uncertainty of predictions

The model you will decide how best to get an uncertainty. If you used Bayesian optimisation (that's a great package for it in Python), for example, you get a covariance matrix along with your expected values, and so inherently get an uncertainty measure. In this case, you can make predictions as to the underlying function of your data, and the (co-)variance will provide levels of uncertainty, as shown by the width of the green bands around the line below:

So the red points show where we have some sample data... notice that we have none e.g. at X = 4 and X = -1, which is why we have high undcertainty; the 95% confidence interval is very large.

If you use a standard deep neural network tpo perform classification, for example, there is no inherent measure of uncertainty. All you really have is your test accuracy, to let you know how well the model performs on hold-out data.  I cannot remember where it is explained, but I believe it is not actually feasible to interpret the class prediction values in terms of uncertainty.

For example, if you are predicting cat or dog for an image, and the two classes receive (normalised) logit values [0.51, 0.49] respectively, you cannot assume this means very low certainty.


A common step in "feature engineering" is parsing these out into multiple values that might give you additional information during ML. For example:

Hour of the day
Minute of the hour
Hours after sunrise
Hours before sunset

If you also had day, that would open a lot more to detect seasonality like:

Month of the year
Month of the quarter
Day of the week
Holiday Y/N


Applying bayesian methods to a simple neural network

Bayesian models specify priors to inform and constrain the models and get uncertainty estimation in form of a posterior distribution. Non-Bayesian Deep Learning computes a scalar value for weights and biases at each layer. Bayesian Deep Learning calculates a posterior distribution of weights and biases at each layer which better estimates uncertainty but increases computational cost.

Edward is a Python package for Bayesian inference, including Deep Learning.


OOB decision function doesn't match prediction in scikit-learn RandomForest

From your example I understand that $y_{pred}$ are the predictions of the random forest on the training set. If this is the case, $y_{pred}$ would give you training performance whereas the oob values will give you validation performance. Because you are overfitting your training predictions are close to the true values whereas your validation/oob predictions are not.

When bagging, on average each bag contains 2/3 of the samples leaving the remaining 1/3 in the oob. When you use the random forest to predict on the training set, for each sample there will be about 17 000 (~2/3*25000) trees which have seen that datapoint already since it's in their bag and results will not be reliable in terms of generalisation. In other words, when you predict on the training set, 2/3 of the trees will give you training predictions and 1/3 (oob) will give you validation predictions so overall it's like getting training performance.

Answering your last four questions:

Random forest in sklearn gets the probability of each class for each tree and then averages them across all bags to get the final probabilities. This is usually called 'soft voting', 'hard voting' being the method you describe.
See 1.11.2.1. Random Forests
I haven't seen it stated clearly in the documentation but I would expect that the same method that is used to make general predictions ('soft voting') will be used to get the oob values.
Yes, cross validation and oob scores should be rather similar since both use data that the classifier hasn't seen yet to make predictions.
Most sklearn classifiers have a hyperparameter called class_weight which you can use when you have imbalanced data but by default in random forest each sample gets equal weight.


In general, when does TF-IDF reduce accuracy?

The IDF part of TF-IDF gives less weight to a word if it occurs in a large fraction of the documents in your corpus.  However, this doesn't necessarily mean that the word is unimportant for distinguishing your two classes.  A word which is common in your corpus, but which also occurs substantially more often in one class than the other, could very well be quite valuable in distinguishing the classes.  This can especially be true if your set is not balanced between the two classes.

For example:
Suppose that 95% of negative reviews contained the term "boring", while only 5% of the positive reviews contained "boring".  If this occurred in a balanced set of negative and positive reviews, that means that half the total reviews would contain "boring", and this might be one of the terms whose weight is more strongly suppressed by IDF, despite the fact that it's obviously a valuable term for distinguishing the classes.  If instead 90% of the reviews are negative (with "boring" again occurring in 95% of negative reviews and 5% of positive ones), then a full 86% of your documents contained the term, and its weight might be highly suppressed by IDF despite its obvious importance.

In essence, the problem is that TF-IDF is agnostic to the class labels in your training set.  One alternative to TF-IDF for text classification that doesn't have this flaw is Bi-Normal Separation (BNS), see for example this paper:

Forman, G.  BNS Feature Scaling: An Improved Representation over TF-IDF for SVM Text Classification.  Hewlett-Packard Labs Tech Report HPL-2007-32R1, 2007.(link to PDF)


What type of NN should be used for a turn-based game (with direct access to data)?

Your question is quite broad so it is not easy to give a direct answer.  However, a simple approach and a good introduction to machine learning would be to use a Multilayer Perceptron (MLP).  This is one of the most basic and fundamental neural network architectures.

You will also want to read about Backpropagation, which is the standard method of training such algorithms.

Here are some code examples/libraries (in no particular order) that can get you started:

Keras - (Python)
Theano - (Python)
DeepLearning4J - (Java)
Caffe  - (C++/Python/Matlab)
TensorFlow - (C++/Python)
Weka - (Java)


If you want to start with research, as you say, I would make sure not to skip the theory and underlying mathematics of machine learning. Knowing what model and hyperparameters to choose for a certain problem and being able to implement this quickly is very useful in corporate settings, but research mostly requires a deep mathematical understanding of what is going on.
In my opinion the book 'deep learning' by Goodfellow and Bengio has a good structure that allows you to get familiar with the mathematical foundations of deep learning. When you understand this, you will be able to read and understand state-of-the art papers if you want and do your own research. You can find the book for free online I think.


Algorithm/Analysis that utilises incremental information?

It looks like you are looking for time series based Machine Learning (or Sequential Machine Learning as Emre pointed out).

This means that you have an entire dataset available to you beforehand. You train on this dataset and then use it to predict next day's weather given the weather of the last n days. Please note that you do not learn incrementally here, i.e. you do not learn from the new data that is rolling in every day. The learning is limited to the dataset that you had beforehand.

Widely used algorithms for this kind of learning are Conditional Random Fields(CRF), Hidden Markov Models etc. Statistical methods like Autoregressive Models can also be used sometimes. A good starting point would be CRFs.

If on the other hand you want to learn incrementally, read about online learning.


How to do postal addresses fuzzy matching?

As you are using R you might want to look into the stringdist package and the Jaro-Winkler distance metric that can be used in the calculations. This was developed at the U.S. Census Bureau for linking .

See for more information on the Jaro and Jaro-Winkler distance in this journal.

For a comparison of different matching techniques, read this paper


How to measure the performance of an imputation technique

I think there is no answer to your question since there is no absolute universal "good". Everything depends on the question you ask and the tools you use. This is why there are a lot of imputation techniques. There is no replacement for a missing value. However, in the constrains given by your question and used tools, you can think of imputation which does not alter your answer or at least measure some effects of the missing values. I will give some simple examples.

One widespread technique is to replace missing values with a marginal central estimator for each variable. This might work for a classifier like decision tree, if there are no missing patterns in your data. However using this imputation would alter your confidence intervals if you study linear relations between variables using regression. This is simply because it will alter sample variance.

Another much sophisticated imputation method is to use EM algorithm to fit the maximum likelihood estimator of the variance-covariance matrix. This estimator is unbiased and using this variance covariance matrix you can recover the linear model in an unbiased way. You then can go to analyze linear relations. But this works only for linear and log linear models and requires a lot of data and also requires to have missing data at random, which is not always the case.

Another one is multiple imputation. What you actually do is to draw data at random for missing values according with a supposed distribution. You do that multiple times, let's say at least 30 times. You analyze each data set with traditional tools and later you aggregate those results into a single set of results. This works well mostly when data is missing at random, but is tedious, sometimes inconsistent and duo to randomness it can produce different results. Ultimately, sometimes is very hard to find proper ways to aggregate the results, it depends on the analysis you use.


How the original data can be written in the space defined by these M principal components?

Assuming your basis vectors are sorted by decreasing eigenvalues, a sample $x_i$ can be transformed into the new space using dot products (projections), i.e. $[u_1^Tx_i,...,u_M^Tx_i]$. Note that, you first de-mean your data.


Tensorflow regression predicting 1 for all inputs

Ok, I've managed to get a decent improvement. Currently producing an MSE of 20 for training and 30 for test. This is after converting all layers to relu and increasing their size.

The problem appeared the activation function on the output. I was under the impression that a sigmoid output layer was common practice.

I was also normalizing before the test/train split and leaking information into the test set! (though this likely wasn't the cause of my problem)

Here is my 'improved' code:

import pandas as pd
import numpy as np
from scipy import stats
from sklearn.cross_validation import train_test_split
from sklearn import preprocessing

raw = pd.read_csv('parkinsons_updrs.data')
print("CSV loaded successfully")

events = raw.shape[0]
#drop events where any feature has a zscore > 3 for that feature
raw = raw[(np.abs(stats.zscore(raw)) &lt 3).all(axis=1)]
print("Removed " + str(events-raw.shape[0]) + " outlier events")

features = raw.drop(['subject#','test_time','motor_UPDRS','total_UPDRS'], axis = 1)
targets = raw['total_UPDRS'].get_values()
print("pre-processing complete")

#Split data into test/train
data_train, data_test, targets_train, targets_test = train_test_split(features,targets,test_size = 0.33,random_state=123)
print("Train/Test split complete")

#Normalize data to mean 0 and std 1
#We normalize after the split to avoid leaking information to the training set
data_train = (data_train-data_train.mean())/data_train.std()
data_test = (data_test-data_test.mean())/data_test.std()

targets_train = targets_train.reshape(targets_train.shape[0],-1)
targets_test = targets_test.reshape(targets_test.shape[0],-1)

#Network Structure
X = tf.placeholder(tf.float32, [None,18])
y = tf.placeholder(tf.float32, [None,1])
prob = tf.placeholder_with_default(1.0, shape=())

w1 = tf.Variable(tf.random_normal([18,72]))
w2 = tf.Variable(tf.random_normal([72,36]))
w3 = tf.Variable(tf.random_normal([36,1]))

b1 = tf.Variable(tf.random_normal([72]))
b2 = tf.Variable(tf.random_normal([36]))
b3 = tf.Variable(tf.random_normal([1]))

layer1 = tf.nn.dropout(tf.nn.relu(tf.add(tf.matmul(X,w1),b1)),prob)
layer2 = tf.nn.dropout(tf.nn.relu(tf.add(tf.matmul(layer1,w2),b2)),prob)
out = tf.nn.relu(tf.add(tf.matmul(layer2,w3),b3))

loss = tf.losses.mean_squared_error(y, out)

train_step = tf.train.AdamOptimizer(0.005).minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

print("Training...")

minibatch = True
batchsize = 512

for i in range(10000):
    if minibatch:
        for x in range(0,targets_train.shape[0],batchsize):
            sess.run(train_step, feed_dict={X:data_train[x:x+batchsize],y:targets_train[x:x+batchsize],prob:0.5})
    else:
        sess.run(train_step, feed_dict={X:data_train,y:targets_train,prob:0.5})
    if i%25==0:
        acc = sess.run(loss, feed_dict={X: data_train, y: targets_train})
        print ("Training accuracy: %.5f" % (acc))

test_acc = sess.run(loss, feed_dict={X: data_test, y: targets_test})
print ("Test accuracy: %.5f" % (test_acc))

output = sess.run(out,feed_dict={X: data_test, y: targets_test})

error = abs(output-targets_test)

print(max(error))
print(sum(error)/len(error))
print(min(error))

I'd be very interested to know why this happened. Both tanh and sigmoid get 'stuck' in the 900s whereas relu sorts itself out and carries on.


Accuracy reduces drastically when using TruncatedSVD with hashingvector

Some possible directions:

when you're unsure how your BoW model works check different n-gram ranges (did you even check how it works with only unigrams?)
TruncatedSVD by default only iterates 5 times (n_iter), so you could increase that
It's not at all surprising to get such drastic change of classification quality for such low dimensionality. Some questions you could ask are why 100 dimensions? Does TSVD work 1000 dimensions or something like that? What is the reconstruction error (how does TSVD's explained_variance_ratio_ look like)?
HashingVectorizer also has parameter that can control number of features, namely n_features.


Are there any rules regarding correlation between the variables used in a CNN?

In theory, convolutional and pooling layers should be able to account for this kind of collinearity. Convolutional filters will learn to extract the relevant information from your pixel data, and pooling layers significantly reduce their dimensionality, reducing multicollinearity as a side effect.

However, if you think multicollinearity is still a problem, you can apply some dimensionality reduction techniques. They work on image data as well!


How to remove irrelevant text data from a large dataset

You can use BERT to create vectors that will capture the context of the whole tweet. Once, you do that, try clustering (K-Means or GMM). You can then look at the clusters found and separate out this unwanted data.


What do all the column names mean in the SentimentAnalysis package's analyzeSentiment() function in R?

The different dictionaries are in the documentation for the function:

For example DictionaryHE is the "Dictionary with opinionated words from Henry’s Financial dictionary" (see DictionaryHE).

Example:

sentiment <- analyzeSentiment(document)   # Document is a Corpus
sentimentData <- sentiment$SentimentHE   # Generates analysis based on Henry's dictionary
plotSentiment(sentimentData) # Quick line plot of sentiment


Multilayer Perceptron: What is the value used to update the weights in the hidden layers?

A Multilayer Perceptron changes your weights by an algorithm called "backpropagation". This algorithm uses gradient descent combined with a learning rate to change every weight in your MLP.

Basically the backpropagation functions by chaining together all functions that are called when calculating the output of one particular node (so chaining together all possible ways). Now using the chain rule a gradient is calculated which directs into the direction of minimal error and changes all weights accordingly.


No, you should go ahead and learn the maths on your own. You will "only" need to learn calculus, statistics, and linear algebra (like the rest of machine learning). The theory of neural networks is pretty primitive at this point -- it more of an art than a science -- so I think you can understand it if you try. Ipso facto, there are a lot of tricks that you need practical experience to learn. There are lot of complicated extensions, but you can worry about them once you get that far.

Once you can understand the Coursera classes on ML and neural networks (Hinton's), I suggest getting some practice. You might like this introduction.


"residual error" of LSTM during backprop vs usual "error"

Residual errors are the errors that remain after a model has tried fitting to some data. It is the error which resides.

People use that term, along with just error or residuals interchangeably, but after a model has been tested, it just means how much of the data cannot be explained by the model.

The letter $\epsilon$ is commonly used to explain stochastic noise inherent in (co)-variates of a model, i.e. noise.error that we cannot explain with the given data.


How to Predict/Forecast street's Traffic based on previous values?

This looks like a Time Series problem. So based on a variable's past values, you try to predict the future values.

Usually an "unheard of" problem with Spark, but you are in luck ; spark-ts library seems to be doing what you need, so you don't need to code your own using MLlib. I recommend you try it out and then circle back to something in MLlib if things don't work.

They have introduced a TimeSeriesRDD and once you can encode your data in this data structure (Note that this still behaves like a normal RDD), you can play around with the models available. For example, implementing the ARIMA model would be as simple as  -

val arimaModel = ARIMA.fitModel(1, 0, 1, ts)

Hope that helps!


Initial embeddings for unknown, padding?

In my experience, what works well is :

For padding, fill a zero vector embedding (as pixel intensity in image data padding) is the only and best solution.
For words that don't have a pre-trained embedding, you should try to train them: as you do, fill them with random values when initializing, but set them to trainable.


Using attributes to classify/cluster user profiles

Right now, I only have time for a very brief answer, but I'll try to expand on it later on.

What you want to do is a clustering, since you want to discover some labels for your data.  (As opposed to a classification, where you would have labels for at least some of the data and you would like to label the rest).

In order to perform a clustering on your users, you need to have them as some kind of points in an abstract space.  Then you will measure distances between points, and say that points that are "near" are "similar", and label them according to their place in that space.

You need to transform your data into something that looks like a user profile, i.e.: a user ID, followed by a vector of numbers that represent the features of this user.  In your case, each feature could be a "category of website" or a "category of product", and the number could be the amount of dollars spent in that feature.  Or a feature could be a combination of web and product, of course.

As an example, let us imagine the user profile with just three features:

dollars spent in "techy" webs,
dollars spent on "fashion" products,
and dollars spent on "aggressive" video games on "family-oriented" webs (who knows).

In order to build those profiles, you need to map the "categories" and "keywords" that you have, which are too plentiful, into the features you think are relevant.  Look into topic modeling or semantic similarity to do so.  Once that map is built, it will state that all dollars spent on webs with keywords "gadget", "electronics", "programming", and X others, should all be aggregated into our first feature; and so on.

Do not be afraid of "imposing" the features!  You will need to refine them and maybe completely change them once you have clustered the users.

Once you have user profiles, proceed to cluster them using k-means or whatever else you think is interesting.  Whatever technique you use, you will be interested in getting the "representative" point for each cluster.  This is usually the geometric "center" of the points in that cluster.

Plot those "representative" points, and also plot how they compare to other clusters.  Using a radar chart is very useful here.  Wherever there is a salient feature (something in the representative that is very marked, and is also very prominent in its comparison to other clusters) is a good candidate to help you label the cluster with some catchy phrase ("nerds", "fashionistas", "aggressive moms" ...).

Remember that a clustering problem is an open problem, so there is no "right" solution!  And I think my answer is quite long already; check also about normalization of the profiles and filtering outliers.


The user has divided it's original dataset into two sets: train set and test set.

The model i.e., lasso regression should be trained using train set. Then, the trained model is used to test the model on the test set.

In fact, when you train your model you are trying to find the optimal hyperparameters such as C and regularization (in your code, Grid) via cross validation (in your code, cv).

After finding the optimal hyperparameters, you should train your recently constructed model on the whole train set to find the coefficients (weights) of regression.

Finally, the model is tested on the test set.


That sounds like a knapsack problem. Knapsack problem can be solved a variety of ways. Given your constraints, an approximate solution, like first-fit decreasing (FFD), would probably be acceptable.


How can I plot/display a dataset or an image distribution?

I want to view a specific image or a dataset's distribution, and see
  if they are different. Does this do the trick?

It depends what you want to understand or learn about your data.

  what does each axis mean then?

In all of your plots, the x-axis ranges from 0-255, which is because in all your plots, you are creating histograms of the individual pixel values of your images. A single pixel consists of a vector (tuple) of three values: (red, green, blue). Each of those three colours can take a value 0 to 255 (usually an unsigned 8-bit integer)So by creating a histogram over your images, you are essentially counting how many time each of the possible values appear.

These plots can therefore tell you something about the average colour distribution. The peaks seem to be around 100 for each of your plots, so I guess there are a lot of mixed colours - colour where the RGB values are in the range 100 - 150. So less pixels are purely reg, green or blue e.g. (0, 0, 255) would be purely blue.

You can also compute/visualise the histogram of colours using the OpenCV library, which has great functions for doing exactly this kind of thing (and will run way faster than matplotlib's histogram method because OpenCV uses the C++ backend library). Have a look at this great walkthrough.

  Why do I get ten bins for the single image as well?

The reason all your histograms have 10 bins, is because you are not specifying a value for the bins argument to plt.hist, so the default value is taken from the basic configurations of matplotlib, which you can see by running this:

print(plt.rcParams["hist.bins"])    # will print 10 by default

The y-axes on those two plots do make sense; the dataset with 50k images has higher counts for each pixel value compared to the 10k dataset.

What should I be looking for when it comes to image/dataset distribution?

  Is it the raw values for the whole dataset only? or Is it the raw values for each class? or even each image?

Each plot is showing the distribution of raw values only, for whichever set of data you use. You used mtdataset and mytestset as input, so in each case you are only seeing the distribution for those images of course. There is no inclusion of the actual labels anywhere, so you are not breaking down the distributions into the target classes, for example. Just raw pixel values are considered.

  What should I be looking for when it comes to image/dataset
  distribution?

This is problem specific.

You could be looking to see that there is indeed a distribution of RGB values, as a sanity check that you don't have some really skewed set of colour images that e.g. are mostly black or white. You might compare the distributions of the training and test sets to one another, to ensure that they are similar - meaning the training set is indeed representative of the test set. If this were not the case, any model you might train on a specific task could be biased towards the training set and perform badly on the test set (it will not have seen images similar to the test set if the distributions are very different!)

For training neural networks with images, it is common to normalise the distribution of pixel values to the range [-1, +1], which helps smoother learning via smoother gradient updates.


How to further improve the kaggle titanic submission accuracy?

big question.

Ok so here's a few things I'd look at if I was you..

Have you tried any feature engineering ?(it sounds like you've just used the features in the training set but I can't be 100%)
Random Forests should do pretty well, but maybe try xgboost too? It's quite good at everything on Kaggle. SVM's could be worth a go also if you're thinking of stacking/ensembling.
Check out some of the tutorials around this competition. There's hundreds of them and most of them are great.

Links:

R #1 (my favourite)

R #2

Python #1

Python #2

...Hopefully this helps


Can the size of a pooling layer be learned?

No it is not possible to learn those meta parameters from a data set using a learning algorithm. There is no way to calculate a gradient of any objective function with respect to pool_size and stride params. Even if there were, they are typically discrete values and relatively small integers, so cannot be updated in the same way as e.g. the weights are.

What you can do is search through different values looking for optimal ones, using cross-validation to pick the best. The size and stride meta parameters can also have a large effect on memory use and speed of learning, so it may be the case you are looking to find a good compromise. With stride in lower layers for instance, it may be more effective (in terms of training time required for similar level of accuracy) to have a higher stride value but increase the overall training set size by adding random x/y translations to the training set images.


Trying to understand Logistic Regression Implementation

There are several issues I see with the implementation. Some are just unnecessarily complicated ways of doing it, but some are genuine errors.

Primary takeaways

A: Try to start from the math behind the model. The logistic regression is a relatively simple one. Find the two equations you need and stick to them, replicate them letter by letter.

B: Vectorize. It will save you a lot of unnecessary steps and computations, if you step back for a bit and think of the best vectorized implementation.

C: Write more comments in the code. It will help those trying to help you. It will also help you understand each part better and maybe uncover errors yourself.

Now let's go over the code step by step.

1. The sigmoid function

Is there a reason for such a complicate implementation in phi(t)? Assuming that t is a vector (a numpy array), then all you really need is:

def phi(t):
   1. / (1. + np.exp(-t))

As np.exp() operates element-wise over arrays. Ideally, I'd implement it as a function that can also return its derivative (not necessary here, but might be handy if you try to implement a basic neural net with sigmoid activations):

def phi(t, dt = False):
   if dt:
      phi(t) * (1. - phi(t))
   else:
      1. / (1. + np.exp(-t))

2. Cost function

Usually, the logistic cost function is defined as a log cost in the following way (vectorized): $ \frac{1}{m} (-(y^T \log{(\phi(X\theta))})-(1-y^T)(\log{(1 - \phi(X\theta))}) + \frac{\lambda}{2m} \theta^{1T}\theta $  where $\phi(z)$ is the logistic (sigmoid) function, $\theta$ is the full parameter vector (including bias weight), $\theta^1$ is parameter vector with $\theta_1=0$ (by convention, bias is not regularized) and $\lambda$ is the regularization parameter.

What I really don't understand is the part where you multiply y * z. Assuming y is your label vector $y$, why are you multiplying it with your z before applying the sigmoid function? And why do you need to split the cost function into zeros and ones and calculate losses for either sample separately?
I think the problem in your code really lies in this part: you must be erroneously multiplying $y$ with $X\theta$ before applying $\phi(.)$.

Also, this bit here: X.dot(w) + c. So c is your bias parameter, right? Why are you adding it to every element of $X\theta$? It shouldn't be added - it should be the first element of the vector $X\theta$. Yes, you don't regularize it, but you need to use in the "prediction" part of the loss function.

In your code, I also see the cost function as being overly complicated. Here's what I would try:

def loss(X,y,w,lam):
   #calculate "prediction"
   Z = np.dot(X,w)
   #calculate cost without regularization
   #shape of X is (m,n), shape of w is (n,1)
   J = (1./len(X)) * (-np.dot(y.T, np.log(phi(Z))) * np.dot((1-y.T),np.log(1 - phi(Z))))
   #add regularization
   #temporary weight vector
   w1 = copy.copy(w) #import copy to create a true copy
   w1[0] = 0
   J += (lam/(2.*len(X))) * np.dot(w1.T, w)
   return J

3. Gradient

Again, let's first go over the formula for the gradient of the logistic loss, again, vectorized: $\frac{1}{m} ((\phi(X\theta) - y)^TX)^T + \frac{\lambda}{m}\theta^1$.
This will return a vector of derivatives (i.e. gradient) for all parameters, regularized properly (without the bias term regularized).

Here again, you've multiplied by $y$ way too soon: phi(y * z). In fact, you shouldn't have multiplied by $y$ in gradient at all.

This is what I would do for the gradient:

def gradient(X, y, w, lam):
   #calculate the prediction
   Z = np.dot(X,w)
   #temporary weight vector
   w1 = copy.copy(w) #import copy to create a true copy
   w1[0] = 0
   #calc gradient
   grad = (1./len(X)) * (np.dot((phi(Z) - y).T,X).T) + (lam/len(X)) * w1
   return grad

The actual gradient descent implementation seems ok to me, but because there are errors in the gradient and cost function implementations, it fails to deliver :/

Hope this will help you get on track.


Is machine learning the right tool for this job?

Yes and no,
it sounds ok, but depends on the complexity of the wizard very much. It is possible that a simple rule-based algorithm will be super useful and easier to implement.

If you really want to try it (for example for learning purposes), the bottleneck could be the data which needs to be available for initially training the model. You should have enough user queries (or at least simulated queries) even before your ML wizard exists so that the model can learn.

If it is something you need for work, I would consider consulting somebody with ML/Data Science experience with the specific details of what you need.

Hope, this helps at least a bit.

Best regards,
Jakub


Skip NA in Mean function within Pandas agg function

NaN values are skipped automatically. For example if we have the following table:
  store  reviewScore
0     a          4.0
1     b          3.0
2     a          3.0
3     a          NaN
4     b          5.0

groupby and find mean for each group:
df.groupby('store', as_index = False).agg({'reviewScore': 'mean'})

what is equivalent to:
df.groupby('store', as_index = False).mean()

Output:
  store  reviewScore
0     a          3.5
1     b          4.0

To use arguments in aggregation functions you can utilize a lambda function:
df.groupby('store', as_index = False).agg({'reviewScore': lambda x: x.mean(skipna=True)})

or
df.groupby('store', as_index = False).agg(lambda x: x.mean(skipna=True))

The result should be the same.


Are there Machine Learning Models for Networks?

Coming from the related field of measuring and predicting network security, I'd strongly suggest trying a time-series forecasting. I assume your data is time-stamp based (network congestion values, sampled at some interval? if not skip to the second idea)

1st idea:
I'd borrow from time-series the concept of flattening the 100 measurements into 1 datum.
So instead of:

[t1+delta1, location1, messurement1]

[t1+delta2, location2, messurement2]

[t1+delta3, location3, messurement3]

Fold into:

[t1-bucketed,  loc1, mess1, loc2, mess2, loc3, mess3.]

This would help the model "grasp" the relation between the different measurements, with emphasis on the time axis

2nd idea
Flatten into each measurement row the closest measurements by topology (or even the entire 100 neighbors) into:

[mess, topol-1-mess, topol-1-dist, topol-1-other, topol-2-mess, topol-2-dist, topol-2-other, ..]

This would help the model "grasp" the relation between a specific measurement and its neighborhood measurements, with emphasis on the topology features from each measurement

Please let us know if that helped :)


Backpropagation with multiple different activation functions

In short, all the activation functions in the backpropagation algorithm are evaluated independently through the chain rule, thus, you can mix and match to your hearts content.

What are we optimizing in backpropagation?
Backpropagation allows you to update your weights as a gradient function of the resulting loss. This will tend towards the optimal loss (the highest accuracy). After each forward pass of your training stage, you get an output at the last layer. You then calculate the resulting loss $E$.
The consequence of each of your weights on your final loss is computed using its partial derivative. In other words, this is how much loss is attributed to each weight. How much error can be attributed to that value. The larger this value is the more the weight will change to correct itself (training).
$\frac{\partial E}{\partial w^k_{i, j}}$
How can we compute such a random partial derivative? Using the chain rule of derivatives, and putting together everything that led to our output during the forward pass. Let's look at what led to our output before getting into the backpropagation.
The forward pass
In the final layer of a 3-layer neural network ($k = 3$), the output ($o$), is a function ($\phi$) of the outputs of the previous layer ($o^2$) and the weights connecting the two layers ($w^2$).
$y_0 = o^3_1 = \phi(a^3_1) = \phi(\sum_{l=1}^n w^2_{l,1}o^2_l)$
The function $\phi$ is the activation function for the current layer. Typically chosen to be something with an easy to calculate derivative.
You can then see that the previous layers' outputs are calculated in the same way.
$o^2_1 = \phi(a^2_1) = \phi(\sum_{l=1}^n w^1_{l,1}o^1_l)$
So the outputs of the third layer can also be written as a function of the outputs of layer 1 by substituting the outputs of layer 2. This point becomes important for how the backpropagation propagates the error along the network.
Backpropagation
The partial derivative of the error in terms of the weights is broken down using the chain rule into
$\frac{\partial E}{\partial w^k_{i, j}}$ = $\frac{\partial E}{\partial o^k_{j}} \frac{\partial o^k_{j}}{\partial a^k_{j}} \frac{\partial a^k_{j}}{\partial w^k_{i,j}}$.
Let us look at each of these terms separately.
1. $\frac{\partial E}{\partial o^k_{j}}$
is the error caused by the output of the previous layer. For the last layer, using R2 loss, the error of the first output node is
$\frac{\partial E}{\partial o^3_{1}} = \frac{\partial E}{\partial y_{1}} = \frac{\partial }{\partial y_{1}} 1/2(\hat{y}_1-y_1)^2 = y_1 - \hat{y}_1$
In words, this is how far our result, $y_1$, from the actual target $\hat{y}_1$.
This is the same for all previous layers, where we need to substitute in the errors propagating through the network, this is written as
$\frac{\partial E}{\partial o^k_{j}} = \sum_{l \in L} (\frac{\partial E}{\partial o^{k+1}_{l}} \frac{\partial o^{k+1}_{l}}{\partial a^{k+1}_{l}} w^{k}_{j,l}) $
where L is the set of all neurons in the next layer $k+1$.
2. $\frac{\partial o^k_{j}}{\partial a^k_{j}}$
This is where the current layer's activation function will make a difference. Because we are taking the derivative of the output as a function of its input. And the output is related to the input through the activation function, $\phi$.
$\frac{\partial o^k_{j}}{\partial a^k_{j}} = \frac{\partial \phi(a^k_{j})}{\partial a^k_{j}}$
So just take the derivative of the activation function. For logistic function this is easy and its
$\frac{\partial o^k_{j}}{\partial a^k_{j}} = \frac{\partial \phi(a^k_{j})}{\partial a^k_{j}} = \phi(a_j)(1-\phi(a_j))$
3. $\frac{\partial a^k_{j}}{\partial w^k_{i,j}}$
a is simply a linear combination of w and the subsequent layers outputs. Thus,
$\frac{\partial a^k_{j}}{\partial w^k_{i,j}} = o_i$
Finally
You can see that the activation functions of your layers are evaluated separately in the backpropagation algorithm. They will just be added onto your ever growing back-chain as independent terms within your chain rule.


Error in FUN(newX[, i], ...) : argument "Iclo" is missing, with no default


Is it possible to customize the activation function in scikit-learn's MLPRegressor?

I see in the code for the MLPRegressor, that the final activation comes from a general initialisation function in the parent class: BaseMultiLayerPerceptron, and the logic for what you want is shown around Line 271.

# Output for regression
if not is_classifier(self):
    self.out_activation_ = 'identity'
# Output for multi class
...

Then during a foward pass this self.out_activation_ is called (defined here):

# For the last layer
output_activation = ACTIVATIONS[self.out_activation_]
activations[i + 1] = output_activation(activations[i + 1])

That ominous looking variable ACTIVATIONS is simply a dictionary ,with the keywords being the descriptions you can choose as a parameter in your MLP, each mapping an actual function. Here is the dictionary:

ACTIVATIONS = {'identity': identity, 'tanh': tanh, 'logistic': logistic,
               'relu': relu, 'softmax': softmax}

With all of this information, you might be able to come up with a few ways of putting in your custom function. Off the top of my head, I can't see a quick way to simply provide a function. You could for example:

define your function where all the other activation functions are defined
add it to that ACTIVATIONS dictionary
make self.out_activation_ equal to your custom function (or even a new parameter in MLPRegressor
cross your fingers it doesn't break something somewhere else
run it and solve the inevitable small adaptations that will be necessary in a few places

I'm, afraid I have never looked at the source code of that library before, so cannot give more nuanced advice. Perhaps there is a beautifully elegant way to do it that we have both overlooked.


What if Training and testing dataset comes from the same source?

You have to use a different set of specimens. Or you can keep one or two specimens from the original set aside and use them as test. Use data augmentation and transfer learning in that case.


How does tf.random.log_uniform_candidate_sampler work while generating negative samples?

This parameter isn't used in this tutorial, is works other return values. you can understand this by read the 122ed line of the source code .
source code


transform a supervised neural network to reinforcement learning?

Interesting question!

I think re-using your model in a DQN setting could be interesting. Even more so if you retrain your supervised model every now and then to update the DQN model (although in that case you'd have to figure out how to re-use what your model learned from DQN vs what it learned supervised).

I think to get you started you would have to define your context that you are using RL on. Basic elements that are needed for RL are:

an Agent (whoever is making a decision on what to do)
a State (how can you describe a snapshot of your current environment that has been influenced by an agent's action)
an Action (so something your Agent can actively "do" to induce another state in its environment)
a Reward (something you can "give" your agent to learn if he has chosen something good or bad)

Assuming you want to use something like DQN you have to define those things to be able to run the algorithm. Your current supervised model would then be the starting status of your NN that is used to choose your Agent's decisions.

The keyword for what you are looking for is "Transferlearning", which describes how to use trained models for other cases. In your case even other learning methods.


Python encode piano notes in 88 bits - int too large to convert to C long

That seems like a bad way to encode the information. A C chord and a C7 chord would be very different numbers even though they are similar conceptually.

I would take your idea, but instead of smashing the information into a single int, make your input a binary vector of length 88 created in the way you describe. In this way, a C and C7 will have similar inputs, differing in only one location of the vector.

As a simplified one octave example going from C to B:

C : [1,0,0,0,1,0,0,1,0,0,0,0]
C7: [1,0,0,0,1,0,0,1,0,0,1,0]

This kind of encoding is making use of “dummy variables”. It is like the “Bag of Words” encoding used a lot in Natural Language Processing.


image recognition: fully connected network vs CNN

You might be able to get pretty good results on a simple task, but the fact of the matter is that taking random pixels (or indeed just flattening out ALL pixels) essentially destroys any structural information that was contained within the original image.

This was the insight behind convolution networks (from original author Yann LeCun), as they really find areas of correlation in the position/structure of the image input across the dataset. So they understand, for example, the a "1" offers high correlations of pixels in a vertical line, normally in the center of the image input.

This information is no longer included with randomly selected pixels and has been almost destroyed by flattening all pixels into a single vector.

If your use-case requires a certain accuracy and you are reaching that with your simple neural network (or otherwise), then of course it is perfectly valid and you can be happy  :-)


Class weight degrades Multi Label Classification Performance

Adding class weight but not changing the way you measure performance will usually degrade overall performance as it is designed to allow increased loss on lower-weighted classes.

I would recommend also weighting your accuracy measures. This is a bit tricky with accuracy/precision etc. so maybe calculated the weighted logloss and compare it to the unweighted logloss of the unweighted model.

Basically it comes down to the question: are you happy with a model that performs worse overall but better on your heavily weighted classes?


Need sugestion to accuire a job in data mining

Welcome to the site! The course you are taking sounds like a good start; but I am not sure if it will be enough to land you a job as a Data Engineer (depending on your other qualifications, of course).
If you are willing to invest more time, here is a great text on how to learn Data Science in one hundred hours.


You are getting error because of below-

y=encoder.fit_transform(y.values.reshape(-1,1))

This is changing the shape of the y. try print(y.shape)

You might want to handle you data once its labels and transformed. I am not sure if you are trying to "learn" to apply OneHotEncoding. If yes, you might wanna look at the documentation in detail. here-https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

Let me know if you have tried it and still have problem.


Is there a way to measure correlation between two similar datasets?

I would take a look at Canonical correlation Analysis.


Lasso regression not getting better without random features

In answer to your first question:
The reason that your RMSE proceeded to increase as you increased the strength of your regularization (the value of $\lambda$) can be explained by reviewing the intuition behind what is happening when you increase the regularization of your model.

Why did could my RMSE have kept increasing as I increased my regularization strength?

When you have no penalty (i.e. $\lambda = 0$) the error for your model (RMSE) is inflated significantly. As you penalize the model this error begins to fall. At some point in your modeling, by increasing the value of $\lambda$ too much, you are adding too much bias to the model. At this point, the bias will become too large and your model will begin to under-fit the data. This will result in an increase in the RMSE.
For your model above, this seems to occur somewhere between $\lambda = 10^{-3}$ and $\lambda = 10^{-2}$.
In answer to your second question:
Lasso Regression can be used as a form of feature selection since it shrinks some of the values for the coefficients down to zero. If you increase the value of $\lambda$ significantly, many of the features will be shrunk to zero.

Does the fact that the coefficients of feat1 and feat3 were pushed to
0 at the same time as the random feature mean that they're not good
and I should remove them from the model? (It felt strange that they
hit 0 at the same time. I was expecting some difference.)

To decide whether you should remove any of these features, you should consider a few things. What is the degree of collinearity that exists between your features. You can check this easily since it looks like your have a small number of features. Also, it is important to consider the number of features you have before regularization. If you have a small number, perhaps you do not want to drop a large amount of them.
Most importantly, you should examine the change in your RMSE error associated with each increase in $\lambda$ and choose your $\lambda$ that minimizes the RMSE. If certain of the features are pushed to zero based on the optimal choice of $\lambda$ then maybe they should be removed. The rate at which they move towards zero (both at the same time) does not necessarily suggest that they should be removed, only that they might be candidate features to remove.


Is there any way to calculate the true,false positives and negatives for a regression problem

I believe that there's no actual function in sklearn library that does that. However, that concept is relatively easy to implement: you can loop and compare each y_prediction with the respective y_true_value and increment TN, TP, FN, and FP accordingly to a certain threshold. For example:

if abs(y_pred[i] - y_true[i]) <= threshold:
     TP += 1
 elif abs(y_pred[i] - y_true[i]) > threshold:
     TN += 1
 ...

Don't forget to reverse transform the scaling on y_true and y_pred values first.


how to run my python code on google cloud without fear of getting disconnected - an absolute beginner?

There are several options to keep your session running on the host machine even if it's disconnected, see for instance this: https://www.tecmint.com/keep-remote-ssh-sessions-running-after-disconnection/

Personally I use the screen command because I'm familiar with it but I think tmux is more recent.


Machine learning algorithm for Low dimension input to high dimension output

If I'm understanding correctly you want to model a set of data (in a network form in this case) for which you give 5 input parameters and you get N objects as output of your model, with N>5.

A network basically classifies an object based on the input parametersand outputs the predicted value. Putting a basic network generation, you can classify what car you are looking at based on 3 certain features: "motor size", "number of seats" and "country where the car was designed". From the network you can extract the model and production year (if enough data is provided)and predict what car you have. However, due to the method you are obtaining an output with smaller dimensionality than the input (usually N_output=1).

However, you can associate certain information to that output. In the previous example it could be a technical datasheet of the car model the network outputs. This technical datasheet can contain many features that were not considered in the model (network), which result in N_output>N_input.

The critical step in this case is selecting correctly what output do you want for your network (classification algorithm) and how do you want to associate that output to further insight of that classification.

The algorithm itself to create the network depends on the data and what output do you want. For example, in a basic KNN of the previous case, you select the output as a "compound weighed effect" of all the inputs that result in the network.


What does the oob decision function mean in random forest, how get class predictions from it, and calculating oob for unbalanced samples

Every Tree gets its OOB sample.
So it might be possible that a data point is in the OOB sample of multiple Trees.
oob_decision_function_ calculates the aggregate predicted probability for each data points across Trees when that data point is in the OOB sample of that particular Tree.
The reason for putting above points is that OOB will give you the mean of probability but it will not tell you anything about the standard deviation of the probability across Trees.

Does the oob decision function provide class probabilities,

Yes

and if so, do I get the class predictions by taking whichever number is higher (e.g. by doing something like pred_train = np.argmax(forest.oob_decision_function_,axis=1))?

Yes

Since my classes are unbalanced, would it be correct to say I can't use sklearn's default OOB score here

OOB score is still the default score i.e. Accuracy. So, will not help for the Imbalanced class.


Validation data shall be in broken down into batches or not?

If validation batch size = 1 it has no difference in comparison with the case that load the data completely. But with any other value of batch size the evaluation metric will be an average of it in different batches; So, its real value will not be resulted.


Confusion matrix. "How close I am to the diagonal?". Is there such metric?

The classes that define the columns/rows can be arbitrarily rearranged. Therefore, the "distance" of a misclassification to the diagonal has no meaning. So no, there is no such metric.
I like @Dave's comment: "Is it worse to call a dog a cat than it is to call a dog a horse?"

Maybe you'd ask yourself, "some classes feel closer together than others"
For example, we could create a confusion matrix like this:
$ \begin{matrix}
&&&&PREDICTED\\
&&Person & Woman & Man & Camera & TV
\\
T&Person & 33 & 5 & 3 & 0 & 1
\\
R&Woman & 10 & 50 & 2 & 22 & 0
\\
U&Man & 12 & 23 & 47 & 1 & 13
\\
T&Camera & 4 & 2 & 7 & 24 & 9
\\
H&TV & 3 & 5 & 8 & 13 & 11
\end{matrix}$
(Where $Person$ denotes non-binary of course).
It feels like misclassifying $Camera$'s as $Woman$ should be "more wrong" than misclassifying $Man$ as $Woman$. After all, women aren't objects.
However, in the world of unfeeling classifiers, a "woman" has no meaning, and neither does "object". Hence we call such classifiers "tools".

Edit to address your comment:
In situations where there's some notion of distance, you'd use regression rather than classification. You can use regression even for discrete dependent variables. In other words, you need either numeric or ordinal data for distance to make sense.
To use the example in your comment of "classifying" a variable that represents performance:

Continuous: If you measure performance as a continuous variable, then it's clear to use regression.

Ordinal: But even if you measure performance on, say, an integer scale from 1-10, you can still regress the data. (As an aside, in practice, all measurements can be considered are discrete if you consider that they are limited by resolution/precision). You can also map ordered concepts to discrete but numeric values. For example, the Likert scale (Strongly Disagree, Disagree, Neutral, Agree, and Strongly Agree) can be mapped to integers 1-5. However, the reason you can't directly determine distance without mapping to numerical values is because there's no intrinsic distance between the nominal values. Strongly Disagree could be two units away from Disagree, and maybe Neutral is a billion units away from Agree.

Nominal: If you measure performance using words like "good", "decent", "fine" where there's no clear ordering, then distance makes no sense.


What metrics determine the quality of the model?

Unscaled and scaled r2's are not highly correlated (0.31 AAMOF). Which one would best describe the accuracy of the model on unseen data?

I don't think this is a matter of which will describe the generalization error better, because both of them are describing the same thing, just on different scales. So, the advice would be to use the accuracy metric consistent with the metric that will be used for predictions on unseen data.

  Why isn't the unscaled r2 the same as the scaled r2?

This is because MSE is scale dependent.

  The model r2 is not the same as any of the validation r2's during training (val_r2_keras). Shouldn't the trained model r2 be the same as the one reported during the training?

Why do you think so?

They are different because the datasets for training and for validation are different.


Why replacing null values with outliers?

-999999 is pretty common when you work with trees or forests (you imputatte missing values with the value that differs much from other values to perform better splitting).

Maybe (!) in some rare cases imputation by -999999 may be useful (for example, I used it by clustering) though.

Mean/median as advised are really good-working approaches. Including, for example, boolean column that indicates missing values (=1; before imputation) and non-missing (=0).


How to decided which test of normality to use

Mathematical detail of the two different tests can be found in two papers:

[1]. http://sci2s.ugr.es/keel/pdf/algorithm/articulo/shapiro1965.pdf

[2]. http://www.jstor.org/stable/2334522?seq=1#page_scan_tab_contents

To summarize, both methods are based on hypothesis testing. But they are using different test statistics.

Shapiro-Wilk test has many good analytical properties and was designed to apply to data with sample size less than $50$. However, when the sample size becomes greater, Shapiro-Walk test might be unreliable. As I quote from the second paper:

  Shapiro and Wilk did not extend their test beyond samples of size 50. A number of reasons indicate that it is best not to make such an extension.

Essentially, that is why people invented the second normality test. As I quote from the second paper:

  We present a new test of normality applicable for samples of size 50 or larger which possesses the desirable omnibus property


How do GD, Batch GD, SGD, and Mini-Batch SGD differ?

Gradient Descent is an optimization method used to optimize the parameters of a model using the gradient of an objective function ( loss function in NN ). It optimizes the parameters until the value of the loss function is the minimum ( of we've reached the minima of the loss function ). It is often referred to as back propagation in terms of Neural Networks.
All the below methods are variants of Gradient Descent. You can learn more from this video.
Batch Gradient Descent:
The samples from the whole dataset are used to optimize the parameters i.e to compute the gradients for a single update. For a dataset of 100 samples, updates occur only once.
Stochastic Gradient Descent:
Stochastic GD computes the gradients for each and every sample in the dataset and hence makes an update for every sample in the dataset. For a dataset of 100 samples, updates occur 100 times.
Mini Batch Gradient Descent:
This is meant to capture the good aspects of Batch and Stochastic GD. Instead of a single sample ( Stochastic GD ) or the whole dataset ( Batch GD ), we take small batches or chunks of the dataset and update the parameters accordingly. For a dataset of 100 samples, if the batch size is 5 meaning we have 20 batches. Hence, updates occur 20 times.

All the above methods use gradient descent for optimization. The main
difference is that on how much samples are the gradients calculated.
Gradients are averaged in Mini-Batch and Batch GD.

You can refer to these blogs/posts:
Batch gradient descent versus stochastic gradient descent
Gradient Descent Algorithm and Its Variants


I don't understand what you mean by

  So I have a report that lists the url and the number of visits a person did. So a combination of ip and url result in an amount of visits.

Assuming that you equate an IP with a user, and you wish to cluster users by their URL visitation frequencies, your matrix, M, would have

One row per IP (user)
One column for each URL that you are tracking (your features)
and the entries in M would be "visits" of a given URL by a particular IP

Given these assumptions, and your report, M would be:

    abc.be  abc.be/a  abc.be/b
123   5        2         2
321   0        0         4


how to consider some miss classifications “half correct” using as base categorical_crossentropy - for a trading system

I worked on a similar problem where classes were "ordered", so mis-classifications of class B as C was better than mis-classification as D.

In that project, "weighted quadratic kappa" loss function worked well.

Details :

https://en.wikipedia.org/wiki/Cohen%27s_kappa#Weighted_kappa

Example of Keras implementation :

https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/quadratic_weighted_kappa.py


Support Vector Machines attempt to find a hyerplane that divides two classes with the largest margin.  Therefore this is an optimization problem.  Support vectors are the points that lie along the supporting hyerplane.  One key factor that plays into the complexity of the runtime for a support vector is the slack parameter C.   The slack parameter allows for a soft-margin and better generalization.  The number of support vectors varies depending on how much slack we allow and how the data is distributed.  The less slack we give the SVM the fewer support vectors we get and converserly the more slack we give it the more support vectors we receive.

There also exist a relationship between the complexity of the model. The more complex the models tend to require more support vectors.

In short, it doesn't really matter how many features you feed it, the SVM computational complexity is linear in the number of support vectors.


StackOverflow Tags Predictor...Suggest an Machine Learning Approach please?

This exact problem was a kaggle competition sponsored by Facebook. The particular forum thread of interest for you is the one where many of the top competitors explained their methodology, this should provide you with more information than you were probably looking for: https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/6650/share-your-approach

In general, it appears that most people treated the problem as a supervised one. Their primary feature was a tf-idf, or unweighted BOW, representations of the text and they ensembled 1000s of single-tag models. Owen, the winner of the competition, noted that the title text was a more powerful feature than the content of the body of the post.


Group-by correlation vs All observations correlation

The name of the procedure you are looking for is

  Autocorrelation

This procedure looks a lot like the first you describe.

Answer fot this question in ResearchGate


Resources for text classification algorithms

So, I believe what this question is asking for literature around sentiment analysis of Facebook comments (positive/neutral/negative) sentiment.
There is quite a good paper on this using an encoder-decoder architecture here: https://www.aclweb.org/anthology/Q18-1002/


Over/Under Sampling for Multi-classification

You are applying  xgb and random forest to a multiclassification task and you are doing under sampling to some how try to improve your class

If i understand your histogram, that is the distribution of labels.

When you say accuracy went down for both classes you are refering to a multiclassficitaion problem right?

Answers directly to your questions:

There are same cases when over or undersampling improves your model, but there are not many. Selecting the right loss is more important.
Consider using another can also help you to improve your model.


Why is video classification still not that accurate?

I've used py-faster-rcnn on video feeds and found it to be pretty usable in a sunny environment. Although on a GTX660, I could only process one image every 600ms, which may highlight a limitation. It still takes a lot of time to compute an inference on a deep net, so for a minimally fast framerate, of 10fps, you're still limited to a speed of 100ms.

There might also be other artifacts due to camera dynamics such as image blurring, sensor saturation, or low light levels that might be more present in video feeds of natural scenes as opposed to the fairly well exposed images in static image datasets.


Keras ModelCheckpoint Callback returning weights only even though both save_best_only & save_weights_only are set to False

Keras should be accessed as tf.keras now with tf2 ,so your import should be written as
tf.keras.callbacks.ModelCheckpoint

Keras own documentation as well as tf api documentation can be easily accessed for this purpose. Keras ModelCheckpoint class mentions the following arguments in official docs:
save_best_only: if save_best_only=True, it only saves when the model is considered the "best" and the latest best model according to the quantity monitored will not be overwritten. If filepath doesn't contain formatting options like {epoch} then filepath will be overwritten by each new better model.
save_weights_only: if True, then only the model's weights will be saved (model.save_weights(filepath)), else the full model is saved (model.save(filepath)).
So both 'save_best_only and save_weights_only'  have default value as False and will save all weights and full model if not True.


What is the function of the hidden layer in a neural network mapping to?

Dependent variable: Yes, the output variable is dependent value but by being fed what do you mean? (We have a training set and a test set. We feed our model via training set (both independent and dependent values) and then predict the values for the dependent variable in the test set)

There are some features for each sample that are independent like temperature, heart rate, etc. Based upon these independent values, a feature (outcome) is predicted, for instance, whether someone is prone to health issues or not (classification problem). So this output (health issues) that depends on independent features ( temperature, heart rate, etc. is dependent)

Note: each pair of independent variables might be correlated.

If you are trying to predict if an image is a cat or dog, then the outcome might be different based on what activation function we have chosen. for instance, if we have chosen sigmoid activation function, the output would be two probabilities, determining the probability of cat and dog. But if we use the threshold function, it gives 0 or 1.

Actually, the way in which the hidden layers are acting in a neural network varies for different sort of layers like Dense, convolutional, recurrent, etc.

For instance, Dense layers try to figure out what the weights of the features (independent variables) should be, meaning, how important each feature is in determining the output (dependent variable), and then based upon the function we utilize (sigmoid, rectifier, etc) would return probability of each outcome, binary outcome for dependent variables with two outcomes, etc.

What do you mean by "How do the nodes learn different features rather than the same one?"


How would one separate digits for number recognition?

In Image Processing, this task is known as localization. You basically want to localize each digit in the image and then use your digit recognizer over the digits. A cursory google search for digit localization in images gives me following papers which seem to be very helpful.

Reading Digits in Natural Images with Unsupervised Feature Learning
Character Recognition In Natural Images


Can I expect necessarily higher accuracy when feature selection/dimension reduction is used to select a subset of features?

Using all features would usually lead to over-fitting, that is your model would not generalise well to unseen data. To overcome this, while using as much information possible we resort to feature selection ( sometimes feature generation ), and dimensionality reduction. Dimensionality reduction techniques like Principal Component Analysis (PCA) tries to find new features that explain maximum variability in data, using the existing features. In this way, we ideally reduce the number of features in the model, while making sure most of the variability is explained.
So as a result:

This new model is computationally faster since we are no longer using all the features and burdening our model.
Generalises well since we are not over-fitting leading to greater accuracy on unseen data.


The question you should be really asking is,

  What is your problem statement? What are you trying to predict?

It will then be automatically clear what your target variable is.


Need some info regarding string matching algorithms?

Try the simplest approach first - deterministic check looking for intersection overlap between the set of fruit names and the set of items bought.

Set comparisons are scalable because the look-up time for each item is constant.

If scaling is an issue with regular set membership check, bloom filter is an option.


Using Mean Squared Error in Gradient Descent

No, it is exactly the same. Optimizing a function and the same function divided by a constant is equivalent, both in the analytical and the numerical sense. You will get exactly the same optimal parameters.


please refer this question

Books recommended by Yann Lecun can be found here and by G. Hinton here


You can take a look at the YoloNet which detects Objects based on Pascal VOC 2012 dataset.
Here is the link to it : https://github.com/pjreddie/darknet/wiki/YOLO:-Real-Time-Object-Detection

You should be able to detect vehicles with this pre-trained model.


Measuring performance of different classifiers without class type in data

Normal classification will not work in this case, classifiers learn functions that are able to seperate the training examples. In your case you only have one class which will not work for all the classical examples. There is a machine learning task called One-class classification, see this Wikipedia page. In your case PU Learning (Positive and Unlabeled) seems most appropriate. Evaluating the performance is extremely difficult without having some negative examples as well however. See this question and answer on the Stats stackexchange.


Hugely dependent on both user and audience preference (violin plots being more unusual could throw people), so it is mostly up to you.

One main reason to go for a violin plot is to give more detail about the distribution, as box plots just give hard stops at the mean, stddev and 2 stddevs. Therefore if you think there is interesting information contained in the distribution between those points go violin.

The other main reason is they're more eye catching on Kaggle ;)


Is over-fitting a matter of features engineering or a matter of parameters set?

A 1/3 - 2/3 repartition is not that unbalanced. Your problem shouldn't require balancing.

The train/test set partition seems to be done correctly, as it seems implied by checking data histograms. Doing that randomly is usually ok, and when it's not it will inflate your test performance with data leakage, which doesn't seems to be the case here.

Imo the problem come from your learning process, or more exactly : when does it stops ?

If you are going to explore your parameters space by hand, I would suggest to go look at the parameters of your learner : https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html, and for each one of them ask yourself what would be the impact of increasing / decreasing them on the learning capacity of your algorithm.

It may be a bit counterintuitive but you have to actcually limit the capacity of your learner, so that it does not learn too much.

Then you can do some grid search on the main parameters driving the learning / performance.


R langauge how to create xgb.DMatrix object from data frame (newbe)

Welcome to the site!

Assume that y is your response, and x is your data set of predictors (where categorical variables have been appropriately converted to numeric). Your data does not necessarily need to be sparse, although sparse data will improve computation speed.

Then dtrain <- xgb.DMatrix(label = y, data = as.matrix(x)).

As you get more proficient with XGBoost you can start exploring the weight and base_margin parameters.


IDE alternatives for R programming (RStudio, IntelliJ IDEA, Eclipse, Visual Studio)

RIDE -   R-Brain IDE (RIDE) for R & Python, Other Data Science R IDEs, Other Data Science Python IDEs. Flexible layout. Multiple language support.
Jupyter notebook -   The Jupyter Notebook App is a server-client application that allows editing and running notebook documents via a web browser. The Jupyter Notebook App can be executed on a local desktop
Jupyter lab - An extensible environment for interactive and reproducible computing, based on the Jupyter Notebook and Architecture.
Radiant – Open-source platform-independent browser-based interface for business analytics in R, based on the Shiny package and can be run locally or on a server.
R Tools for Visual Studio (RTVS) -
A free, open-source extension for Visual Studio 2017, RTVS is presently supported only in Visual Studio on Windows and not Visual Studio for Mac.
Architect -  Architect is an integrated development environment (IDE) that focuses specifically on the needs of the data scientist. All data science tasks from analyzing data to writing reports can be performed in a single environment with a common logic.
displayr - Simple and powerful. Automation by menu or code. Elegant visualizations. Instant publishing.
Collaboration. Reproducibility. Auto-updating. Secure cloud platform.
Rbox - This package is a collection of several packages to run R via Atom editor.
Use below for more IDEs:
RKWard - an easy to use and easily extensible IDE/GUI for R
Tinn-R - Tinn-R Editor - GUI for R Language and Environment
R AnalyticFlow - data analysis software that utilizes the R environment for statistical computing.
Rgedit - a text-editor plugin.
Nvim-R - Vim plugin for editing R code.
Rattle - A Graphical User Interface for Data Mining using R.
How to Turn Vim Into an IDE for R


How to deal with missing data for Bernoulli Naive Bayes?

Your search results are on point: without dropping or imputing data, there's no built-in way to do what you want with BernoulliNB.

There is, however, a way out: train separate Bayesian models on filtered samples from your data, and then combine their predictions by stacking them.

Filtering

Filtering here means:

Isolating samples from your original df, each having only a subset of df.columns. That way, you'd have a DataFrame only for content_2, one for content_2, content_3, in a sort of a factorial combination of columns.
Making sure each sample is made only of rows that have no NaNs for any of the columns in the subset.

This part is somewhat straightforward in your case, yet a bit lengthy: you'd have $n!$ (n factorial) combinations of columns, each of which would result in a separate sample. For example, you could have a sample named df_c2 containing only content_2 rows valued 0 or 1, df_c2_c3 with only content_2 and content_3 columns filled, and so on.

These samples would make NaN values non-existent to every model you'd train. Implementing this in a smart way can be cumbersome, so I advise starting with the simplest of scenarios - e.g. two samples, two models; you'll improve gradually and reach a solid solution in code.

Stacking Bayesian Models

This is called Bayesian Model Averaging (BMA), and as a concept it's thoroughly addressed in this paper. There, weight attributed to a Bayesian model's predictions is its posterior probability.

The content can be overwhelming to absorb in one go, be at ease if some of it doesn't stick with you. The main point here is that you'll multiply each model's predicted probabilities by a weight 0 < w < 1 and then sum (sum results shall be in $[0, 1]$). You can attribute weights empirically at first and see where it gets you.

Edit:

Due to the added complexity of my proposed solution, as stated in this (also useful) answer, you could opt to implement Naive Bayes in pure Python, since it's not complicated (and there are plenty tutorials to base upon). That'd make it a lot easier to bend the algorithm to your needs.


Keras: Merged/Concatenated model perform worst than separate models memes recognition

I had the same problem and I solved it adding a BatchNormalization layer.
For example:

# create model ConvLSTM
input_convlstm = Input(name='convlstm', shape=(n_steps, 1, n_length, n_metadata))
branch_convlstm = BatchNormalization()(input_convlstm)
branch_convlstm = ConvLSTM2D(filters=64, kernel_size=(1,3), activation='tanh', input_shape=(n_steps, 1, n_length, n_metadata))(branch_convlstm)
branch_convlstm = Flatten()(branch_convlstm)
branch_convlstm = Dense(128, activation='tanh')(branch_convlstm)

# create model metadata
input_metadata = Input(name='metadata', shape=(21,))
branch_metadata = Dense(neurons[0], activation='tanh')(input_metadata)
branch_metadata = BatchNormalization()(branch_metadata)
branch_metadata = Dropout(dropout_rate)(branch_metadata)

# create final model
concat = Concatenate()([branch_convlstm, branch_metadata])
out = Dense(n_outputs, activation='softmax')(concat)
model = Model(inputs=[input_convlstm, input_metadata], outputs=out)


pandas groupby.count doesn't count zero occurrences

Use crosstab instead:
pd.crosstab(combine_df.class, combine_df.feature)


Have you tried a batch normalization layer between the conv2D layers and the dense top? I'd also keep the number of filters low ,maybe 32 or 64. Same for the number of nodes in the top, no point in having a huge dense top with hundreds of nodes.


Assigning numerical IDs to variable values in a data file

You can use the Python sklearn preprocessing LabelEncoder. Here is some example code from this page with my comments:

# Make a Label Encoder instance
le = preprocessing.LabelEncoder()

# Show it the data it has to encode, so your column
le.fit(["paris", "paris", "tokyo", "amsterdam"])

# Get a ordered list of all classes it found
list(le.classes_)

# Transform a column/list
le.transform(["tokyo", "tokyo", "paris"])

# Transform encoding back to original
list(le.inverse_transform([2, 2, 1]))


Why use convolutional NNs for a visual inspection task over classic CV template matching?

The answer depends on the task.  Template matching can work for some tasks but not for all.  CNNs potentially have the ability to generalize to unseen inputs that don't match any of your templates, so can potentially generalize better.

But whether CNNs will beat template matching will depend on the specific task and what specifically you're trying to achieve.  This is an empirical science; ultimately, the way you find out which works better is to try them both -- or learn from others who have tried them (e.g., by reading the literature).  I don't think you're going to find some theory or taxonomy that is going to substitute for empirical evaluation on real-world data.


Comparing files (text sources) in Orange

The similarity hashing widget can compute document similarity within a certain corpus.


Linear Regression: Why use global basis functions instead of local basis functions

I see a couple of parts to your question

Part 1

  Why would we use global basis functions instead of local basis
  functions to introduce non-linearity into our linear machine learning
  model?

Local (non-global) basis functions are used sometimes. For example, radial basis functions (RBF) are sometimes used as a basis in linear regression (see these lecture notes).

However, often it matters that a model is interpretable and more complicated models are harder to interpret / draw certain conclusions from.  For example, imagine you're doing a controlled experiment, you may not care about "just" predicting some variable from (the variable you're manipulating + the variables you're controlling for).  But rather, how much of an effect your manipulation has on the output.

Part 2

  Why don't we discretize our feature space and optimize on a linear function space?

RBFs kind-of discretize feature space.  People don't usually literally discretized feature space because discrete optimization is much much more complicated than continuous optimization.


It should be hp['income2'] because you can't assign a mutable object to an immutable object such as a string


Timeseries LSTM: does test data need to come after training data?

If I got it correctly, you might have the following scenearios:

labeled samples with tag A or B per date-time index, having as informative attributes the present + some lag values of interest --> (this would be a standard classification approach without time ordering bein necessary)

sliding window of samples (what you mean by subsequences) --> here you should respect the time ordering at least for the validation set, so you make sure you evaluate your LSTM with a real scenario with future never seen sequences.
With this second approach, you can indeed shuffle the training batches (via the keras shuffle(BUFFER_SIZE).batch(BATCH_SIZE) functionality for instance (info here), but leaving the validation set without shuffling, as follows:
  BATCH_SIZE = 256
  BUFFER_SIZE = 10000

  train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi,y_train_multi))
  train_data_multi = train_data_multi.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()

  val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))
  val_data_multi = val_data_multi.batch(BATCH_SIZE).repeat()

You can find a complete worked-out example here
You can also make use of the time series data preprocessing helper where you can decide whether to shuffle or not.


It is a typo in the solution.
The author corrects it by correctly computing the Bayes discriminant in the next step.


How to use class_weight parameter for validation set?

You are not supposed to apply weights to your validation set since it is supposed to measure your model's performance. If you'll do that you'll probably get better results for validation but once your model is deployed it will perform worse on new data.
Weighting, resampling techniques etc. - they all should be done on the training set only!


Does the training set of one topic will be useful to predicate the sentiment for any other topic?

Since you have trained your model on air-line tweets, the model will learn the characteristics of the air-line tweets. There could be words used in air-line tweets which are positive and contain great weightage but never even used in other tweets leading to negative results.
I suggest you to try

Get a dataset which contains all kinds of tweets which you desire essentially similar to what your task ahead is.
Look into Transfer learning.


Is there a way to Label/Annotate My Bubble Plot (Scatter plot with a z-axis) on matplotlib?

You can use the seaborn package, using the scatterplot marker size to generate your bubbles. Then you need to loop over the datapoints and add a text labet to each point in your scatterplot.

# Load libraries
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns

# Create example dataframe
df = pd.DataFrame({
'x': [1, 1.1, 1.2, 2, 5],
'y': [5, 15, 7, 10, 2],
's': [10000,20000,30000,40000,50000],
'group': ['Stamford','Yale','Harvard','MIT','Cambridge']
})

#Create figure
plt.figure(figsize = (15,10))

# Create scatterplot. alpha controls the opacity and s controls the size.
ax = sns.scatterplot(df.x, df.y, alpha = 0.5,s = df.s)

ax.set_xlim(0,6)
ax.set_ylim(-2, 18)

#For each point, we add a text inside the bubble
for line in range(0,df.shape[0]):
     ax.text(df.x[line], df.y[line], df.group[line], horizontalalignment='center', size='medium', color='black', weight='semibold')

Which outputs:


Input 0 is incompatible with layer conv2d_2: expected ndim=4, found ndim=3 I get this error in Tensor flow, What does it mean and how can I fix it?

The error is likely to come from this line:
model.add(Conv2D(32, kernel_size=(3, 3),activation='relu',input_shape=X.shape[1:]))

Input_shape should be a 4dim vectors as stated in the keras doc:

Input shape
4D tensor with shape: (batch, channels, rows,
cols) if data_format is "channels_first" or 4D tensor with
shape: (batch, rows, cols,
channels) if data_format is "channels_last".

You may have to reshape your data, as stated here:
https://stackoverflow.com/q/43895750/8119313


Wrong Graph Plot using K-Means in Python

Well, there are some issues:

Dimension vs K: Before talking about visualization I would like to address some clustering concept. Your data is in 7 dimensions but it does not mean that you have 7 clusters! Be careful here. For instance I have two features of people let's say salary and number of years they have working experience. Here I have two features but does it mean that there necessary two categories inside the data? sure not!
Visualization: Your data is in 7 dimension which is not visualizable. So you decided to reduce this to two which is a correct approach but you did a wrong thing for this correct approach. You can not take the first two features to visualize 7 dimensions, you need to REDUCE it to two features using Dimensionality Reduction algorithms like PCA, NMF, etc. What you did is actually IGNORING 5 dimensions of the points which are extremely informative for placing them in a 7-dimensional space.

Solution

Everything is right. Just add a PCA to your code like this:

From sklearn.decomposition import PCA
Model = PCA(n_components=2)
X_new = Model.fit_transform(X)
... Use X_new instead of X for K-means procedure

Please note that I wrote this relying on my memory so better to check the documentation if I had a typo or smth. In case you have more question you can comment here.

Good Luck!


How to detect overfitting of a stock screener

Learning curves or bias-variance decomposition are the gold standard for detecting high variance, aka: overfitting.  Separate your data (in your case the "back data") into 60% training data and 40% testing data.  Fit the model on the training data as you usually would and see how well it is working with the test data.

Finally, when you think you have the model that you want, split each of the training and test sets into 10-100 subsets and retrain and test with incrementally larger sets.  Apply your favorite performance metric and plot the results of performance vs. the number of cases used for testing and training.

The curves will never come together if the model is overfit (high variance).  The curves will come together but the performance will be lower than desired if the model is underfit (high bias) and the lines will come together at an acceptable performance for a well performing model that is not overfit.

Here is an example of overfitting and underfitting with root mean square error as the performance metric:

Here is a pretty good link on the process and here is another one.  Hope this helps!


Dropout Decreases Test and Train Accuracy in one layer LSTM in Pytorch

It might be because of learning rate. You need to reduce your learning rate and let it run for more epochs.


How to extract specific parts of text from a string?

It is a specific pattern which can be detected by numbers and after that a unit name with two or three letters.
In the other words, detect numbers first, and then find numbers which after of them there is two or three letters.
Moreover, you can check these letters are in a predefined list of units or not.


Trying to understand the result provided by np.linalg.norm function in numpy (normalisation)

The formula you cited is not the formula scipy is using. According to the documentation it calculates the "Frobenius norm" which is defined to be the root of the squared sum of the elements of each row (row since you specified dim=1)

$ O_i = { \biggl(\sum_{n=1}^N \mathbf{A}_{i, n}^2\biggr) }^{1/2} $

Thus, $(3^2+4^2)^{1/2}=5$ and $(2^2+6^2+4^2)^{1/2}=56^{1/2}$

EDIT: Also, the keepdims argument just says whether to collapse the dimension that you calculated or not, a simple example is to look at the output shape of the matrix that you get with and without the argument.

a = np.array([[0, 3, 4], [2, 6, 4]])

np.linalg.norm(a, axis=1, keepdims=True).shape # output is (2, 1)
np.linalg.norm(a, axis=1, keepdims=False).shape # output is (2,) because the second dimension was collapsed


Text classification with thousands of output classes in Keras

The ratio of number of examples to number of classes is not large. There are few classes which have high number of occurrences (from the second graph) and the distribution seems to be following power law.

In such cases, I'll advise the following strategy,

Sort your tags by number of occurrences and discard tags which occur very few number of times. This will make the problem more tractable.
You can benchmark the accuracy that you get from classical machine learning techniques. Many classical methods support multilabel output, you can check the documentation of support in scikit-learn library here.
You can do a mix of unsupervised learning and nearest neighbours approach. For example, learn doc2vec embeddings on your data including tags and suggest tags from nearest matching document for a new input. The number of documents is small by doc2vec standards and you will need careful tuning of doc2vec parameters.
With neural networks, you can use more suitable loss functions like multilabel soft margin loss.

I'd advise you start with classical techniques to first benchmark the potential accuracy.


How to test/validate unlabeled data in association rules in R?

You may want to consider using your own APparameter object to put "significance" constraints on the rules learned by Apriori. See page 13 of the arules documentation. This could reduce the number of uninteresting rules returned in your run.

In lieu of gold standard data for your domain, consider bootstrap resampling as a form of validation, as described in this article.


Use machine learning to predict next schedule meeting for sales officers

What I understood is, since your sales person skipped the customer meeting you want to display details or send a notification like:

  Last date of conversation
  Discussion points and so on.

such that sales person decide her/his priority based upon the importance of discussion.

If so, I'm not sure how much it's possible with ML. But one thing is possible, where you can trained RNN(Recurrent Neural Network) based time series model. This model will predict the next meeting time, which is based upon the pattern of historical meeting data.

In that way you can book sales person calendar along with the last meeting Key-Discussion points or send an update in case sales person hasn't attended that person.


Product classification based on Description of a product in eCommerce website

http://jmcauley.ucsd.edu/data/amazon/

Who else but Amazon!

(the exact link you want I believe) -
http://jmcauley.ucsd.edu/data/amazon/amazon_readme.txt


What is the best way to encode features when clustering data?

Try Hierarchical Clustering, Kmeans will not serve the purpose if you have categorical data.


Sources that I've found so far;

datakind
Data4Democracy
Data Science for Social Good
eScience DSSG projects

Other ideas: find a set of data that interests you depending on where you live for example and contribute to your community. Things like data.gov in US and open-data in Canada.


Correlation Matrix for non-numeric features

There are measures of association for categorical variables. If you are looking at two ordinal variables you may use Spearman's correlation coefficient. There are also many measures for association for purely categorical variables, such as gender and race. Yule's Q and Crammer's v are popular choices.


What does big O mean in KNN optimal weights?

From Optimal weighted nearest neighbor classifiers Richard J. Samworth
derived an asymptotic expansion for the excess risk (regret) of a weighted nearest-neighbor classifier. This allows us to find the asymptotically optimal vector of nonnegative weights, which has a rather simple form. He showed that the ratio of the regret of this classifier to that of an unweighted k-nearest neighbor classifier depends asymptotically only on the dimension d of the feature vectors, and not on the underlying populations.
From the paper in (Page 11-12)

In a complementary approach to that taken in most of this paper, Audibert and Tsybakov (2007) study the minimax properties of plug-in classifiers.
They show in particular that a certain classifier obtained by modifying a
the local polynomial estimator of the regression function $η$ attains the minimax
rate over a set of distributions $P$ of random vectors $(X, Y )$ on $\mathbb{R}^d * \{ 1,2 \}$ for which the regression function belongs to a H¨older class, $P$ satisfies a margin condition and the marginal distribution of $X$ satisfies a so-called strong density assumption. This rate is $O(n^{−4/(d+4)})$ when the H¨older smoothness parameter is 2, and the margin power parameter is 1.

From the properties of Hölder condition

Any α–Hölder continuous function f on a metric space $X$ admits a Lipschitz approximation by means of a sequence of functions $(f_k)$ such that $f_k$ is k-Lipschitz and
$ {\displaystyle \|f-f_{k}\|_{\infty ,X}=O\left(k^{-{\frac {\alpha }{1-\alpha }}}\right)}$.

Here we can state that the Big $"O"$ is just an $"O"$ in Holder condition with Lipschitz approximation and not a Big-O-notation


Negative examples and false positives in object detection using SSD model

I have found a way to create TFRecord that accepts images with no annotations using this code.


Applying mean encoding before or after splitting into train and test set

The purpose of having a test set or a validation set is to be able to check the performance of your model on data it has not seen before. If you perform feature engineering with the test data present you will get a data leakage. That happens when give your model information about your test data during training.

It is especially bad when doing target encoding with the label mean since it will give your model information about the distribution of the labels in the test set. The effect is that you will get an overly optimistic test score that will not reflect the performance on new truly unseen data.


If you read the entire file without skipping jumbled lines, does it stil work? What values appear in those jumbled cells?

You could go down the road of imputation, i.e. filling the missing gaps, based on something you can deduce from the data that is there.

Example are:

fill-forward: fill will preceding non-missing value
fill-backwards: fill with following non-missing value
average-filling: if numerical, fill gaps with the mean/median etc. of that column. You could also use something like a moving average.
model-based: fill missing values using e.g. a regression model on available values (this needs a target variable, or at least timestamps)

One last snity-check: I assume it isn't possible for you to try creating the CSV file again and addressing the problem at its source?


validation_curve differs from cross_val_score?

First of all, you have to shuffle your data because it seems that the model has learned a special pattern in the training data which has not occurred in the test data so much. After that, suppose that you get a validation curve like the current one. As you can see, Increasing the value of depth, does not change the learning. The two lines are parallel. In cases which each of the lines may have intersection, the upper line has negative slope and the lower one has positive slope, in the future on seen levels, you may want to increase the number of levels, not in this case.

Having same error, means that you are not over-fitting. but as you can see the amount of learning is not too much which means that you are having high bias problem, which means you have not learned the problem so well. In this case means that your current feature space maybe has high Bayes error which means that there are samples which have same features and different labels. Actually the distributions of different classes overlap.

There is something to argue about decision trees. If you have numerical features which are continuous, you may not have exactly same input patterns but they have overlap in their range.