ML models: average of all versus average of averages?

It shouldn't matter if you average all of them or take the average of the average for each person. It turns out that it will give you the same number. This is known as the law of total expectation. Essentially,

E(X)=E(E(X|Y))

Here's the wikipedia article for more details:

https://en.wikipedia.org/wiki/Law_of_total_expectation


What does images per second mean when benchmarking Deep Learning GPU?

I'm wondering if a given number of images per second, say 15000, means that 15000 images can be processed by iteration or for fully learning the network with that amount of images?.

Typically they specify somewhere whether they talk about the forward (a.k.a. inference a.k.a. test) time, e.g. from the page you mentioned in your question:

Another example from https://github.com/soumith/convnet-benchmarks (mirror):


Machine Learning - Input Prepocessing - NLP email classification model

So, from what I have gathered, you are asking how to preprocess the new (I suppose unobserved) emails, which do not appear in the training set.
In that case, you should convert your email text into 1000-dimensional vector, where each value corresponds to a particular feature value.

I am going to go with the basis that you are simply counting the number of times any of the most frequent 1000 words occur in a new email (let's call it $x^{(1)}_{test}$).

To convert the email into a vector form, here is one way of doing it:

import pandas as pd
import numpy as np

words = ["blah", "tea", "tetra", "pak"]

def vectorise_email(words, e_mail):

    """Vectorise email to a vector of word counts based on a list of words.

    :param words: (List of Strings) List of frequent words in training set
    :param e_mail: (String) E-mail string

    :return word_counts: (Numpy Array) containing counts of words based on words list.
    """

    e_mail = pd.DataFrame(e_mail.split())

    e_mail = e_mail[e_mail[0].isin(words)][0].value_counts()
    word_counts = np.zeros(len(words))

    for w_idx, word in enumerate(words):
        word_counts[w_idx] = e_mail.at[word]

    return word_counts

print(vectorise_email(words, "this is a blah tetra pak tea tea blah blahh"))

Here we firstly tokenise sentences into words (I used the standard string split method, but you could use nltk's tokenise methods [https://www.nltk.org/api/nltk.tokenize.html]). Then we convert this list into a pandas DataFrame to use the value_counts method (Ref: https://stackoverflow.com/questions/22391433/count-the-frequency-that-a-value-occurs-in-a-dataframe-column) to get the word counts for those words which appear in the word list.
We then complete the vectorisation process by mapping these counts into a Numpy Array where each element in the array corresponds to a particular word count in the input e-mail.

Hope that helps


Is SVD non-linear while PCA (by eigendecompostion) is linear?

To the best of my knowledge no.

SVD and PCA are both linear dimensionality reduction algorithms. Some nonlinear dimensionality reduction algorithms are e.g. LLE, Kernel-PCA, Isomap, etc.

About t-SNE I would like to add a point. It reduces the dimensionality (and does it pretty well!) but it is only for visualization and can not be used in learning process! So be careful putting all these next to each other. In other words, they are all dimensionality reduction algorithms however, PCA and SVD can be used for feature extraction but t-SNE can not. All can be used for visualization purposes (in EDA).

I certainly recommend reading this answer. Probably the fact that "the square roots of the eigenvalues of $XX^⊤$ are the singular values of $X$" confused your friend that it's a nonlinear method.

Hope it helps. Good Luck!


Slice your DataFrame accordingly, and then apply your previous routine (you may want to put this in a function) to this part of the data:

df2 = df.loc[df.Year >= 1950]


What is the difference between cross_validate and cross_val_score?

cross_val_score is a helper function on the estimator and the dataset.
Would explain it with an example:
>>> from sklearn.model_selection import cross_val_score
>>> clf = svm.SVC(kernel='linear', C=1)
>>> scores = cross_val_score(clf, iris.data, iris.target, cv=5)
>>> scores
array([ 0.96...,  1.  ...,  0.96...,  0.96...,  1.        ])

This example demonstrates how to estimate the accuracy of a linear kernel support vector machine on the iris dataset by splitting the data, fitting a model and computing the score 5 consecutive times (with different splits each time)
The cross_validate function differs from cross_val_score in two ways -

It allows specifying multiple metrics for evaluation.
It returns a dict containing training scores, fit-times and score-times in
addition to the test score.

Note: When the cv argument is an integer, cross_val_score uses the KFold or StratifiedKFold strategies by default, the latter being used if the estimator derives from ClassifierMixin
You can go through this link for better understanding
Different examples using cross_val_score, you can go through about its different implementations.


Building a data frame and matrix for citation network

Are the 1s and 0s only to indicate that the source matches the citation? If so, try this :

library(tidyverse); library(magrittr)
cit.total %>%
mutate(score = 1) %>% # so that all 'matches' in your original data frame are tagged '1'
spread("Citations", "score", fill = 0) # each source and citation pair gets a number, 1 if the match exists, 0 if it doesn't


Tradeoffs between Storm and Hadoop (MapReduce)

MapReduce: A fault tolerant distributed computational framework. MapReduce allows you to operate over huge amounts of data- with a lot of work put in to prevent failure due to hardware. MapReduce is a poor choice for computing results on the fly because it is slow. (A typical MapReduce job takes on the order of minutes or hours, not microseconds)

A MapReduce job takes a file (or some data store) as an input and writes a file of results. If you want these results available to an application, it is your responsibility to put this data in a place that is accessible. This is likely slow, and there will be a lag between the values you can display, and the values that represent your system in its current state.

An important distinction to make when considering using MapReduce in building realtime systems is that of training your model, and applying your model. If you think your model parameters do not change quickly, you can fit them with MapReduce, and then have a mechanism for accessing these pre-fit parameters when you want to apply your model.

Storm: A real-time, streaming computational system. Storm is online framework, meaning, in this sense, a service that interacts with a running application. In contrast to MapReduce, it receives small pieces of data (not a whole file) as they are processed in your application. You define a DAG of operations to perform on the data. A common and simple use case for Storm is tracking counters, and using that information to populate a real-time dashboard.

Storm doesn't have anything (necessarily) to do with persisting your data. Here, streaming is another way to say keeping the information you care about and throwing the rest away. In reality, you probably have a persistence layer in your application that has already recorded the data, and so this a good and justified separation of concerns.

If you want to know more...
If you would like to learn more about real-time systems that that fit parameters with MR and apply the models a different way here are slides for a talk I gave on building real-time recommendation engines on HBase.

An excellent paper that marries real-time counting and persistence in an interesting way is Google News Personalization: Scalable Online Collaborative Filtering

Another interesting marriage of MR and Storm is SummingBird. Summingbird allows you to define data analysis operations that can be applied via Storm or MR.


Proper feature selection methods for classifying signal into two category

Generally, feature selection is somewhat of a fuzzy process. Since you usually don't have a ground truth in predicting biology, you will always have to consider how realistic whatever you came up with is. I would recommend to start with the most simple method and see how your model performs. After establishing this as a baseline, you can then explore other feature selection methods to see if that improves your model.
For most models, there are general assumptions that should be met - check this before trying to optimise your features. So, for example, if one feature is just the sum of featureA and featureB, you don't need it in your data. Also, of course, always be cautious about over/under-fitting and do proper cross-validation. And if you come up with a set of features, it might make sense to think about their implications since ML is prone to find shortcuts that are only specific to your data.


What are good toy problems for testing Transformer architectures?

I would recommend the pre-processed datasets for language translation in the Stanford NLP group: https://nlp.stanford.edu/projects/nmt/

There are three datasets with sizes 0.13Mb, 4.5Mb and 15Mb - something for everybody  :)

Translation is something that is very difficult to solve for normal FF-networks and something that transformers dramatically improved due to attention mechanisms.

Also check out the related explanations of those ideas in The Annotated Transformer and the Illustrated Transformer


The usual rule is to ask only one question per post.  I will answer your first question.

Build a dictionary, once, in advance.  With a bit of effort you should be able to construct a dictionary that has all the words that are likely to appear in the future. Take all news articles over the past year, or all of Wikipedia, or some other very large collection, and it's likely to contain all the words you need; the likelihood that you're missing some important word is pretty low.  Now, each time you see an article, throw away any words not in the dictionary.  That way, you won't need to vectorize the old news over and over every time; you can do it once and be done.  That should improve performance.

Another variation: if you see a new word on some day, you can add that word to the dictionary; at the end of the day retokenize everything you saw that day.


This is an example of multi-parallel-time-series.

Example shared on github MultiParallelTimeSeries, believe that will be helpful.


One-hot encoding with values other than 1

If you use 0s and 1s, the design matrix is composed of indicators variable. An indicator variable tells the presence (1) or absence (0) of a variable. The model weights are the magnitude. If you choose something other than boolean values, the model weight will the relative weights to the larger values in the design matrix.
Additionally if you also have numeric values and are fitting a linear model, you should standardize the numeric values. Then all variables are scale from 0 to 1 which allows the model to learn better and faster.


"Stationarity of statistics" and "locality of pixel dependencies"

At the time of writing the article, CNNs were not yet a particularly popular architecture for neural networks (and neural networks in themselves weren't as popular and common as today). In fact this article can be seen as the one who started the current age of deep learning as a default machine learning approach.

All this intro was for the soul purpose of saying that by "standard feed forward neural networks", the author meant neural networks that consist only out of dense layers (fully connected).

So to answer your last question first, in theory, anything a convolutional layer can do, a fully connected layer with the same number of input parameters and output parameters can also accomplish. i.e. you can produce a convolutional layer from a fully connected one (it will just have the same weights repeat themselves in a pattern of the same size as the convolution kernel), and you can also produce a lot more operations that a simple convolutional layer cannot perform. The simplest example can be a location dependent convolutional layer (where different areas of the image are convolved with different kernels). In reality, the search space for optimal parameters of such a layer would be so huge (and so non-convex), that it won't be able to converge.

The limitations on the convolutional layer are actually what helps her shine, which brings us to your first question. The "locality of pixel dependencies" is exactly what allows such a limited operation, to achieve such fine results. The meaning of the sentence is very simple, close pixels are very likely to be dependent on each other, so we can and should leverage this dependency to process them together (as is done with convolutional kernels).

Regarding the "stationary of statistics" phrase, I'm not 100% sure, but I think it refers to the fact that at small local patches, there is a higher probability of finding recurring patterns (the smaller the patch size, the smaller the possible variance of the patch). However as I said, I'm not sure about this one.


What is the name of the sub-field within spacial Data Science where the data relates in some way to physical objects viewed as 3D (graphics) Models?

There are many possible answers to this question.
Data science applied to mechatronics and cybernetics can involve physical information, which I believe is what you are describing. The DS side of mechatronics often involves IoT and engineering cybernetics principles. More specific sub-fields include design optimization and control theory.
Edit to add a few more examples:

Fault detection
Anomaly Detection
Simulations

Monte Carlo

Modeling and Simulation


Accuracy keep changing by changing randomState of classifier

Pal, you have set alpha to 1. Alpha is a L2 regularization term, its value is normally around 0.0005 ... 0.0001. By setting it to 1 you force the optimizer to make your model's weights almost zero.

P.S.
After setting l2 reg term to its default value, you might get better accuracy, but if not, pay attention to the following. The ratio between the amount of data size (1500 samples, how many feature vectors?) and the number of weights (20x60x60x60xN, so more than 4M weights) is too small. It might well be that the model simply memorize the training set and does not generalize for sound classification.

Here are some suggestions that might help in training a sound classifier:

Start with a simpler model. Sounds are known to be well classified with Gaussian mixture models, for instance. Unlike deep learning models, those are easier to train.
Use data augmentation to increase the amount and diversity of your training data. You can mix the sounds with some light background noises.
Try to reduce the amount of weights. Three layers with 60 neurons each is too much. Usually, layers become smaller with going up, something like 64->32->16. Also, convolutional layers can be very useful here, as they share the weights across their neurons. I used them quite successfully for sound classification.
scikit-learn is quite unusual choice for training deep learning models. I would try Keras, it works perfectly with numpy arrays.


Error on multitask neural nets where all outputs not observed for every example

You have already found the answer by yourself: if there are missing values, you should not backpropagate on them. I do not know of any particular name for this action.

I do not think that there are already implemented functions to do so, but it seems quite straightforward to embed your loss function (e.g. mean squared error) into a personalized one.

For example, let's say that you filled all missing values with $-1$:

(Disclaimer: I did not test this code but got inspiration in this answer)

import theano
from keras import backend as K

def custom_objective(y_true, y_pred):
    indices = y_true==-1
    y_pred2 = y_pred[indices]
    y_true2 = y_true[indices]
    # then you could use any already implemented loss function, or
    mse = K.mean(K.square(y_pred2-y_true2))
    return (mse)

Note that with nan coded as 0 you could use flatnonzero().


as.POSIXct formats dates so as to include the time, while as.Date formats so as to include just the year, month, and date, i.e. yyyy-mm-dd.

Assuming you have a column x with the dates and times included as you mentioned, i.e. 2004/12/15 9:30, then you can simply apply:

date<-as.Date(x)

Now, you can check if it is in date format through using:

str(Date)

You should find that the times are no longer visible and only the dates are displayed.


(De-)Scaling/normalizing input and output data inside Keras model as layer

Let's start from X. As you use the sklearn-style interface, the natural choice is to use StandardScaler. You could use pipeline to integrate it in the grid search.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipeline = Pipeline([
   ('scaler', StandardScaler()),
   ('model', your_cool_model)])
param_grid = {...}
search = GridSearchCV(pipeline, param_grid, cv=5).fit(X, y)

That would do the trick.

Now about y. The chances are you don't need to scale target. The good discussion is in this question

If you are certain that targets should be scaled, the sklearn provides TransformedTargetRegressor just for the case

wrapped_model = TransformedTargetRegressor(
    regressor=model,
    transformer=StandardScaler())


How to refit GridSearchCV on Multiclass problem

Nevermind, I realized my mistake.
precision and recall don't work with multiclass data. They should be replaced by precision_macro and recall_macro instead.
Speaking of which, the one place where I tried using them on the code should be also changed into:
best_precision = clf.cv_results_['mean_test_precision_macro'][i]
best_recall = clf.cv_results_['mean_test_recall_macro'][i]


Appropriate algorithm for string (not document) classification?

You might find it useful to treat n-grams of characters as your feature space. Then you could represent a string as a bag of substrings.  With N = 4 or greater, you would capture things like ".com" in emails, "##.#" in dates, etc.

It might also help to encode all single digits as one reserved number-only-character.

An easy way to to this might be to create all the n-gram substrings for each string in your dataset, then simply treat that list of words as a document.  Then you could use term frequency or tf-idf vectors in your supervised step.

For example to create the substring uni-, bi-, and tri-grams for "whatever@gmail.com":

a = "whatever@gmail.com"
b = set([a[x:x+y] for y in range(0,4) for x in range(0,len(a))])

set(['', 'co', 've', 'ai', 'eve', 'r@', 'at', '.co', 'gm', 'ev', 'tev', 'er', '@gm', 'ver', '@g', 'r@g', 'ail', 'il.', 'gma', '.', 'te', 'hat', '@', 'wha', 'om', 'wh', 'er@', 'mai', 'ma', 'ha', 'l.c', 'a', 'c', 'ate', 'e', 'g', 'i', 'h', '.c', 'm', 'l', 'o', 'l.', 'r', 't', 'w', 'v', 'com', 'il'])


Actually you can often just split on the white space, remove punctuation, and lowercase. This will provide your tokens. For example, if I had a string "jdf asdsa sdfr" (no English), then I could derive the tokens jdf, asdsa, and sdfr. The only thing that is known before hand would be the stopwords (and, be it the), which do come from the English dictionary. However, in this case, it does not sound like you would need stopwords. I would suggest looking at the following libraries available in python:

SpaCy

NLTK

Scikit-learn


One option is hashing, assigning a numerical value to each widget. The best hashing option is if each widget had a unique id, something like a serial number. If a widget does not have inherent unique, then a hash value can be created by applying a hash function to the features of widget.
After creating hash values for each widget, set comparison can be used to see if any hashed values are different between the two sets of widgets.


Best forecast model for insurance policies volumes

I assume Renewals/Cancellations are your expected output, so it can be considered as a binary classification.
From where I stand, all you mentioned (policies) in your first paragraph is meaningless because I cannot see how I can use them to build any model. Rather, you better study what the clients who used to renew/cancel respectively look like. And that is the mighty "feature"! For example, you might take into consideration their gender, age, ethnicity, and annual income, etc. That would be more helpful than thinking only about your policy.
Or, you can also try a probabilistic solution: think about the probability of the occurrence of cancellation of your company in the past, and the probability of the occurrence of cancellation of other companies given that they execute the same policy. What can you predict?


Big Data and a fwf files reading with filters

Try the LaF package. See this:
https://stackoverflow.com/a/24716798/1427069

Once you create a laf object, you can create a data frame or data table using your row filter condition.

library(LaF)
laf <- laf_open_fwf("foo.dat",
                    column_widths = colWidths,
                    column_types=rep("character", length(colWidths)),
                    column_names = colNames)
castData <- laf[RecordID == "Cast", ]


Low accuracy using Multi Layer perceptron network on CIFAR10 dataset?

CIFAR10 has 10 class label. So by random guessing, you should achieve an accuracy of 10%. And this is what you are getting. This means your algorithm is not learning at all. The most common problem causes this is your learning rate. Reduce your learning rate by replacing your line,

model.fit(X_tr,Yt,validation_data=(X_ts,Yts),epochs=10,batch_size=200,verbose=2)

with

optimizer = keras.optimizers.Adam(lr=1e-4)
model.compile(loss='squared_hinge',optimizer=optimizer,metrics=['accuracy'])

I just did a test run, it is learning slowly.


How do I use recency/time as a feature after I've a word2vec model

The purpose of the Word2vec is it will itself learn a hidden structure in your text data. If "recovery" comes after rap and music is because Eminem is more frequent with these context words than with recovery. You can't simply change that. You should avoid making a manual intervention on your model.

However, here are 2 ideas you could try if you really want "recovery" to be closer of Eminem word :

Try an other Word2Vec algorithm architecture. There are CBOW and skip-gram you can try. In my opinion CBOW could perform better because the CBOW model trains each word against its context. It asks "given this set of context words, what missing word is likely to also appear at the same time?". Skip-gram trains each the context against the word. It asks "given this single word, what are the other words that are likely to appear near it at the same time?"
Add sentences relative to recovery album in your data. Because words recovery and Eminem should dominate in these sentences, It could reconcile the 2 words. Same, you can replicate sentences containing both Eminem and recovery word and randomly change the order of words in the sentence.


how to make sklearn pipeline using custom model?

Implementing a custom transformer is simple. You have to implement the fit and transform methods like below. Since your ANN is already trained (right?) the fit method has to do nothing, just return self. And the transform method has to pass the incoming data to the ANN and return its output.

from sklearn.base import BaseEstimator, TransformerMixin

class MyANNTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, ann):
        self.ann = ann

    def fit(self, X, y):
        return self

    def transform(self, X)
        return self.ann.predict(X)

Now you can include that in pipelines:

from sklearn.pipeline import make_pipeline
from sklearn.svm import SVC

pipe = make_pipeline(MyANNTransformer(ann),
                     SVC())

pipe.fit(Xtrain, ytrain)
pipe.predict(Xtest)


difference between logistic regression and binary logistic regression

binary:logistic is used for binary classification where the target variable takes binary output [0, 1]

reg:logistic is used for regression where the target variable is continuous between [0, 1]

Quote from xgboost doc:

We use linear regression here, if we want use objective = reg:logistic logistic regression, the label needed to be pre-scaled into [0,1]

https://github.com/dmlc/xgboost/tree/master/demo/regression


how to provide alternate for a word in corpus for word2vec model

Add a regularization term for synonymity to the objective function (to be maximized):

$\log P(w_i|h) \rightarrow \log P(w_i|h) + \lambda \frac{1}{|S|} \sum_{(i,j) \in S} sim(w_i, w_j)$ where $S$ is the set of synonyms, and $sim$ is your word similarity function (e.g., cosine similarity).


Why don't we convolve our images against the last convolution layer and see how many of these complex feature filters get activated?

The answer is that all the layers are fully dependent on the exact features of the previous layer. The last layer is simply not capable of taking a raw image as input and outputting meaningful values. Most likely it is not even going to be able to fit to the shape of the output, it will require dozens of channels, where a colour image only has 3. If you somehow forced it to fit, then the colour image data is meaningless to it. It might still be made to output something, but it would essentially be gibberish, and unrelated to the desired outcome.

Each layer is a function of the previous layer. Ignoring the details of convolution, a neural network is essentially a composition of multiple functions (let's call them $f, g, h, i, j$ for example) so that:

$$y = j(i(h(g(f(x))))$$

You are essentially asking here, can you just do $y = j(x)$ instead of running all those functions in sequence. And the answer is no.


How similar is Adam optimization and Gradient clipping?

Both have a different role and approach. So, I will say both are not comparable.
Gradient Clipping has a simple task to clip the Gradient to a certain threshold. Its job is done as soon as this is achieved, i.e. Gradient is brought to a decent level to avoid the Explosion.
It has no responsibility to see if learning will converge to best possible minima of Loss.
Adam, on the other hand, is an optimizer. It came as an improvement over RMSprop. The improvement was to have the goodness of both, i.e. Momentum and RMSProp (Read this answer)
Adam is expected to help the learning converge towards the minimum loss function when it is going in Valley or Plateau by managing both the Momentum and Coordinate specific Gradient.
Coming back to your question

why do we need gradient clipping to solve the problem of gradient explosion when we can use Adam optimizer to do a controlled search of the space for the minima.

Let's say, in the first iteration, the Model accumulate very large Gradient because of its depth. So, it will take a considerable jump when updating the weight ($\theta$) and can land to any random coordinates in the Loss function space.
Adam can definitely provide a brake on the Gradient, but it has few challenges -

It needs a few iterations to accumulate the $v$ to apply the brake.
But it is very much possible that during these iterations, the point will take any random coordinates in the space. This randomness will also cause all the coordinates to have random Gradients i.e. no relation to the last Gradient.
This will not let Adam do any accumulation and it will have no clue to act in a corrective manner. Everything will happen haphazardly.

Also, keep in mind, exploding Gradient has bidirectional causation, i.e. explosion causes more Gradient, and then this significant Gradient adds more to the Explosion. So, the learning reaches very quickly to NaN state.


Does statsmodels compute R2 and other metrics on a validation-/test- set?

No.
There's another question over on Stack Overflow concerning wrapping statsmodels models to use the sklearn conventions, so that you can use sklearn cross-validation, hyperparameter searches, etc.


border_mode for convolutional layers in keras

With border mode "valid" you get an output that is smaller than the input because the convolution is only computed where the input and the filter fully overlap.

With border mode "same" you get an output that is the "same" size as the input. That means that the filter has to go outside the bounds of the input by "filter size / 2" - the area outside of the input is normally padded with zeros.

Note that some libraries also support the border mode "full" where the filter goes even further outside the bounds of the input - up to "filter size - 1". This results in an output shape larger than the input.

There's a short explanation in.. numpy's convolve documentation:

http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.convolve.html


What's the math for real world back-propagation?

There is Matrix Calculus, (and I would recommend the very useful Matrix Cookbook as a bookmark to keep), but for the most part, when it comes to derivatives, it just boils down to pointwise differentiation and keeping your dimensionalities in check.

You might also want to look up Autodifferentiation. This is sort of a generalisation of the Chain Rule, such that it's possible to decompose any composite function, i.e. $a(x) = f(g(x))$, and calculate the gradient of the loss with respect to $g$ as a function of the gradient of the loss with respect to $f$.

This means that for every operation in your neural network, you can give it the gradient of the operation that "consumes" it, and it'll calculate its own gradient and propagate the error backwards (hence back-propagation)


A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation

Again, the entropy equation coded was this:  entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.)) which looks different from what's given in the textbook photo above.

They are the same after simple algebraic manipulations.
The entropy of a single variable Gaussian distribution with pdf $p(x|\mu, \sigma)$ is
\begin{align}
  \mathcal{H}(p) & = ln(\sqrt{2 \pi e \sigma^2}) \\
                 & = 0.5 \cdot ln(2 \pi e \sigma^2) \\
                 & = 0.5 \cdot (ln(2 \pi \sigma^2) + 1)
\end{align}
In policy gradient, we assume these $\sigma$s are isotropic. Thus the total entropy is the sum above $\mathcal{H}$s.

  Also, why is the loss a negation? actor_loss = -exp_v? Is it negated because it is gradient ascent rather than gradient descent of the objective function for a policy gradient?

Yes. In policy gradient you would like to maximize the likelihood log_pdf of getting higher episode returns advantages. At the same time we prefer a policy of high entropy. Deep learning frameworks usually only provide optimizer for minimizing the loss. Therefore you negate exp_v as loss.


Using extracted feature vector to perform zero shot detection

The differentiation of shape starts in the initial layers. Colour information is exploited in further layers. Your network has less layers, try increasing your layers (keep filters low in the initial layers).
Also, if you are looking at similarity models, look at contrastive training or ArcFace loss or siamese type models. If you want to stick with classification networks, look at GeM pooling.
Best way would be to use MobileNet (or better networks such as EffNetB4) to finetune with GeM pooling.


Unsupervised Clustering for n-length word arrays

Assuming the dimensionality is reasonable, I would not use K-means or any generic algorithm, instead I would write a code which directly gives me the exact result by building a map of the groups:

// Assuming data is an array of size N containing all the arrays
// clusters is a map associating each group with a set of arrays
for i=0 to N-1
  for j=i+1 to N-1
    group = overlap(data[i], data[j])
    add data[i] to the set clusters[group]
    add data[j] to the set clusters[group]

An alternative version if the number of different values and size of the sets allow it and/or if it's possible to precompute the groups of interest:

for i=0 to N-1
  for every subset S of data[i]
    add data[i] to the set clusters[S]


Relative feature importance w.r.t hyperparameters

Yes. The most obvious example is when using a Lasso regression : for an increasing $\alpha$ parameter you will have more and more coefficients set to zero. This resulting in a smaller set of features and thus a bigger share of feature importance for remaining features.


Decision Trees - how does split for categorical features happen?

You are right on all counts:

  If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits?

Yes absolutely, for exactly the reason you mention below:

  Should it rather perform the split based on == and != for this variable? But then, how will the algorithm know that it is a categorical feature?

Yes, as you correctly assume a (true) categorical variable should be compared only for equality, not order.

In general the algorithm cannot guess the nature of the feature, there has to be some parameters in the implementation which provide it with this information. Some implementations allow this, for example with Weka the features are typed with either a "numeric" or "nominal" (categorical) type.

  Also, will one-hot encoded values make more sense in this case?

Correct again, that's what should be done for a categorical feature in case the implementation treats all the features as numeric values.


Mercer's Theorem and infinite-dimensional spaces aren't used directly. It justifies use of things like the Gaussian kernel in SVMs. Mercer's theorem says this kernel is just an inner product in some other space, but we need not figure out what that space is, or a mapping to it. The fact that it exists is essential to proving that the SVM still works.


When making decisions about which data to use in a model you have to be aware of several pitfalls. One of them is information leakage i.e. including data that contains information that you shouldn't have at the time of prediction.
Both Duration and Goals are data points that you do not have at the time of prediction (that is before a match) and therefore should not be used in a prediction model.
This leaves us with only one information point: the teams involved. You could still make a ML model but it would only tell you something trivial: which of these teams won more head 2 head games in the past because that is the only information it would use to determine the likely winner (therefore the model would also always predict the same outcome because the input does not change).
This means you need much more data to actually make a model. Modern models for predicting the outcomes of sport events use data such as recent performance of the teams, time (e.g. time of day, weekday, season, etc.), recent performance of all involved players, context (e.g. weather, etc.).
It does not seem like your dataset contains this information so I am afraid you will be unable to create a model.
I would recommend to start with a different problem that has a better dataset available and is easier to crack for a beginner. The IRIS dataset is a wellknown beginner ML problem and teaches you how to model a classification algorithm.


When to Perform non-linear dimension reduction

The question is too general to have a specific recipe as an answer. Depends on the data and the relation between features but to the best of my knowledge:

When the relation between your features are non-linear i.e. when a non-linear interpretation of correlation between features makes more sense than the linear interpretation.

A detailed explanation of what I mean can be found in this nice answer. Please note that the $X^TX$ in the 4th line of the first paragraph should be $XX^T$, because in data mining, we conventionally use $N\times D$ format for data matrices. So in this example, we are looking for the gram matrix of the rows and not columns.

Hope it helps :)


PCA simply finds more compact ways of representing correlated data. PCA does not explicitly compact the data in order to better explain the target variable. In some cases, most of your inputs might be correlated with each other but have minimal relevance to your target variable. That's probably what is happening in your case.

Consider a toy example. Lets say I want to predict stock prices. Say I'm given four predictors:

Year-over-year earnings growth (relevant)
Percent chance of rain (irrelevant)
Humidity (irrelevant)
Temperature (irrelevant)

If I apply PCA to this data set, the first principle component would relate to weather since 75% of the predictors are weather related. Is this principle component relevant? It's not.

The two options you've highlighted boil down to using CFS or not using it. The option that uses CFS does better because it explicitly selects variables that have relevance to the target variable.


Different definitions of Macro F1 score, which one is used in Scikit-learn?

The first variant is implemented:
$$F1_{macro}= \ \sum_{classes} \frac{F1\text{ }of \text{ }class}{number\text{ }of\text{ }classes}$$

You can find an example calculation in this answer.

Sometimes the scikit learn documentation does not include all the details. In these cases it is often helpful to look into the source code which is linked on all help-sites. Here you can find some more details on the f1 score calculation.


Feature selection - QR code localization

The idea would be to find

1) Rotational independent and
2) Shift invariant features.

Simply speaking the Fourier transform is a rotational and translational (shift) independent feature.
There are SIFT features used in computer vision that can extract such features. This is something you can try.

Since you are using QR codes,I think edge magnitude maps would be better than images because you could use image processing techniques to suppress the  noise in the background and highlight the edges on the QR code. You could even look at autoencoder and CNN related feature extraction.


Why Decision Tree Classifier is not working with categorical value?

Its all ok.

Exactly because your values are between 1 and 2 does he chooses 1.5. In other words trees will select 1 and 2 as values (integers not real numbers) but 1.5 is there as cut-off point between these classes.


What should I do with the NaN values on this stock quote data?

Filling ups the nan based on the quote that already started is possible with time series models, but it won't bring new information that you can learn on.  At best it will obfuscate what the model has learned on that period.

Then there is the more general question about the performance : does the model perform better with or without that period of reconstructed data ? Only you can answer that by working it up... But I am not sure you want an answer to that question. The way to go imo is to just drop that period and see if the performance fits your need. If it isn't the case then you should consider some ways to add up more data.

In this cases i'd suggest either to just drop the NaN if you want to work with neural nets or work with a method that accept Nan as inputs (Xgboost for exemple).


How can calculate Efficiency for predictive models based on accuracy or error over time?

Given that computation time is system dependent maybe you could consider MAE/number of parameters (in your case it would be weights since they are all NNs)

If you want to use time, your are right cant compare apples with oranges, so just drop the percantage and make sure they are normalized values (all future metrics)


Policy gradient/REINFORCE algorithm with RNN: why does this converge with SGM but not Adam?

Without the code there's not much we can do but, I'd guess you need to significantly lower the learning rate. From my experience Adam requires a significantly lower learning rate compared to SGD.


This seems to me as the plain old recommendation problem. The Accounts are the USERS and the Symptoms are the ITEMS. Each time an Account shows a particular Symptom your system will increment a count value.

Creating the following dataset:

ACCOUNT, SYMPTOM, COUNT

Now you can use/implement any sort of recommender system (Mahout is only an option, have you seen MyMediaLite) or you can even implement yours.

Let's reuse your ideas:
* You'd like to use a user-based similarity
* If an Account has shown 2 or more times the same symptom it seems to be important

So you can filter out the Account, Symptom pairs with less than 2 counts, and with the rest you create the following datasets:

User, Item dataset:

ACCOUNT, SYMPTOM

Table with a unique column containing all Users:

ACCOUNT

Table with a unique column containing all Items:

SYMPTOM

Now you can use directly the User-KNN algorithm from MyMediaLite.

With the recommender model already trained you can pass any ACCOUNT as input and it will give you a ranked list of the most probable SYMPTOMS that might appear.

Obs.: Initially ignore the time, then you could use it to partition your data in past/future and evaluate the recommendation in a more realistic way. ;-)


Do I need to encode the target variable for sklearn logistic regression

The string labels work just fine, here is an example:

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
import numpy
X, y = load_iris(return_X_y=True)
y_string = numpy.array(['YES' if label == 1 else 'NO' for label in y])
clf = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial').fit(X, y_string)
y_pred = clf.predict(X[50:100, :])
print(y_pred)

Output:

['NO' 'NO' 'NO' 'YES' 'NO' 'YES' 'NO' 'YES' 'NO' 'NO' 'YES' 'NO' 'YES'
 'NO' 'NO' 'NO' 'NO' 'YES' 'YES' 'YES' 'NO' 'NO' 'YES' 'YES' 'NO' 'NO'
 'YES' 'NO' 'NO' 'YES' 'YES' 'YES' 'YES' 'YES' 'NO' 'NO' 'NO' 'YES' 'NO'
 'YES' 'YES' 'NO' 'YES' 'YES' 'YES' 'NO' 'NO' 'NO' 'YES' 'NO']

Yo can replace y_string to y for the numerical example.


Why Pair plot is taking up the target variable into consideration?

maybe because Surv_status is number.
there are two solution:

define variables:

sns.set_style("whitegrid"); sns.pairplot(patients, hue="Surv_status",vars=["AGE", "Op_Year", "axil_nodes_det"], size=3);

change target value to a string:

patients['Surv_status'] = patients['Surv_status'].astype(str)

or

    patients['Surv_status'] = patients['Surv_status'].str


Error while using lightGBM's cv() function for a regression problem

Answer
In the param_grid dictionary, pass every hyper-parameter as array e.g., in your code above, you missed-out to place 'objective': 'regression' part of param_grid as an array. Even though, you have are using single value place it as array i.e. 'objective': ['regression']
Therefore, you can update below piece of code in your script and re-run. It will remove the error.
`
param_grid = {
    'class_weight': [None, 'balanced'],
    'num_leaves': list(range(30, 150)),
    'learning_rate': list(np.logspace(np.log(0.005), np.log(0.2), base = np.exp(1), num = 1000)),
    'subsample_for_bin': list(range(20000, 300000, 20000)),
    'min_child_samples': list(range(20, 500, 5)),
    'reg_alpha': list(np.linspace(0, 1)),
    'reg_lambda': list(np.linspace(0, 1)),
    'colsample_bytree': list(np.linspace(0.6, 1, 10)),
    'objective': ['regression'] # <<< CHANGE REQUIRED TO remove ERROR
 }


Web Scraping: Multiple small files or one large file?

The danger of one large file is that you may not be able to read the whole thing into memory at once.  Since you already know that 3 months of data will be ~2GB, this should not be a problem, so I would recommend option 2.

Perhaps an even better option is to save the data as several medium-sized files.  For example, you could combine all the observations for a week into a single file.  This way, you eliminate nearly all of the file I/O overhead (~13,000 files down to 12-14) but also free yourself from potential memory problems.
This is certainly what I would do if I were collecting data for a very long time (years instead of months).


How to improve identification of outliers for removal

You can try a quantile-quantile plot: you fit your data to a given distribution and plot your empirical distribution quantile against the theoretical quantiles.

This can help you infer outliers in your sample as outliers for the fitted distribution.


Which algorithm to use to predict the duration of some task

What others have said is accurate, that you need to build a regression model of some sort. Depending on the scale of the duration of your task, you will have to model it slightly differently.

In fact, there's an entire class of models that try to predict duration. These are called survival models. Here's a Python library on survival analysis.

But these models are fairly academic. A typical hack is to use Gamma, Poison, or Log-Normal regression which works out nicely because these models predict non-negative values.


Low leves of probability observed after modelling.Is it right to scale the probability

It sometimes happens that a classification model will output probability estimates that are all in the low range. That means the model does not make any predictions that it is very sure are the positive class. Since only 10% of your data falls into the positive class, it appears to be a difficult problem to predict using your model.

Do not transform the probability estimates. When this happened to me multiple times, the predictions were still useful. For example, you may be interested in the highest 10% of predictions. You can always set your prediction threshold anywhere between 0 and 1.


Neural Network accuracy and loss guarantees?

For my answers, I assume you are talking about batch (not mini-batch or stochastic) gradient descent.

No. Assume you initialize all weights with the same value. Then all gradients (in the same layer) will be the same. Always. Hence the network effectively only learns one parameter per layer. It is possible (and likely) that this is neither a gloabl nor a local minimum of the network (which has more parameters).
Yes, as the learning rate is "small enough for all practical purposes". (No, if you use SGD or mini-batch gradient descent)
Usure. I think the correct answer is "No, the network can make more mistakes in between with cross entropy.". It is certainly sure to improve CE loss while at the same time getting worse at accuracy (see proof below). However, I'm not sure if the gradient would ever lead to such a result.

Example for 3

#!/usr/bin/env python

from math import log

def ce(vec):
    """index 0 is the true class."""
    return -(log(vec[0]) + sum([log(1-el) for el in vec[1:]]))

a = [0.1001, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0999]
print(ce(a))
b = [0.49, 0.51, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
print(ce(b))

gives:

ce(a) = 3.24971912864
ce(b) = 1.42669977575

Hence the cross entropy loss dropped (as expected), the probability for the correct class increased (as expecte) but it makes a mistake (if you simply take the argmax).


Why do we add "αd" to N in Laplace Smoothing?

You're right, the idea is that, if $\alpha d$ is not used, then you might have $\hat{\theta_i}$ bigger than 1.
That being said, I don't think the main reason of Laplace smoothing is to have probabilities greater than 0. I think the important thing is that it acts as a regularization technique (a smoother, indeed).
The smoother works the following way: we are mixing the empirical observations ($\frac{x_i}{N}$) and the theoretical distribution without data ($\frac{1}{d}$), and $\alpha$ is a parameter that controls how much do we care about the theoretical distribution. The idea is: the more we trust the theoretical distribution, the more regularization we are adding. It even has a Bayesian interpretation in terms of priors and posteriors.
So I don't think it has to do with probabilities being between 0 and 1, I think it has to do with performing the mixture of distributions (or update of the posterior, if you want) properly.


Clusterize item set with items as vectors of features

1.you just have to represent those  features as numeric in a vector
eg:[2,4,8,10]
2.Its a good  practise to normalize vector i just took sum of elements and divide by each element by that sum of elements  =[0.06666666666666667,0.13333333333333333,0.2,0.26666666666666666,0.3333333333333333]
normalize the values in that vector to be between 0 -1
3.feed the vectors into clustering algorithm (you can try with kmeans)


Choosing k=2 because you want to find classes "dead" and "alive" does usually not work. That is a common toy approach that barely works on exercise data such as the Iris data set.

Because k-means is unsupervised, there is no guarantee that for k=2 it will find dead/alive. It may as well find male/female (k=2) or Chuck Norris/everybody else (k=2).


NameError 'np' is not defined after importing np_utils

Y_train = np_utils.to_categorical(y_train, n_classes)
Y_test = np_utils.to_categorical(y_test, n_classes)

please update code line as np_utils not np.utils


Should I use MongoDB instead of storing data in CSV in python?

If you're a beginner it might be useful to experiment with a database. In realistic settings, data is usually found in databases. So knowing how to interact with a database is also important for a data scientist.

cv data is tabular, while mongodb is key-value so it is also an opportunity to explore new data representation schemes.


How do i convert a Named num [1:4] to a data frame in R?

My preferred way to transpose a data.frame (or data.table) is to use the transpose function found in the data.table package.

It means you might have to install it: install.packages("data.table").
This give you a function that will do what you want. Here is a demo how to use it:

library(data.table)    # makes the transpose function available

col_names <- colnames(worldcup)      # keep track of original column names
wc_2 <- colMeans(worldcup)           # compute the means
wc_2 <- transpose(as.data.frame(wc_2))              # this gives you generic column names
colnames(wc_2) <- col_names          # reapply the column names

Or combining it with your example (not fully tested):

library(magrittr)    # to import the pipe operator: %>%

wc_2 <- worldcup %>%
  select(Time, Passes, Tackles, Saves) %>%
    colMeans() %>% as.data.frame() %>% transpose()       # you might need to put a dot (`.`) in the empty brackets to pass the argument before the pipe operator

then maybe add the names you would like:

colnames(wc_2) <- col_names

If you like the sound of the package, I recommend going through the mini intro, built in to the package:

install.packages("data.table")        # install it
library(data.table)                   # load it
example(data.table)                   # run the examples section of ?


First of all check this carefully. You'll find a simple dataset and some papers to review.

BUT as you want to start a simple learning project I recommend to not going through papers (which are obviously not basic) but try to build your own bayesian learner which is not so difficult.

I personally suggest Andrew Moore's lecture slides on Probabilistic Graphical Models which are freely available and you can learn from them simply and step by step.

If you need more detailed help just comment on this answer and I'll be glad to help :)

Enjoy baysian learning!


To be perfectly honest, most NoSQL databases are not very well suited to applications in big data. For the vast majority of all big data applications, the performance of MongoDB compared to a relational database like MySQL is significantly is poor enough to warrant staying away from something like MongoDB entirely.

With that said, there are a couple of really useful properties of NoSQL databases that certainly work in your favor when you're working with large data sets, though the chance of those benefits outweighing the generally poor performance of NoSQL compared to SQL for read-intensive operations (most similar to typical big data use cases) is low.

No Schema - If you're working with a lot of unstructured data, it might be hard to actually decide on and rigidly apply a schema. NoSQL databases in general are very supporting of this, and will allow you to insert schema-less documents on the fly, which is certainly not something an SQL database will support.
JSON - If you happen to be working with JSON-style documents instead of with CSV files, then you'll see a lot of advantage in using something like MongoDB for a database-layer. Generally the workflow savings don't outweigh the increased query-times though.
Ease of Use - I'm not saying that SQL databases are always hard to use, or that Cassandra is the easiest thing in the world to set up, but in general NoSQL databases are easier to set up and use than SQL databases. MongoDB is a particularly strong example of this, known for being one of the easiest database layers to use (outside of SQLite). SQL also deals with a lot of normalization and there's a large legacy of SQL best practices that just generally bogs down the development process.

Personally I might suggest you also check out graph databases such as Neo4j that show really good performance for certain types of queries if you're looking into picking out a backend for your data science applications.


Use correlation matrix scores as starting weights and bias inputs for neural network?

The way I see it is that NN can learn nonlinear features, so I wonder: does your correlation matrix on the raw data take into account nonlinear connections between features to see the full picture? Probably, it doesn't. As the topic is not about the single-layer nn (which would be an example of linear regression), the final connections (values of each weight) will be created by the contribution of many layers. Therefor it seems to me, that setting start weights on the proposed way wouldn't be high relevant.

Speaking about imbalancing, I'd recommend to experiment with loss functions. You can set higher penalty for mistakes on a certain class.


how to find the best parameters to solve a differential equation?

If nothing else is possible, you could try using a genetic algorithm.


How does class_weights work in RandomForestClassifier

Maybe try to encode your target values as binary. Then, this class_weight={0:1,1:2} should do the job. Now, class 0 has weight 1 and class 1 has weight 2.


TensorFlow1.15, multi-GPU-1-machine, how to set batch_size?

Tensorflow handles batches differently on distribution strategies if you're using Keras, Estimator, or custom training loops.

Since you are using TF1.15 Estimator with MirroredStrategy in one worker (1 machine), each replica (one per GPU) will receive a batch size of FLAGS.train_batch_size. So, if you have 4 GPUs, then the global batch size will be 4 * FLAGS.train_batch_size.

Here's the explanation:

  In Estimator, however, the user provides an input_fn and have full control over how they want their data to be distributed across workers and devices. We do not do automatic splitting of batch, nor automatically shard the data across different workers. The provided input_fn is called once per worker, thus giving one dataset per worker. Then one batch from that dataset is fed to one replica on that worker, thereby consuming N batches for N replicas on 1 worker. In other words, the dataset returned by the input_fn should provide batches of size PER_REPLICA_BATCH_SIZE. And the global batch size for a step can be obtained as PER_REPLICA_BATCH_SIZE * strategy.num_replicas_in_sync.

Source: TF1.X Distribution Strategy Notebook


How to split a dataset into train and test sets for time series (multiple step-multiple output forecasting)?

Assuming you have sales data for 36 month and you want to forecast for next 6 or 9
months. Lets create the train and test data at 70 and 30.

  train data= 36*70/100= 26(rounding value of 25.2)
  test data= 10(leftout sample)

Now we'll build a model on train data and forecast values on test data(those 10 points), to measure model performance we use MAE or MSE, always MAE or MSE values should be less. Once you model is doing good then forecast for future 6 to 9 months. i.e multiple output in one time prediction.


In most cases, you would use a file-storage solution such as Amazon S3 or Google Cloud and many others, which provide designated solutions for large object storage and retrieval.

You would then ideally want to update your code to retrieve the model directly from the file storage. Whether this download needs be done on every run or only once (storing the model locally for future runs), should be decided based on your specific needs.


How to add a new category to a existing trained deep learning model?

You cannot do that without re-training at least part of the model.

You will have to replace the existing output layer with a new output layer that has the desired number of neurons. That, of course, means that you will have to retrain at least the last layer or the last few layers. But you can freeze the weights of all the other layers.

In keras you can freeze the weights of a specific layer by setting its trainable property to False.


The questions of whether and why it's important, depends on the context.

For gradient boosted decision trees, for example, it is not important - these ML algorithms "don't care" about monotone transformations to the data; they just look for points to split it.
For linear predictors, for example, scaling can improve interpretability of the results. If you'd like to think of the magnitude of the coefficients as some indication of how much a feature is affecting outcome, then the features has to be scaled somehow to the same area.
For some predictors, in particular NNs, scaling, and in particular scaling to a particular range, can be important for technical reasons. Some of the layers use functions that effectively change only within some area (similar to the hyperbolic-family of functions), and if the features are too much out of the range, saturation can occur. If this happens, numerical derivatives will work badly, and the algorithm might not be able to converge to a good point.


What is "lag" in time series forecasting?

Lag features are target values from previous periods.

For example, if you would like to forecast the sales of a retail outlet in period $t$ you can use the sales of the previous month $t-1$ as a feature. That would be a lag of 1 and you could say it models some kind of momentum. But you could also apply a lag of 12 to model the sales of the same month a year ago (since retail sales are often seasonal depending on the format, category and SKU). Accordingly, it depends on the dataset which lags work best and looking at correlation is one way to select the lag values.

Have a look at this blog post for a simple explanation of lag features.


I had the exact same problem using the pytorch implementation, until I realised I didn't set the model in eval mode. Hence, dropout was still activated. I guess this is also the source of your undeterministic behavior.

Fix using pytorch:

model.eval()

Using tensorflow with an Estimator, ensure you call estimator.evaluate or estimator.predict during testing. If you exported the model in some ways, then you should check the tensorflow documentation.


How to use r2-score as a loss function in LightGBM?

$R^2$ is just a rescaling of mean squared error, the default loss function for LightGBM; so just run as usual.  (You could use another builtin loss (MAE or Huber loss?) instead in order to penalize outliers less.)


Machine learning algorithms for correct words formation from jumbled words

from itertools import permutations
import string
permutation_list = []
s = "BOX"
a = string.ascii_letters
p = permutations(s)

# Create a dictionary
d = []
for i in list(p):

    # Print only if not in dictionary
    if (i not in d):
        d.append(i)
        permutation_list.append(''.join(i))
        print(''.join(i))

import nltk
nltk.download('words')

from nltk.corpus import words
for word in permutation_list:
    print(word.lower() in words.words())

output -
BOX True
BXO False
OBX False
OXB False
XBO False
XOB False


Knowing Feature Importance from Sparse Matrix

You would have a map of your features from the TFIDF map.

column_names_from_text_features = vectorizer.vocabulary_
rev_dictionary = {v:k for k,v in vectorizer.vocabulary_.items()}
column_names_from_text_features = [v for k,v in rev_dictionary.items()]

Since you know the column names of your other features, the entire list of features you pass to XGBoost (after the scipy.hstack) could be

all_columns = column_names_from_text_features + other columns

(or depending on the order in which you horizontally stacked)

Now, once you run the XGBoost Model, you can use the plot_importance function for feature importance. Your code would look something like this:

from xgboost import XGBClassifier, plot_importance
fig, ax = plt.subplots(figsize=(15, 8))
plot_importance(<xgb-classifier>, max_num_features = 15, xlabel='F-score', ylabel='Features', ax=ax)
plt.show()

These features would be labeled fxxx, fyyy etc where xxx and yyy are the indices of the features passed to xgboost.

Using the all_columns constructed in the first part, you could map the features to in indices in the plot encoding.


So, overfitting occurs when the model is complex enough to fit very well with examples observed in the training data, such that the model is not able to generalise well over unseen instances (validation data).
Therefore, for overfitting, we expect the training F1 score to continually decrease, whilst the valid_1 F1 score increases.
Here, the plot shows that both training and validation F1-score has stabilised over epochs/iterations. Arguably though, we can see that valid_1 marginally increases as the training F1 score decreases. This can be indicative of (very mild) overfitting.


Multilabel Classification - increasing the confidence

It's hard to say without knowing the data.

Often times the classes are imbalanced. This could cause your problems.
So let's say that the "features are right" for 5% of your data and they belong to class $A$. You would expect the 5% to be classified correctly as $A$ almost all the time. But what about the remaining 95% in class $B$? The classifier learns that it would be often correct if the sample is classified as $B$. This leads to a low classification rate for class $A$.

You can tackle this by setting different miss-classification costs for the classes. Often you multiply the error rate by these costs to scale them accordingly. Some classifiers let you set this during training.

The standard case would be as follows:

$$C = \left( \begin{matrix}
0 & 1  \\
1 & 0  \\
\end{matrix} \right)$$

Meaning that correct classification (diagonal) is not punished and miss-classifying $A$ as $B$ and vice versa is punished in the same way. However, this assumes uniformly distributed classes. To account for the case explained above, you may set it the following way:

$$C = \left( \begin{matrix}
0 & 0.05  \\
0.95 & 0  \\
\end{matrix} \right)$$

Here i set it as the inverse of the class distribution. However, you may try different settings.


Should I put time on my Vanilla ANN for classifying MNIST Dataset

Some quick notes:

A mini-batch size of 4096 is rather large, try training with sizes like 16, 32, or even 1 to see what happens.

Your test accuracy seems to be around random since there are 9/10 classes to predict. Does your model predict randomly (i.e. every class gets a similar score) or is it just wrong often?

is it necessary to implement it from scratch?

Necessary for who? Unless it's required for your coursework, nobody is forcing you to implement this from scratch.

P.S.: Does it create any problem if a DL model is not written using OOP?

If an implementation is correct, it's correct. Whether it's spaghetti code, whether it's a single file, whether it's beautiful, tested, and reviewed OOP software.

OOP is there to help human write more maintainable, robust, and readable software, but under the hood, the computer really couldn't care less.


Using feature importance to decet latent variables and grouping

You might be able to segmentate your items into groups that share a beer consumption trait by focusing on those variables. It's a good idea if you started with a lot of columns . Of course, expect more meaningful groups if your model is simpler rather than complex - a small decision tree rather than a random forrest, for example.

Why don't you do some descriptive statistics on these columns, dividing first your data in the target variable? The visualization of the different patterns can be quite inspiring.

Also, test your intuition: cluster your data based solely on these columns and check whether each cluster is significantly biased towards one of the target values.

Something that you can also try is using the notion of proximity between data points you get from a random forrest model in your clustering algorithm.


Which model can solve the "sequence demand" problem?

Any regression algorithm can address this problem, to a greater or lesser degree of success- the important thing is in how you form your data.

One reasonable form to use would be a structure with one row for each day and features describing the arrival of trucks over an appropriate timeframe- say, over the past 14 days. Each row would then also have the demand for the day associated with it as the regression target.

You would then be able to train a model that predicts the demand for a given day provided the appropriate truck arrival history.


Advice on dealing with very large datasets - HDF5, Python

One option is to switch to pytables which is built on top of HDF5 and is designed to mitigate many of these issues.


It could possibly be caused by exploding gradients, try using gradient clipping to see if the loss is still displayed as nan. For example:

from keras import optimizers

optimizer = optimizers.Adam(clipvalue=0.5)
regressor.compile(optimizer=optimizer, loss='mean_squared_error')


This tutorial explains in a nice and simple way how to create a data generator which you can pass to your Keras model to train using fit_generator(). Two things to keep in mind about Keras generators, in order to be compatible with Tensorflow 2.x requirements:

Your generator should inherit from keras.utils.Sequence, which allows for internal parallelization of training
Populate the __getitem__() and __len__() methods. The first makes your generator iterable and the second returns its length using the builtin len command on the generator object. Both are required by Keras.


EM clustering with missing and misspelling data

Convert your key-Value Table to a table in wide format (1 row per person) , then calculate the distance matrix using the Jaccard Distance (which can convert categorical values into numeric values)

library(dummies)
dummy_dat <- dummy.data.frame(my_data)

jaccard_dist <- dist(dummy_dat, method = "binary")

#  Distance Matrix
jaccard_dist


How to solve XNOR problem with neural network?

Combine a NN for x1 AND x2 [your a1] with a NN for (NOT x1) AND (NOT x2) [your a2] to get the XNOR. You end up with inputs +1, x1, and x2 going to your hidden neurons a1 and a2, then +1, a1, and a2 into your output neuron.


Why do people import weights for DenseNet when Keras includes them?

Because it must have been a kernel only competition, with:  'the following conditions must be met:"Internet must be turned off".'

When we use  " weights='imagenet' " , the notebook will connect to internet to download the weights. Since the user cannot connect to internet for that particular competition it is better to upload the weights as data source, and then import it for use.


Date and time extraction from Excel file in Python or Pandas

Pandas can parse most dates formats using
import pandas as pd
pd.to_datetime(df["name of your date column"])

You can also cast the desired column to datetime64 which is Numpy dtype
df = df.astype({"column name": "datetime64"})


What is the relationship between Markov Random Fields and Conditional Random Fields?

Conditinal Random Fields (CRFs) are a special case of Markov Random Fields (MRFs).

  1.5.4 Conditional Random Field

  A Conditional Random Field (CRF) is a form of MRF that defines a posterior for
  variables x given data z, as with the hidden MRF above. Unlike the hidden MRF,
  however, the factorization into the data distribution P (x|z) and the prior P
  (x) is not made explicit [288]. This allows complex dependencies of x on z to
  be written directly in the posterior distribution, without the factorization
  being made explicit. (Given P (x|z), such factorizations always exist,
  however—infinitely many of them, in fact—so there is no suggestion that the CRF
  is more general than the hidden MRF, only that it may be more convenient to
  deal with.)

Source: Blake, Kohli and Rother: Markov random fields for vision and image processing. 2011.

  A conditional random field or CRF (Lafferty et al. 2001), sometimes a
  discriminative random field (Kumar and Hebert 2003), is just a version of an
  MRF where all the clique potentials are conditioned on input features:
  [...]

  The advantage of a CRF over an MRF is analogous to the advantage of a discriminative
  classifier over a generative classifier (see Section 8.6), namely, we don’t need to “waste resources”
  modeling things that we always observe. [...]

  The disadvantage of CRFs over MRFs is that they require labeled training data, and they are slower to train[...]

Source: Kevin P. Murphy: Machine Learning: A Probabilistic Perspective


Transform test data when using a persistent model

In the same way that you use joblib to persist your saved model, you should persist the transformers that you use in the pipeline too. So for example if you're using StandardScaler() and OneHotEncoder(), those also need to be joblib.dump()ed so you can import them into your prediction script.

The simplest way to achieve this is to add both transformers and estimator into a Scikit Learn pipeline and use joblib to persist that.


Though translations are not on a word-word basis, nonetheless there is a strong merit in retaining the sequence of the words at the encoder side. There is a huge penalty to pay for this as this causes serialization, but in spite of it LSTMs and GRU's became so popular so one can imagine that sequence order matters. After the encoder is done processing in sequence, the final state that is generated is sort of a sentence embedding and contains the essence of the sentence. This is a good starting point for the decoder to pick and use. Unlike what you have assumed, the model does not look 'only' at the context generated by the attention layer to make predictions. It also uses the prev LSTM state along with the context (and the last translated word to make the next prediction). If you trace back the prev LSTM state right to the beginning of the decoder, you can see it has its origins in the final LSTM state of the encoder.
Having said that, your question still is very pertinent. The concept of Attention is so powerful that with self attention and multi-head attention, it is now possible to do away with the RNN at the encoder end altogether and just use the representation generated purely by the process of 'Attention'. But even here the authors of the landmark paper - Attention is all you need - add in a small hack to retain the order of the sequence of the words of the input sentence. This seems to improve the model in better prediction.


Comparing training and validation data set Root MSE for a best subset regression?

Im a little late, but better late than never.  It looks like your line where you find the coefficients:

regfit.exh=regsubsets(HPV~. -Model.Types..code.-Year..code.,data=Mydata, nvmax=NULL,force.in = NULL, force.out = NULL, method="exhaustive")

should have data=Mydata[train,] instead of data=Mydata.

Your model had already seen your test samples, so the validation error is not an accurate assessment.


From where should I start Machine Learning?

You can start learning machine learning by attending the lectures of Andrew Ng at Coursera. This course helps you to develop basics as well as advanced concepts in machine learning.
Following is the link to the course:
Machine Learning by Andrew Ng
Start learning Python for implementing algorithms related to machine learning.


As you probably know, naive here implies that the "fields" are independent. So your question boils down to does correlation imply dependence. Yes, it does. See here.

https://stats.stackexchange.com/questions/113417/does-non-zero-correlation-imply-dependence

So, if your features show correlation then this will have an adverse effect on the naive assumption. Despite this fact, Naive Bayes has been shown to be robust against this assumption. If your model still suffers from this, however, you could consider transforming the space to be independent with methods such as PCA.


So, the question is asking for advice on what to do next, given the model's performance.
Firstly, in terms of confusion matrices, it is often useful to display proportions, rather than number of examples. It makes it easier for us to gauge where the error is occurring.
Looking at the most recent confusion matrix, it is clear that a lot of the examples are classified into classes 2 and 3. This could be due to a couple of things, potentially class imbalance, especially since the class weights is commented out in the code. Also, looking that the loss over epochs, it looks like the loss could continue to decline if you gave the model more epochs (unless you put a callback on it to prevent it from continuing training, which doesn't look to be the case).
Try that out and edit the post accordingly with the updated results!


Analysis of railway data - Detecting outliers

There are a number of directions you can take, and there are a number of questions related to this topic (anamoly-detection-for-transaction-data).  One question to answer first is whether the data is seasonal or not.

You already suggested a valid approach.  Determine which areas are correlated to low and high speeds and than determine when a speed differs.  I think it's best to start with a heuristic approach and than try more complicated methods to try to improve.  If you want to use an SVM, a one-class SVM may work well (one-class SVM)


How to best forecast simple binary data?

Do some feature engineering. As they told you in the comments, try
to see if the event is more likely to happen during weekends or
weekdays, during certain months, during a certain season or quarter,
etc. To read some more on this topic.

https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/
Explore different machine learning models. I suggest to start with a Random Forest.
Check your model accuracy (confusion matrix).
Iterate. Try other features, other models...

Just to get you up and running, you can take a look at the Titanic dataset in Kaggle. It will help you understand a bit more of the problem you are trying to solve.


What is "$D$ equal to $\frac{1}{2}$ everywhere" in original GAN paper

$\mathbf{D}$ is the Discriminator network, but it really is being modelled as a probability distribution - the discriminator:

  estimates the probability that a sample came from the training data rather than G

where $\mathbf{G}$ is the Generator network.

In the context of a two-player min-max game, the authors say a unique solution exists in the case that the Generator perfectly learns the underlying distribution of the dataset. When this point is reached, the Discriminator's estimates will not be better than random guesses.

This means that the accuracy of the estimates will just be 50% (assuming a 50-50 split of real versus generated samples), so it doesn't know if the Generator created the image (which would correspond to an estimate of 100%) or definitely did not create that image (an estimate of 0%).

So $D = 0.5$ everywhere means the Generator has perfectly learned the data and can produce images that are indistinguishable from the real images... so the Discriminator is basically clueless and is flipping a coin.


I summarize your questions and then try to answer under each bullet point:

How to remove punctuation marks (e.g. # for hashtags which is used in social media)

The first goto is a regular expression that is used in data preprocessing very frequently. But if you are looking for all your punctuation to be removed from the text you can use one of these two approaches:
import string
sentence = "hi; how*, @are^ you? wow!!!"
sentence = sentence.translate(sentence.maketrans('', '', string.punctuation))
print(sentence)

output: 'hi how are you wow'
or use a regular expression, example:
import re
s = "string. With. Punctuation? 3.2"
s = re.sub(r'[^\w\s]','',s)
print(s)

output: 'string With Punctuation 32'

What are the possible python library for data cleaning?

generally, NLTK and SciPy are very handly. But for specific purposes, other libraries also exist. For example library contractions, inflect.

Which libraries are efficient in data cleaning when using pandas dataframes?

You can apply any function from any library to your pandas dataframe using the apply method.
Here is an example:
mport pandas as pd
s = pd.read_csv("stock.csv", squeeze = True)

# adding 5 to each value
new = s.apply(lambda num : num + 5)

code source


Getting sentence embeddings with sentence_transformers

The encode method only works with single sentences as strings, i.e., you need to call it for each sentence independently:
embeddings = [model.encode(s) for s in sentences]


Does modeling with Random Forests require cross-validation?

By default random forest picks up 2/3rd data for training and rest for testing for regression and almost 70% data for training and rest for testing during classification.By principle since it randomizes the variable selection during each tree split it's not prone to overfit unlike other models.However if you want to use CV using nfolds in sklearn you can still use the concept of hold out set such as oob_score(out of bag)=True which shows model performance with or without using CV.
So in a nutshell using oob_score=True with or without nfolds can itself tell whether using CV is good for your data.Generally if your target is following a certain distribution and you don't have much observation data with you then using CV will not give much improvement.


Augmenting the Cost Function with a Gaussian Prior

Let's look at the example of linear regression. Instead of deriving it from solving the normal equations, we can motivate it by thinking about it as finding the conditional distribution $P(y|x)$. Let's assume that this distribution follows a Gaussian distribution with fixed variance $\sigma^2$ and mean $\hat{y}(x,w)$ and weights $w$. Assuming that samples are i.i.d. we can easily show that applying maximum-likelihood gives the same values of $w$ than minimizing the mean squared error. See example 5.5.1 for a mathematical explanation.

If we additionally assume a prior Gaussian distribution for $P(w)$ (w.l.o.g. we can assume unit variance $I$), we can now show that the posterior $P(w | x,y)$  of $w$ is also Gaussian distributed (that is because the prior is "conjugate") with variance $\frac{1}{\alpha} I$ and we recover linear regression with weight decay $\alpha w^T w=\alpha \lVert w \rVert^2_2$  (i.e. $L^2$ or Tikhonov regularization).

Intuitively, a Gaussian distribution's probability mass is centered around the mean (assumed here to be zero). If we assume such a distribution for the coefficients in the linear regression (as our prior knowledge), linear regression will favor coefficients close to zero.

All of this is nicely explained here (chapter 5) or here (chapter 18).


Using keras tokenizer on Spark DataFrame. Column is not iterable

I'm not sure that one can use Keras on Spark to process data in parallel (using multiple workers).

But if your data can fit in RAM on a single node, that you can easily create a Pandas DataFrame from a Spark DataFrame and pass that DF to Keras. You can also convert result back to Spark DF if you need.

Spark DF --> Pandas DF:

pdf = df.toPandas() # df - is a Spark DataFrame

Pandas DF --> Spark DF:

sdf = spark.createDataFrame(pdf)


tuning a convolution neural net, sample size

If you use pre-trained weights, you need significantly lesser data as the initial layers have already learned from a ton of data and you just need to fine tune the later ones.

What you said is not true, you can train on CIFAR10 and get 90%+ and that is not 100k+. It depends on the complexity of the data and how similar the features are. If they are easily Seperable -less data. If the disctintions are harder then the model needs a lot of examples to figure out which of features are seperate.

I would say you could IF you sample was representive of the population.


UPDDATED with second resource

In general as I understand this is question about 'data quality'. There are lot of manuals, i like suggestions from this one - this is lets say semantic level:

SIX CORE DATA QUALITY DIMENSIONS
    COMPLETENESS
    UNIQUENESS
    TIMELINESS
    VALIDITY
    ACCURACY
    CONSISTENCY

This M&E Attribute: Data Quality Assessment checklist is more detailed.

Then, of course, you should go deeper, to the syntax level.


What is the difference between feature generation and feature extraction?

Feature Generation -- This is the process of taking raw, unstructured data and defining features (i.e. variables) for potential use in your statistical analysis. For instance, in the case of text mining you may begin with a raw log of thousands of text messages (e.g. SMS, email, social network messages, etc) and generate features by removing low-value words (i.e. stopwords), using certain size blocks of words (i.e. n-grams) or applying other rules.
Feature Extraction -- After generating features, it is often necessary to test transformations of the original features and select a subset of this pool of potential original and derived features for use in your model (i.e. feature extraction and selection). Testing derived values is a common step because the data may contain important information which has a non-linear pattern or relationship with your outcome, thus the importance of the data element may only be apparent in its transformed state (e.g. higher order derivatives). Using too many features can result in multicollinearity or otherwise confound statistical models, whereas extracting the minimum number of features to suit the purpose of your analysis follows the principle of parsimony.
Enhancing your feature space in this way is often a necessary step in classification of images or other data objects because the raw feature space is typically filled with an overwhelming amount of unstructured and irrelevant data that comprises what's often referred to as "noise" in the paradigm of a "signal" and "noise" (which is to say that some data has predictive value and other data does not). By enhancing the feature space you can better identify the important data which has predictive or other value in your analysis (i.e. the "signal") while removing confounding information (i.e. "noise").


Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question

The metrics you calculate are of two types, metrics that depict the entire prediction model you have built like accuracy which will be same in both the cases of your pseudo code. While the others like precision says how precise are you in explaining particular class of interest (accuracy can also be expressed this way in multi-class classification, see the diagram). This score depends on which class you had selected as a positive one. If you put positive class as face of your model, then it is called ppv or precision and npv, if vice versa.

Coming to the multi-class classification, the core definition holds the same. Now the matrix will be n x n(n being number of classes). The sample matrix looks likes this.

The diagonal elements explains the number of 1 class's predicted as 1. Now there are n precision values for each class. Precision for class 1 is how many values were truly predicted as 1 divided by how many are predicted as 1 (FP's included), which is the sum of the first column.

Not but not the least if you adamantly want a precision like metrics for the entire model, you got micro & macro averaging methods, which are helpful in giving a combined metric. This blog post explains it pretty well. Hope this clears something.


Yarn service parameter for pseudodistributed

The mapreduce_shuffle in this config file is part of Plugable Shuffle and Sort.  Shuffle and Sort are what connect the mappers to the reducers. A nice graphical representation of this is the following (shuffle is called "copy" in this figure):

The Hadoop arcitecture alllows these steps to be customized:

  The pluggable shuffle and pluggable sort capabilities allow replacing the built in shuffle and sort logic with alternate implementations. Example use cases for this are: using a different application protocol other than HTTP such as RDMA for shuffling data from the Map nodes to the Reducer nodes; or replacing the sort logic with custom algorithms that enable Hash aggregation and Limit-N query.

The mapreduce_shuffle parameter is mentioned in the Hadoop Docs, and is the default value for this configuration.


Lower performace with same script on google cloud vs laptop

After some experimentation I found out that reducing the amount of jobs actually increases the speed. I'm now running 8 machines with a single core where each machine is running faster than the 8 core machine.
I either did something wrong in my code or this library is not built for multi core processing..


Keras: do `sample weights` take part in the derivatives

sample_weights affect the total_loss, which is used to calculate the gradients.

  Because the loss function is not actually defined based on the sample weights, its rather passed in (as an argument) to the fit function in Keras

I think that what you're missing is that the loss function actually returns a list of loss values (one per sample in the batch). This is precisely because during fit, it is multiplied by the sample_weight of that batch.

Please note that although "class based" losses support reduction (over the batch), the Loss base class implements the weighting during reduction.


Unexpected results from scikit learn regression decision tree

This is the visualization for the inducted tree from your data:

As you can see, it can never predict [[1.5, 1.5]]. Your data is just too pure, the inducted tree will fit it perfectly, having all leaf nodes with only one sample. If a leaf node has more than one sample in it, then the predicted value is the mean of the y values for those samples.

Furthermore, trees are not the best model to induct the identity function ($y = f(x) = x$). Trees work by creating rules and dividing the feature space orthogonally (i.e. only vertical or horizontal lines in the feature space).

In other words, your model didn't learn that f(1.5, 1.5) = (1.5, 1.5) because a) there's not enough data, b) trees are not good at doing that anyway.


If I understand correctly, you want to create features? there are a few ways to do this. I will first talk about Label Encoding and Hot Label Encoding(link is the first I found on Google). This depends on the model, I will explain below.

Since we want to use categorical data in a model, most models behave better with numerical data. So let's convert categorical -> numeric:

import pandas as pd
from sklearn.preprocessing import LabelEncoder

df = pd.DataFrame({'user':['adam','adam','susie','jane'],
                   'browser':['chrome','firefox','chrome','opera']})

   browser   user
0   chrome   adam
1  firefox   adam
2   chrome  susie
3    opera   jane

label_encoder = LabelEncoder()
# convert browser to numeric
df['browser_encoded'] = label_encoder.fit_transform(df['browser'])

   browser   user  browser_encoded
0   chrome   adam                0
1  firefox   adam                1
2   chrome  susie                0
3    opera   jane                2

In the "real world" you would drop browser column as its now encoded. I left it to show you what's happening.

Now notice how our data is now numeric. But now there is a problem. If you're using something like multiple linear regression, you've just assigned a higher weight to the Opera browser since it's a higher number. (2>1 and 2>0). But since this is categorical, and Opera isn't "weightier" than other browsers in our model, we need to solve this. This is where One Hot Encoding comes in to play.

pd.concat([df,pd.get_dummies(df['browser'])], axis=1)

   browser   user  browser_encoded  chrome  firefox  opera
0   chrome   adam                0       1        0      0
1  firefox   adam                1       0        1      0
2   chrome  susie                0       1        0      0
3    opera   jane                2       0        0      1

Now we have our data in columns. the 1 value appears when someone has used that browser for that record. See the first record? where chrome column is 1, because adam used 'chrome' in the browser column?.

There is one last thing when it comes to label encoding! You don't want to fall in the dummy variable trap with certain models. The basic logic is: If you know the browser is not Chrome or Firefox, it must be Opera. If you know the browser is Chrome, you know that it is not Firefox or Opera. So you only need N-1. If you use all the columns, your model may not perform well. You need to drop only one of the columns. you can do that with drop_first=True

pd.concat([df,pd.get_dummies(df['browser'], drop_first=True)], axis=1)
   browser   user  browser_encoded  firefox  opera
0   chrome   adam                0        0      0
1  firefox   adam                1        1      0
2   chrome  susie                0        0      0
3    opera   jane                2        0      1

Another way to create features is to make them yourself. You mentioned that a user can use more than one browser. I'm making something up for the sake of a simple example - but maybe the number of browsers they use can be a feature? You can create one like so:

df['num_of_browsers_used'] = df.groupby('user', as_index=False).transform('nunique')['browser']
    browser user    browser_encoded num_of_browsers_used
0   chrome  adam    0   2
1   firefox adam    1   2
2   chrome  susie   0   1
3   opera   jane    2   1


How do I print full date in the x axis of the line plot here?

For better control over the x-axis formatting, you can use the matplotlib.dates methods. In your case, MonthLocator and DateFormatter could be of interest. These can be used to adjust the x-axis as follows:
import matplotlib.dates as mdates

def time_series(start, end):
    time_series_df  = df.loc[(df['Date'] >= start) & (df['Date'] <= end), ['Date', 'Value']]
    time_series_df = time_series_df.set_index('Date')

    fig, ax = plt.subplots(1)
    ax.plot(time_series_df)

    # Adjust the x-axis
    ax.xaxis.set_major_locator(mdates.MonthLocator(interval=3)) # Month intervals
    ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m-%d')) # date formatting
    # automaticall set font and rotation for date tick labels
    plt.gcf().autofmt_xdate()

    plt.xlabel('Time')
    plt.ylabel('PM2.5 Value')
    plt.title('PM2.5 Time Series')
    plt.show()

Note the interval=3 argument for MonthLocator. This is the frequency of the x-axis ticks.
Resulting plot:

Dataframe use for the plot above:
df = pd.DataFrame({'Date': pd.date_range(start='2014-01-01', end='2018-12-31'), 'Value': np.random.uniform(size=1826)})
df['Date'] = pd.to_datetime(df['Date'])


I'll make a few observations that will hopefully help.

I would remove the dropout layer until you have evidence of overfitting on the data.  A dropout layer is generally used to make a model more generalized which is not your problem in this case and it may be hurting your model.
The images may not contain enough information to distinguish between the classes.  Can an expert assign a class label by looking at the images? If not, it may be that it is just not possible from the images.
Your accuracy scores imply that the model is just guessing since a random chance classifier would be expected to perform as well. However, it is curious that you seem to get reasonable accuracy on one class.  You may find that you need multiple models configured as one-vs-the-rest binary classifiers. Each model can be uniquely configured and trained to specialize on one class.  After the training is complete, which could be time consuming, prediction should be efficient.
Lastly, do you think that the model has stopped learning by the time you stop training?  Intuitively, I would anticipate the need for tens of thousands of epochs, and potentially millions, before seeing a really refined model.  100 epochs seems like it would be just starting to learn. Because of this cost, I would recommend you configure your model so you can continue training from a stopping point.

And by all means, keep trying different configurations of your layers.  One thing to remember though is that each time you use a convolutional layer you are losing information.  You need to consider how much information you are losing with each convolutional layer.


Help me choose a Data Science book in Python

I am reading Hands-On Machine Learning with Scikit-Learn and TensorFlow, its a great book (1/2 way thru).  However it uses Scikit as per the title and some of the mechanics inside Sckit are a black box (not explained in depth).  The general concepts however are still quite well written.

The 2nd half of the book is about TensorFlow so again, perhaps not what you want if you're focused on PyTorch ?


How would you visualize data that comes in the millions of records?

Holoviews visual library can handle very large data http://holoviews.org/ http://holoviews.org/user_guide/Large_Data.html


How is GPT able to handle large vocabularies?

GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token.
BPE tries to find the pieces of words that are most reusable. BPE also keeps character subwords (e.g. "a", "W").
The subword vocabulary used by GPT-2 is 50k tokens big. You can take a look at the vocabulary here. There, you can see that token #12175 is "retro" and token #11147 is "fit", so when tokenizing word "retrofit", you would probably get the 2 subword tokens #68 and #12541. When tokenizing a low-frequency word (i.e. that did not appear in the BPE training data), you probably end up with small subwords tokenization, e.g. "Kilimanjaro" --> "Kil", "iman", "jar", "o".
Using a word-level vocabulary is very troublesome, because:

the amount of existing word surface forms exceeds the size that it is manageable for a neural net, even for a morphologically simple language like English. For morphologically-rich fusional languages and, especially, agglutinative languages, using word-level vocabularies is much more inconvenient.
language is "open", in the sense that new words can be created, either by stitching together existing words (e.g. manspreading), or by making up new words altogether (e.g. bazinga).
the less frequently a token appears in the training data, the less our net will learn to use it. If we have many many different tokens, the frequency of appearance of each token in the training data will be very low (i.e. data scarcity), so our net will not learn properly.

Therefore, subword vocabularies are the norm nowadays.


Finding both count and average of a column in R data.table, after group by

If you want to get but the sum and length of id with data.table, you can do it like this:

dt[,.(total=sum(id), n=length(id)), by=cid]

You could even write this in shorter format with the .N operator of data.table:

dt[,.(total=sum(id), .N), by=cid]


That is a common case on image and audio processing, you need to find a way in which dimensions stay the same, such as normalizing per channel.
If you have a 1D vector of features, taking mean and variance of all variables will end up normalizing it in a way, it works in Computer Vision like a charm. It is also a way to reduce the space cost of your normalizing algorithm.


How to build a symptom checker and medical diagnose chat bot

We built the symptom checker with AI API endlessmedical.
We provide free to download for any use source code of entire symptom checker here:
https://endlessmedical.com/chatbot-like-symptom-checker-free-source-code-download/
Have fun!


Can You Purposely Bias A Clustering Model?

In constrained clustering you can provide examples of objects that should, or that must not, be in the same cluster.

This can be used, e.g., for model selection: run several times and return the result with the fewest violations. Or to guide cluster extraction from a hierarchical result.