Determining correlated product categories using store purchase history

This is classic market basket analysis.

Clustering is the weong tool, you want frequent item set mining and association rules instead.

https://en.wikipedia.org/wiki/Association_rule_learning


The are some techniques to choose the number of clusters K.

The most common ones are The Elbow Method and The Silhouette Method.

Elbow Method

In this method, you calculate a score function with different values for K. You can use the Hamming distance like you proposed, or other scores, like dispersion.

Then, you plot them and where the function creates "an elbow" you choose the value for K.

Silhouette Method

This method measure the distance from points in one cluster to the other clusters. Then visually you have silhouette plots that let you choose K.

Observe:

K=2, silhouette of similar heights but with different sizes. So, potential candidate.

K=3, silhouettes of different heights. So, bad candidate.

K=4, silhouette of similar heights and sizes. Best candidate.


These are different for a valid reason
The standalone model is overfitting. You can see that with r2score of 84 Vs 22.
The reason for that is, the standalone model goes to full depth and hence overfit the train data and badly fit the test data.

max_depthint, default=None
The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.

While with CV, you have parm for max_depth. Hence bad but Truth.


Tuning a classifier for high precision, with no regard for recall

To make your learner cost sensitive, you can increase your training data by more "no" instances. If there are 10 times more "no" instance in your training data, the errors on "no's" will hurt much more and your learner will come up with a dicision scheme which will be biased that way. The variance on the "yes'es" will be lowered on the other hand (pay attention on overfitting here).
After the training, you use your original data for testing and you should get good results here.


Maybe it is a bit too much just for this use case, but I have good experience with airflow.

It is an Apache project and quite helpful to automate some stuff.

Otherwise there are a lot of commercial platform helping you. Just google for data science platform.


Question about (Python/Orange) Apriori associative algorithm

If Python 3 is an option, Orange 3 features FP-growth in its Orange3-Associate add-on:

pip3 install Orange3-Associate

It's well documented: http://orange3-associate.readthedocs.org/

Apriori due to its slowness and resource requirements, as you'd noticed, really isn't an algorithm to consider anymore.


What is the difference between parameters & cooficients in Machine learning?

When talking about coefficients, we usually refer to some fixed numbers or constants, e.g. learning rate coefficient is 0.001 and it will not be changed during training. Parameters are usually changing, e.g. model is training to adjust right parameters. However, there are no strict borders between these two notions.


train Neural Network with SGD and see that it overfits data.

Of the options you listed, I would try adding a dropout layer first.

Another option you did not mention, but may, or may not be feasible, is to get more training data.  Often overfitting is a result of not having enough training data.


Other options would be to...

Compare similar text sequences
Compare similar string sequences
Use fuzzy matching

Fuzzy Matching:
library(fuzzyjoin)
# https://stackoverflow.com/questions/26405895/how-can-i-match-fuzzy-match-strings-from-two-datasets

a <- data.frame(name = c('Ace Co', 'Bayes', 'asd', 'Bcy', 'Baes', 'Bays'),
                price = c(10, 13, 2, 1, 15, 1))
b <- data.frame(name = c('Ace Co.', 'Bayes Inc.', 'asdf'),
                qty = c(9, 99, 10))

# Find matches
stringdist_join(a, b,
                by = "name",
                mode = "left",
                ignore_case = FALSE,
                method = "jw",
                max_dist = 99,
                distance_col = "dist"
) %>%
  group_by(name.x) %>%
  top_n(1, -dist)

This gives you a "distance" between one and another word. So if you know the real product name, you might be able to find the erroneous names via the distance.
# A tibble: 6 x 5
# Groups:   name.x [6]
  name.x price name.y       qty   dist
  <fct>  <dbl> <fct>      <dbl>  <dbl>
1 Ace Co    10 Ace Co.        9 0.0476
2 Bayes     13 Bayes Inc.    99 0.167
3 asd        2 asdf          10 0.0833
4 Bcy        1 Bayes Inc.    99 0.378
5 Baes      15 Bayes Inc.    99 0.2
6 Bays       1 Bayes Inc.    99 0.2

Alternatively, you could look into similarity in string sequences:
If your product names are not to common from their string sequences and the erroneous names get only part of the name wrong, you could try something like:
library(dplyr)
library(tidytext)
library(fuzzyjoin)
library(tokenizers)

##############################
# Compare text sequences

text1=as.character("Hi my name is Bixi and I like cycling a lot. It is just great!")
mytext1=data_frame(text1)

text2=as.character("Hi my name is Lissi and I'm good in swimming. It is just great!")
mytext2=data_frame(text2)

ngram1 = unnest_tokens(mytext1, ngram, text1, token = "ngrams", n = 4)
ngram2 = unnest_tokens(mytext2, ngram, text2, token = "ngrams", n = 4)

# Find matching sequence(s)
semi_join(ngram1,ngram2)

##############################
# Compare sequences of single letters

ngram3=tokenize_character_shingles(mytext1$text1, n = 10, n_min = 10, strip_non_alphanum = FALSE)
ngram4=tokenize_character_shingles(mytext2$text2, n = 10, n_min = 10, strip_non_alphanum = FALSE)

ngram3=as.data.frame(ngram3)
ngram4=as.data.frame(ngram4)

# Find matching sequences of single letters
semi_join(ngram3,ngram4)

You may have a look at my Github. There are some more related options/solutions.


removing words based on a predefined vector

texts <- c("This is the first document.",
       "Is this a text?",
       "This is the second file.",
       "This is the third text.",
       "File is not this.")

test_stopword <- as.data.frame(texts)
ordinal_stopwords  <- c("first","primary","second","secondary","third")

(newdata <- as.data.frame(sapply(texts, function(x)   gsub(paste(ordinal_stopwords, collapse = '|'), '', x))))

The output is getting skewed when added in a code block (maybe a bug in SE). But, you would get the desired output.


how do the number of classes in an object detection model affect accuracy?

The general consensus in machine learning problems is that it becomes tougher to get higher accuracy results when there is more data with more class splits. The simplest of examples would be cifar 10 and cifar 100. While they are practically the same models tend to vary very differently with respect to the efficiency.
The moment more classes are added, there is more variability under the final output. I am not aware of any soft limit though few papers might be there which focus on this. If you take a model A and train it on data with 10 classes and 50 classes, it is bound to perform better on the 10 classes provided there is an ideal set-up in both terms.
I have not come across any benchmark which says which is the ideal number of classes. Importing many models takes a lot of computational power therefore the question of many to one models varies mostly on the cost rather than any soft limit of the performance.


Select best answer from several existing ones for a question

This problem is multiple choice answering question. I can see you have already tried gensim, doc2vec etc. You can try pytorch based transformer solution. Here is the link: multiple-choice . You can create your data in swag format and remove --do_train in below code for prediction on your dataset.

It has been trained on swag dataset and has given decent accuracy.

If it works fine for you then good, else you would like to finetune it. For finetuning --do_train should be mentioned below:

#training on 4 tesla V100(16GB) GPUS
export SWAG_DIR=/path/to/swag_data_dir
python ./examples/run_multiple_choice.py \
--model_type roberta \
--task_name swag \
--model_name_or_path roberta-base \
--do_train \
--do_eval \
--do_lower_case \
--data_dir $SWAG_DIR \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--max_seq_length 80 \
--output_dir models_bert/swag_base \
--per_gpu_eval_batch_size=16 \
--per_gpu_train_batch_size=16 \
--gradient_accumulation_steps 2 \
--overwrite_output


Have a look at Deep Walk.  It's essentially word-to-vec applied to "sentences" created from a graph (nodes and edges).  If you can express your data as a graph, you can apply it.


You could consider PowerBI. It offers a couple of useful licensing scenarios (bundled with Office 365 etc.) and also has connectors to source data that already works well with SQL Server.

You did not say if SQL Server is running on premises or in the cloud, but PowerBI should work in both cases.


reliability of human-level evaluation of the interpretability quality of a model

This is specifically for interpretability of outcomes, i.e. a task where non-expert humans outperform machines.
There is a problem in collecting labels in machine learning, whereby labelling datasets is very expensive and time consuming (due to size of datasets & cost of experts' time).
So it's less about trust, its more about practicality. Consider hiring a data scientist to develop an algorithm to automatically label a dataset based of expert heuristics (e.g. "label the data as cancerous if it looks red"), it might take 6 months to collect data, plan, develop & test - therefore for certain use-cases hiring 10 non-experts and telling them the heuristic might be cheaper and faster.
The book uses an example "show a user different explanations and the human would choose the best." in the context of radiology, it could be something like:
"Look at the images of the patient, and compare it to this dictionary of images and diagnoses, combine multiple sources and then report what the diagnosis is"
Of course if you have an algorithm which outperforms non-experts, you might just want some expert labels to validate your algorithm, and forget the non-experts.


I cannot recommend highly enough this online book on neural networks.

The tricky part about neural networks is the stuff you mentioned -- what value to use for the learning rate, what topology to use for the network, etc etc -- we call these things hyperparameters to distinguish from parameters which are estimated by the optimization process.

This big section of the aforementioned book covers exactly that. I don't know if you'll find what you want though. The reason is because there is no silver bullet.


In R, Boruta relies on the ranger implementation of random forest. So:

Converting input variables from categorical to numeric is not necessary.
You will need to address NA values prior to running the algorithm.

Be aware that Boruta can be very slow!


You should pass X and Y collectively to the ImageDataGenerator.flow() method.

Please refer to this answer in case you are looking for a multi-output classification model using ImageDataGenerator in Keras.

https://datascience.stackexchange.com/a/75034/98109


The problem is that your error is accumulating and diverging. In other words, a small error in the first prediction is leading to a larger error in your second prediction, which is leading to an even larger error in your third prediction, and so on.

With that said, this LSTM example seems to do well predicting multiple future values for a sine wave.


Multi-dimentional and multivariate Time-Series forecast (RNN/LSTM) Keras

The input shape for an LSTM must be (num_samples, num_time_steps, num_features). In your example case, combining both cities as input, num_features will be 2x3=6.

If you lump all your 365 time steps into one sample, then the first dimension will be 1 - one single sample! You can also do sanity check by using the total number of data points. You have 2 cities, each with 365 time-steps and 3 features: 2x365x3= 2190 . This is obviously the same as 1x365x6 (as I said above) - so it would be a possibility (Keras will run) - but it obviously won't learn to generalise at all, only giving it one sample.

Have a look at this relevant question, which I recently answered. There I speak a little about using a rolling window (check the comments of the answer for more info). That will buy you more samples if you need them.

If you want to train a single model with data for both cities as input, then making predictions for both cities at each time-step is as simple as defining a final Dense layer, which outputs 2 units. Your validation/test data must then of course contain a tuple of (city1, city2).

A perhaps more sophisticated way to approach this would be to create datasets on a single-city basis, then train several sub-models on each city individually (say for 5 layers), then Merge/Concatenate them and put several further layers on top. This will mean you are combining the learnt features of each city, which are in turn being combined to a higher level of abstraction. Here is the first image I got from a search engine, which sketches the idea.


Interesting question  :)

Using spectrograms means you are essentially using images (of frequencies varying over time). I understand the content is in understood like the graph i.e. with axes time and frequency, but as far as the network knows, you are giving it black and white images; assuming your last dimension (=1) is the channels dimension.

You normally want to take the batch-norm over the features, so it could depend on what you see as a feature. Do you care about the shape of the peaks/troughs of the spectra, or the values encoded in the final channel? The meaning of axis when used in terms of BatchNormalization might be a little confusing. Have a look at these explanations and some of the points here.

So as far as pure images go, I would recommend keeping the default axis=-1.

Remember that a 2d convolution operation is looking for spatial correlations over the image itself - the kernel slides from left to right, top to bottom. So mixing that idea with then taking batch norm over your second axis (frequency) is literally an orthogonal idea.

I don't know if it will work out, but I think it might make most sense if your spectrograms are aligned in time, that is they cover the same nominal time period and so for two of them with the same size, the x-axis has the same scale. Otherwise you would be taking batch-normalization over different time-frames and therefore stirring up temporal information between the samples.


How to classify test objects with this ruleset in order of priority?

Suppose your test object is (sunny, hot, normal, TRUE). Look through the rules top to bottom and see if any of the conditions are matched. The first rule for example tests the outlook feature. The value doesn't match, so the rule isn't matched. Move on to the next rule. And so on. In this case, rule 5 matches the test case and the classification for the p lay variable is "yes".

More generally, for any test case, look at the values its features take and find the first rule that those values satisfy. The implication of that rule will be its classification.


Algorithm to find Unique users from their transactions

You don't need any ML here: in the question you just gave the full algorithm to find the target output ;) At best a ML system would find just the same method after using a lot of annotated data.
The method is essentially what you said:

Group the users by their card number or email (easily done with a map for each), this gives you the cases of "sure" identical users.
Group the users by their name, this gives you the cases of "possible" identical users. For this part you might want to use string similarity measures.


Converting sparse IndexedSlices to a dense Tensor

I've had the same issue with the Embedding layer in Keras. The solution is to explicitly use a TensorFlow optimizer, like here:

model.compile(loss='mse',
               optimizer=TFOptimizer(tf.train.GradientDescentOptimizer(0.1)))


How to search for an optimal dithering pattern?

I think you're on the right track. If you can produce an objective scoring algorithm for the output which more closely resembles your subjective perception than 1:1 error you could definitely use simulated annealing to derive from that an optimal matrix at a given size N.

Your current scoring calculation for error seems to focus too much on high-frequency information (details), and not enough on low frequency information (larger areas) - notice the results you deem subjectively superior, if blurred (low-pass-filter) more closely resemble the input image (if also blurred) compared to the results you have deemed inferior.

If it were me, my next step might be to try something hierarchical in order to avoid bias toward any given frequencies, maybe a quadtree where at each level 1x1, 2x2, 4x4, etc. I average the values of the node's children, then scoring uses the error between trees.


You haven't actually used accuracy_score, and the default scorer for sklearn regressors is R^2:
https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor.score

A negative value for R^2 generally means a very bad fit:
https://stats.stackexchange.com/q/12900/232706


When to use k-medoids over k-means and vice versa?

1) Time complexity of KMEANS

As explained in this post :

  KMeans is an NP-hard problem. However, running a fixed number $t$ of iterations of the standard algorithm takes only $O(t*k*n*d)$, for $n$ (d-dimensional) points, where $k$ is the number of centroids (or clusters). This what practical implementations do (often with random restarts between the iterations).

2) When would you use one over the other?

As mentioned in this Wikipedia article, K-medoids is less sensitive to outliers and noise because of the function it minimizes.

  It is more robust to noise and outliers as compared to k-means because it minimizes a sum of pairwise dissimilarities instead of a sum of squared Euclidean distances.

Also, K-medoids can use a variety of similarity measures where K-means is limited to euclidian (pairwise) distance. Excellent explanation [here].(https://stats.stackexchange.com/a/81496/279276)

3) What qualities does one look for to use k-medoids?

I would recommend using it whenever Euclidean Distance does not make sense in your data. If Euclidean Distance does not make sense (i.e. unrelated categorical variables : "has wings", "# of legs"), minimizing the sum of squared Euclidian distances probably won't either.

4) What are the differences in the output

The main difference is that medoids (equivalent to centroïds in K-Means) belong to the data points. You will never have a medoid that is somewhere between points. Instead, it will be superimposed on an existing point. This post shows it clearly.

It makes sense, especially for a categorical feature (# of legs), not to have a cluster center at 3.347 legs.

Hope this helps.


Do I need to include a squared and linear variable in a random forest to achieve a parabolic effect?

Random Forest uses a ensemble of Decision Trees using CART algorithm.

Since CARTs (Classification And Regression Tree) are a non-parametric algorithm, they should be able to find interactions between variables and non-linear behaviors.

Nevertheless, building polynomials can help them have a better performance.


Extracting model equation and other data from 'glm' function in R

In order to extract some data from the fitted glm model object, you need to figure out where that data resides (use documentation and str() for that). Some data might be available from the summary.glm object, while more detailed data is available from the glm object itself. For extracting model parameters, you can use coef() function or direct access to the structure.

UPDATE:

From Princeton's* introduction to R course's website, GLM section - see for details & examples:

  The functions that can be used to extract results from the fit
  include

- 'residuals' or 'resid', for the deviance residuals
- 'fitted' or 'fitted.values', for the fitted values (estimated probabilities)
- 'predict', for the linear predictor (estimated logits)
- 'coef' or 'coefficients', for the coefficients, and
- 'deviance', for the deviance.

  Some of these functions have optional arguments; for example, you can
  extract five different types of residuals, called "deviance",
  "pearson", "response" (response - fitted value), "working" (the
  working dependent variable in the IRLS algorithm - linear predictor),
  and "partial" (a matrix of working residuals formed by omitting each
  term in the model). You specify the one you want using the type
  argument, for example residuals(lrfit,type="pearson").

*) More accurately, this website is by Germán Rodríguez from Princeton University.


Sparse Representation vs Dense Representation

Sparse representations are not always better than Dense, though they do have an upper hand. In terms of various inputs being handled (varying amount of information), without a sparse prior we shall be trying to learn same size fit for all representation, that may be inadequate. Here dense representations will have lesser degrees of freedom, resulting in non-linear relationship. Overall our representation mathematically needs to mimic the topology of underlying manifold. But in how many scenarios can we have sparse representation, is the bigger question/problem, where we we have sufficient uncorrupted identities. Approaches like SLR dictionary decomposition come in handy in such scenarios.


to extract multiple of n images using open cv in python

You can use the modulo operator to identify multiples of count. Whenever it is a multiple of 5 save the image, otherwise just disregard it and move to the next iteration of the while loop.

Make sure the counter is outside the if statement. It will count every iteration. That is how we will determine if the current image is a multiple of 5.

import cv2
import os
os.chdir('D:/Python images/videofolder')
cap = cv2.VideoCapture('D:/Python images/test_videos/testcutvideo1.avi')
cap.isOpened()

count = 0
success = True
while success:
    success,image = cap.read()
    if count % 5 == 0:
        cv2.imwrite("image%d.jpg" % count, image) # save frame as JPEG file
    count += 1


Loading a Model with weights and optimizers without creating an instance in PyTorch

If you are open to using huggingface transformer for fine tuning which is really popular, here is a code sample:
import transformers
class Model(nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.Bert = transformers.CamemBertModel.from_pretrained('camembert-base')
        self.fc0 = nn.Linear(768,1)

        nn.init.normal_(self.fc0.weight,std= 0.1)
        nn.init.normal_(self.fc0.bias ,0.)

    def forward(self,input_ids,attention_mask):
        hid= self.Bert(input_ids,attention_mask = attention_mask)
        hid= hid[0][:,0]
        x = self.fc0(hid)
        return x

You can change your last layer as you need.It is just a sample. It will load the pretrained weights from huggingface.You need not provide it.You can install transformers by the following line in terminal.
pip3 install transformers


Convolutional neural network block notation

3x3 conv, 256, /2

stands for:

3x3 Kernel
256 filters
a stride of 2 halving the spatial dimensions

The latter is explained on page 3 where the authors state

  (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer. We perform downsampling directly by
  convolutional layers that have a stride of 2.

This means ResNet does, except for the beginning and end of the network, not use pooling layers to reduce spatial dimensions but conv. layers.

Also, table 1 shows what is happening:

The part you have highlighted in your screenshot is the transition from conv3_x to the conv4_x layer of the 34-layer network. As you can see in the table the output size is reduced from 28x28 to 14x14 (that is what /2 does) while the filters are doubled from 128 to 256.


How to use vectors produced by TF-IDF as an input for fuzzy c-means?

Assign a unique column to each word.

All remaining values are zero, obviously.


Are there any deep learning models to forcast waveforms from waveforms?

All sequence-to-sequence models will for this. If you want to do image to image translation, then, just convert your image into sequence using CNNs (see text recognition models).


Understanding the filter function in Convolution Neural Networks

The function set_weights on a Keras layer requires the shape of the inputs to match the shape of the weights which you are replacing. You can find out which dimensions these are by calling the get_weights method on the layer in which you are interested.

In that tutorial, it would look something like the following. We only need the first element returned by get_weights(), hence the [0]. We then see the shape of it:

In [7]: model.layers[0].get_weights[0].shape
(4, 4, 1, 1)

So the person who wrote that tutorial needed to match that shape and, as it is only considering one layer of the defined model, it is able to be hard-coded in their example.

Here are the docstrings for the two main functions I mentioned above:

set_weights()

In [8]: l1.set_weights?

Signature: l1.set_weights(weights)
Docstring:
Sets the weights of the layer, from Numpy arrays.

# Arguments
    weights: a list of Numpy arrays. The number
        of arrays and their shape must match
        number of the dimensions of the weights
        of the layer (i.e. it should match the
        output of `get_weights`).

# Raises
    ValueError: If the provided weights list does not match the
        layer's specifications.

get_weights()

In [9]: l1.get_weights?
Signature: l1.get_weights()
Docstring:
Returns the current weights of the layer.

# Returns
    Weights values as a list of numpy arrays


I my opinion, the best way is if you  create a column for every(!) product. Yes, u will  get a high dimensional sparse Matrix but it should solve you problem quickly.


Prediction for not completely well classified data

You are dealing with noisy labels. I would not switch the labelling according to a trained model that learned on that particular data set, since probably you don't know which patterns lead to your models decision.

Otherwise if you know the reason for the wrong labelling, you could try to build methods yourself that run a sanity check on your data.

Nonetheless, models that don't overfit often can deal with a bit of noise in the labels. But you can also try methods that incorporate the noise aspect. Maybe check out https://stats.stackexchange.com/questions/218656/classification-with-noisy-labels


Interpretation of 'recall' as a conditional probability P( X=>+ | X=+ )

Your interpretation is correct.
$ F_n $ represents the number of false negatives whereas $ F_p $ represents the number of false positives.
Recall
answers the question:
when presented a positive example, how often does the classifier get it right?
aliases: sensitivity, true positive rate (TPR)
Precision
answers the question: out of all the examples the classifier thought were positive, how often were the examples actually positive?
aliases: positive predictive value (PPV)


Knowledge Graph as an input to a neural network

You can represent this type of knowledge graph as a binary $n \times r \times n$ tensor.  (You can think of this as a 3D matrix if it helps.)
The first dimension is for the node on the left side of the relationship, the second dimension is for the relation type, and the third dimension is for the node on the right side of the relationship.  Then you can represent any relationship between two nodes by inserting ones in the correct indices.
To demonstrate, suppose we have 4 nodes: Mike at index 0, Sully at index 1, chocolate at index 2, and Biology at index 3.
And suppose we have 2 relations: likes (index 0), studies (index 1).
We would first create a $4 \times 2 \times 4$ tensor filled with zeros:
import torch

graph = torch.zeros([4, 2, 4], dtype=torch.bool)

""" Output:
tensor([[[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]]])
"""

To represent the relationship :Mike :likes :chocolate; we would insert a one at the position [0, 0, 2]  (0 for Mike, 0 for likes, 2 for chocolate).
To represent :Mike :studies :Biology, we have a one at [0, 1, 3].
# :Mike :likes :chocolate
graph[0, 0, 2] = 1

# :Mike :studies :Biology
graph[0, 1, 3] = 1

""" graph:
tensor([[[False, False,  True, False],
         [False, False, False,  True]],

        [[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]],

        [[False, False, False, False],
         [False, False, False, False]]])
"""

Of course we can represent any relationship between 2 nodes. Let's add a few more:
# Mike and Sully are friends

# :Sully :likes :Mike
graph[1, 0, 0] = 1

# :Mike :likes :Sully
graph[0, 0, 1] = 1

# I suppose chocolatogoly is a subject in Biology
# :Biology :studies :chocolate
graph[3, 1, 2] = 1

Concerning architecture, there are at least two viable ways to feed this representation to a neural network.  You could flatten the whole graph and treat it like a 1-dimensional input.  Another option is to retain the 3 dimensions and use convolutional layers (or something else) to extract features (3D convolution works just like 2D convolution).
As for the rest of the architecture, that's up to you!


Can I modify a Logistic Regression classifier to out put more than one class based on the probabilty?

We can't use the same data with a different label in the same model.
It will confuse the model.
What you need is multiple models e.g. OneVsRest strategy. i.e. one for each Class.
Little clarification-
What you are looking for is Multi-label, not Multi-class. Please check the internet on this. May read this SO Answer for a quick understanding.
You can implement an OneVsRest multi-label yourself but better use the Scikit-learn implementation Link.

OneVsRestClassifier can also be used for multilabel classification. To use this feature, provide an indicator matrix for the target y when calling .fit.

What it means is to encode your data in a multi-label format when calling fit().
You may use the from sklearn.preprocessing import MultiLabelBinarizer for this.
classif = OneVsRestClassifier(LogisticRegression())
classif.fit(X, Y)

Read this SO Answer for an example.
You may also use a sklearn.multioutput import MultiOutputClassifier wrapper for the same. Check this SO Answer


can i make model that can take N shape input?

The first thing, which comes to mind speaking about unfixed input, is a recurrent neural network. Such a model adresses dynamic behavior. Maybe, it's possible to represent data as a sequence? I don't know nature and specificity of your data, but if each instance in the dataset consists of a different number of some "similar events", it might be reasonable to think in this direction.

The other way which might help is to compute statistics on the data. Then it will be another feature space, but in some cases it could be enough to achieve high accuracy of predictions. For example if there are k features of the same nature (say, temperature in different days), then average, standard deviation, min, max, etc. could be considered. And statistics don't depend on the length of each instance in the dataset, so this number k could differ.


Is the external memory of Differtiable Neural Computer limited?

If the reference to 7 chunks of information for humans is a reference to the 'Magical Number Seven' Paper, which is the human working memory, then:

Consider the high level operation of DNC at a timestep. An input comes in, and is processed by an LSTM to generate a 'Interface vector' which interacts with memory with Write, Erase and Read operations. The operations are then executed, and a final layer produces an output based on the retrieved information and the input.

Where is the analogy to human 'working memory'?

1). If you consider the working memory is what is retrieved on read, then there is a hyperparameter for number of memory reads. This applies generally to Write and Erase as well as those are all hyperparameters which determine the size of the Interface Vector. In the paper they used a write size of 1 in all cases for purposes of controlling the experiments. Note that the 'size' of what is read/written in a single operation is a separate parameter, which just depends on how the problem is encoded (see the methods section of the paper).

2). If you consider working memory as external memory, then this is another hyperparameter. I think it is a conceptual mistake to view the external memory as short term memory, as the addition of this and all the gating was specifically developed to store information long term and then recall when needed.

3). There is a part of the DNC's operation which queries a link matrix (storing what was written to long term memory and when) during the read phase. This can be seen as being related to human working memory, as numerous behavioral economics studies showed that what we recall and how we process it is effected by what we have just seen. This is the only part in the paper where the computation is O(N^2) w.r.t to the size of the memory, (though they approximate with an O(NlogN) method). This portion of retrieval/bias of what is retrieved to most recently accessed information can then be said to have a limit, since for large memories it will not scale with rest of network, but this is still not constant.

4). Probably the best candidate for the working memory analogy IMO is the hidden state of the controller LSTM, since this part of the algorithm is responsible for storing what is needed over time for retrieval, and on its own is used for storing a state over time. The reason that this is short term memory in this analogy is just that the external memory is an explicit external memory, and this is the only other place where a state is maintained.

So in short no, there is no Constant, as just about every conceivable analogy to 'working memory' and the entire architecture is controlled by parameters. It is worth checking out the code and paper for more details on exactly how the parameters are set for different experiments.

Let me know if this makes sense.


Ok, after some digging around I found that I can pass a range = (1,100) and that does the trick.


Spam filtering, especially in email, has been revolutionized by neural networks, here are a couple papers that provide good reading on the subject:

On Neural Networks And The Future Of Spam
A. C. Cosoi, M. S. Vlad, V. Sgarciu
http://ceai.srait.ro/index.php/ceai/article/viewFile/18/8

Intelligent Word-Based Spam Filter Detection Using
Multi-Neural Networks
Ann Nosseir, Khaled Nagati and Islam Taj-Eddin
http://www.ijcsi.org/papers/IJCSI-10-2-1-17-21.pdf

Spam Detection using Adaptive Neural Networks: Adaptive Resonance Theory
David Ndumiyana, Richard Gotora, and Tarisai Mupamombe
http://onlineresearchjournals.org/JPESR/pdf/2013/apr/Ndumiyana%20et%20al.pdf

EDIT:
The basic intuition behind using a neural network to help with spam filtering is by providing a weight to terms based on how often they are associated with spam.

Neural networks can be trained most quickly in a supervised -- you explicitly provide the classification of the sentence in the training set -- environment.  Without going into the nitty gritty the basic idea can be illustrated with these sentences:

Text = "How is the loss of the Viagra patent going to affect Pfizer", Spam = false
Text = "Cheap Viagra Buy Now", Spam = true
Text = "Online pharmacy Viagra Cialis Lipitor", Spam = true

For a two stage neural network, the first stage will calculate the likelihood of spam based off of if the word exists in the sentence.  So from our example:

viagra => 66%
buy => 100%
Pfizer => 0%
etc..

Then for the second stage the results in the first stage are used as variables in the second stage:

viagra & buy => 100%
Pfizer & viagra=> 0%

This basic idea is run for many of the permutations of the all the words in your training data.  The end results once trained is basically just an equation that based of the context of the words in the sentence can assign a probability of being spam.  Set spamminess threshold, and filter out any data higher then said threshold.


Heuristics, methods to speed up searches over subsets of big set (combinatorially NP hard probably)

This is not ML but your setup reminds me of the Covering Array Problem, which is used for Software Testing.
If you consider your configuration arrays [0 0 1 0 1 0 1 0 1 .... ] as certain tests run , a typical question is what is the minimum number of tests I need to run while I can cover most different setups? Important to increase test completeness but also minimize time it takes to run them.
Possible some covering array solutions might help. There are even tables that are pre-computed that you can use for different number of N and coverage strengths https://math.nist.gov/coveringarrays/coveringarray.html
And as you pointed out this is a combinatorial problem that suffers from combinatorial explosion. You might want to divide/conquer (split the N into k manageable groups).


How to know the name of the predicted class?

You as the designer of the network specify each class in the training example. You set e.g. a car class to label $0$ and another class to $1$. During training your classifier tries to map the inputs to the corresponding class which are those numbers where you have specified in the dataset. This phase is called encoding. After outputting the label, you should decode the label which is a trivial task!


Loss function for an RNN used for binary classification

The second option is the good one.

You can select the last output that correspond to a not padded input and using it for your loss.
Or you can directly express that explicitly : in keras set the flag return_sequence to False and your RNN layer will only give you the last output that correspond to a not padded input.

If I were you I would try to put a dense layer between the RNN layer and the last output. Don't forget to use a softmax activation function for the last layer in order to get probabilities.


Python writing to Excel file: writerow() takes no keyword arguments

Please see doc of csv.writer
delimiter param is used when instantiating the writer, not on writerow method.
Furthermore from what I understand, your approach is wrong.
For example you want to scrape the html table and store it as csv file.
But what you do is get the text from table
table = soup.find('table', id='tablepress-94').text.strip()

which is a concatenated string of all columns with no structure and try to split by nothing, which is wrong
while you should be getting a dict of table values, eg using:
table = soup.find('table', id='tablepress-94')
# assuming table has a
#

Column 1	..Column n

# section with column headers, else adjust accordingly
column_headers = [td.get_text() for td in table.find('thead').find('tr').find_all('th')]
print(column_headers)
data = {}
row_index = 0
for row in table.find('tbody').find_all('tr'):
    row_index += 1
    column_index = 0
    columns = row.find_all('td')
    for td in columns:
        if column_index >= len(column_headers):
            column_headers.append('Column_'+str(column_index+1))
        if column_headers[column_index] not in data:
            data[column_headers[column_index]] = []
        data[column_headers[column_index]].append(td.get_text())
        column_index += 1

print(data)
# now output your scarped table data, data into csv properly
writer.writerow(column_headers) # print headers

# print each row of data
for row in range(row_index):
    writer.writerow([data[column][row] for column in column_headers])


Batching in Recurrent Neural Networks (RNNs) when there is only a single instance per time step?

Your edit is not right. from the keras documentation you can actually understand the difference between timesteps and batches. I take your examples:

For the first example.
You have 4 instances, or samples, or sequences. The length of each sequence is 30, now
you actually take 3 batches, each of these batch is composed by 4 instances, or samples, or subsequences, of length 10(timesteps). your batch size is not 10 but 4, the timesteps don't affect the batch size. the subsequences of each batch can have the length that you want, this doesn't change the fact that the batch size is 4.

For the second example is the same.
Online learning doesn't mean that the number of timesteps of each subsequence is 1, you can take the number of timesteps you want, but when you update the parameters of the network you should consider only 1 subsequence at the time.

You can actually confirm this from here:

http://philipperemy.github.io/keras-stateful-lstm/

In this part:

"Said differently, whenever you train or test your LSTM, you first have to build your input matrix X of shape nb_samples, timesteps, input_dim where your batch size divides nb_samples. For instance, if nb_samples=1024 and batch_size=64, it means that your model will receive blocks of 64 samples, compute each output (whatever the number of timesteps is for every sample), average the gradients and propagate it to update the parameters vector."

So the timesteps don't affect the batch size.


Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you use the training data from the models themselves to also combine the model output, you risk over fitting your final model. If you have a small data set, trying to build an ensemble on top of it might lead to worse performance in deployment since the model is only memorizing the training data. In that case, a simple average might do you better than trying to anything more sophisticated.

But, assuming you have more data to use, or if you want to use the data you trained on anyways, you can either design a weighted average or create a second model.

If you have an idea of which model performs better, you might want to try manually assigning weights for each model's output. A simple average assigns an equal weight to each output, but you can experiment with shifting the weight around a bit.

For an example, given two models, a basic average yields

new_pred =  0.5 * mod_1  + 0.5 * mod_2

However, if you see the performance on model one does better overall, you might shift the average to

new_pred =  0.9 mod_1  + 0.1 mod_2

If instead you want to create a second model, you can try using any machine learning model suitable for classification.  I would probably stay away from larger models like Random Forests, but doing something like using the predictions as features into a Logistic Regression model can work well. But again, I would emphasize only trying to use the predictions from a holdout data set into the logistic regression model to make sure you aren't over fitting.


First let's look at how these are related. The cumulative distribution function (CDF) is the cumulative sum of the probability density function (PDF).
You take the sum of all the probabilities of the previous values.

For example, if we roll a dice, then there is a 1/6th chance of getting any value. Thus the PDF is

Then the cumulative distribution function at any point is the sum of all the points before it. So the CDF of rolling a 2, is 2/6th, a 3 would be 3/6th. The graph looks like

Mathematically we define the CDF as $F$ as
$F(x) = Pr(X < x)$.
For our dice example
$F(1) = P(X=1) = 1/6$
$F(2) = P(X=1) + P(X=2) = 2/6$
$F(3) = P(X=1) + P(X=2) + P(X=3) = 1/6 + 1/6 + 1/6 = 3/6$
$F(4) = 4/6$
$F(5) = 5/6$
$F(6) = 1$.
So you can see how we get the plot above.


Mathematic explanation needed for a univariate classification method based on solving a quadratic equation

Actually, it is very straightforward to derive.

Hint 1: where would you most likely find mean and variance used? Gaussian!

Hint 2: how would you do classification using Gaussians? Simple, you fit a Gaussian to each group and compare the PDFs of new data $x$. Even simpler, you fit a Gaussian to each group and calculate the Mahalanobis distance of $x$ to each group.

A threshold is called a threshold because at this value you cannot tell which class it is in, aka, the Mahalanobis distances to two groups equal:

$$\frac{(x-\mu_A)^2}{\sigma_A^2}=\frac{(x-\mu_B)^2}{\sigma_B^2}$$

Expanding this, you will get the quadratic formula.


Time series feature extraction from raw sensor data for classification?

You can also use an open source python library called 'tsfresh' (https://tsfresh.readthedocs.io/en/latest/) to extract time series features


How are pictures pre processed before being used as ML data

First images are turned into pixel values. If it is a color image we have RGB combination of values. It it is black and white then the pixel values for each image in the dataset are unsigned integers in the range between black and white, or 0 and 255. This is the first transformation into numbers.
After that we apply an embedding to it to convert it into an array of numbers. We then use this embedding for further processing.
Images cannot directly be used because AI/ML algorithm deals with numbers only.
You may want to take a simpler image base like the MNIST which is a simple 0-9 representation on a 28*28 pixel image and then play around with recognition algorithms for a better understanding.


Assuming your dataset includes $k$ samples:

In cross-validation, there are $N$ partitions, and the test split for each partition will have size $\frac{k}{N}$.

Leave-one-out validation is a special type of cross-validation where $N = k$.  You can think of this as taking cross-validation to its extreme, where we set the number of partitions to its maximum possible value.  In leave-one-out validation, the test split will have size $\frac{k}{k} = 1$

It's easy to visualize the difference. Here's two figures which contrast cross-validation and leave-one-out.  In these figures, each sample in the dataset is represented by a colored circle. The training set is represented by the green circles, and the testing set is represented by the yellow circles.

5-fold cross validation:

Leave-one-out cross validation:


What is the loss function defined by Mnih and Hinton in their paper “Learning to Label Aerial Images from Noisy Data”?

Section 3.3 simply gives the equation for the negative log-likelihood.
They say that it takes the form of cross-entropy (because it just looks like a cross-entropy equation, perhaps?), but mathematically it seems to come from the fact that they define the model in equation 2 to follow a Bernoulli distribution, which can be 0 or 1 with probabilities p or q respectively:

$$
\Pr(X=1)=p=1-\Pr(X=0)=1-q
$$

The probability mass function of the Bernoulli looks like this:

$$
f(x;p)=px+(1-p)(1-x)\!\quad {\text{for }}x\in \{0,1\}
$$

and the maximum likelihood for the Bernoulli looks like this:

$$
L(p) = \prod_{i+1}^n p^{x_1} (1 - p)^{1 - x_1}
$$

Taking the log (to get the log-likelihood that they mention), gives something of the form we see in equation 3:

$$
\log{L(p)} = \log{p}\sum_{i=1}^n x_i + \log{(1-p)}\sum_{i=1}^n (1-x_i)
$$

Then you can takes the sum out the front as it is over the image patches:

$$
\sum_{i}^{w^2_m} (\tilde{m}_i ln\hat{m}_i + (1-\tilde{m}_i)ln(1- \hat{m}_i))
$$
So here
This sums over all the pixel-wise output units in each image patch (map: m), hence why the summation goes to $w_m^2$; the width of the image patch squared. $\tilde{m}$ is a realisation (possible outcome) of $\hat{m}$, just as $x$ is a realisation of distribution $p$ in the above equations.

This looks like the binary cross-entropy loss function, which is why I think they say:

  For the model given in Equation 2 the negative log likelihood
  takes the form of a cross entropy between the
  patch $\tilde{m}$ derived from the given map and the predicted
  patch $\hat{m}$


Does ridge regression always reduce coefficients by equal proportions?

As I know we have below equation for Ridge Regression:

\begin{equation}
RSS_{Ridge} = \Sigma_{i=1}^{n} (\hat{y}_{i} - y_{i})^2 - \lambda \Sigma_{j=1}^{p}(\beta^2)
\end{equation}

First of all, it seems to me, if lambda goes higher does not mean that coefficients go down with inverse relation to lambda. Because the power of beta is two and the lambda is one.

I think you refer to page 226 of "An Introduction to Statistical Learning" book. In the footnote of the figure, the writer says:

"The ridge regression and lasso coefficient estimates for a simple setting with n = p and X

a diagonal matrix with 1’s on the diagonal. Left: The ridge regression coefficient estimates

are shrunken proportionally towards zero, relative to the least-squares estimates.

Right: The lasso coefficient estimates are soft-thresholded towards zero."

In that figure shows that if we got +2 and -2 coefficient in the OLS model and if Ridge shrunk +2 to +1.8 then we are sure Ridge will shrink -2 to -1.8. So, in both cases with the same proportion of 0.2 coefficients go to near zero.


Doc2vec(gensim) - How can I infer unseen sentences’ label?

The title of this question is a separate question to its text so I will answer both separately (given that one leads into the other).

How can I infer unseen sentences:

# ... trained model stored in var model
list_of_words = ["this", "is", "a", "new","unseen", "sentence"]
inferred_embedding = model.infer_vector(list_of_words)

How does this work ? As per the original paper (https://cs.stanford.edu/~quocle/paragraph_vector.pdf) we have two weight matrices initialized at random $W \in \mathbb{R}^{N\times P}$ which is the same matrix from Word2Vec in which every column or row maps to a word vector and $D \in \mathbb{R}^{M \times R}$ which is the document matrix where each column or row maps to a sentence/document. During training a softmax classifier of fixed window $k$ size moves (in a moving window like fashion) minimizing the following log likelihood (multi-class cross-entropy):

$$
\frac{1}{M}\sum_{i=1}^{M}\frac{1}{|D_{i}|}\sum_{t=k}^{|D_{i-1}|-k}log(p(w_{t}^{i} | w_{t-k}^{i}, ..., w_{t+k}^{i},D_{i} ))
$$

Where $D_{i}$ corresponds to the vector representing the $i^{th}$ sentence, $|D_{i}|$ its number of words in that document and $w_{t}^{i}$ is the $t^{th}$ word in the $i^{th}$ document. All back-propagation remembers is the document we are currently moving our windowed softmax over and only updates that row in matrix $D$ along with the words in that window.

Furthermore when we want to infer something not in the training set we fix $W$ so that it is not updated and augment matrix $D$ with the new randomly initialised row and just train for several iterations (with the new row holding the embedding for the inferred vector). This leads into question 2.

Can we infer that a possibly unseen sentence exactly corresponds to a sentence in the training set ?

The short answer is no and this is not what Doc2Vec is for. Because of the random initialization + the complexity of convergence and training your inferred vector will never be exactly the same as its corresponding vector in $D$ this is why Gensim has not built in a function to support this, how ever given that the model has been well trained these two vectors should be arbitrarily close to each other  thus you can conclude that they are extremely similar.

Even fixing the random seed may not work, there are so many other variables that can affect its convergence please see first answer on https://github.com/RaRe-Technologies/gensim/issues/374 .

In any case you can find the most similar label in your data set to an inferred sentence just by iterating over your training set and comparing the similarity to the inferred vector. But why would you want to do an exact match to something in the training set ? that's what regular expressions are for the purpose of these document embedding's is for supervised or unsupervised learning tasks (i.e. classification and clustering).


Where to get the Datascience Use cases for practice

Maybe you can start with the Kaggle datasets: https://www.kaggle.com/datasets
There are more then 22.000 datasets and they are very well documented, enough for hours and hours of practice.


Some robust perhaps more complex as neural networks to measure the distance between data?

It all depends how you frame your problem.

Whats complex? more parameters, more time complexity, more exotic looking etc...

Here are 2 cool ideas how to measure similarity:

Mahalanobis distance measure of the distance between a point P and a distribution D. Cool but more importantly its robust because you are using parameters of your distribution to measure the distance.

Neural networks. If we relax the definition of "measuring distance between data" then Check siamesse networks. The gist of it is, can you group similiar sequences of data together. Why I like it? cause of one-shot learning


Are all DNN models supervised learning (or reinforcement learning)?

There are many types of unsupervised learning models in deep neural networks. One common type is an autoencoder, which learns a compressed representation of the data.


If you use the anomaly detector to label the data directly, there is no way the  supervised step that follows can be better than that. One can of course go in an "adjust" the labels afterwards, but there is risk of being biased by the pre-existing labels if a human sees it up front.

Instead of sampling data to label randomly, you could sample weighted based on the anomaly score. This has two effects 1) reduces class imbalance 2) focuses labeling effort on likely anomalies

This assumes a well-tuned anomaly detector though. Which there is no good way to do without a validation set. That dataset should preferably be sampled randomly, to avoid bias.


Quantifying the imbalanceness of a dataset

I think "balanced" can have different meanings in different contexts, but typically a "balanced" dataset is one in which the class labels are uniformly distributed.  Of course, an "imbalanced" dataset is one in which the distribution of class labels is not uniform.

I'm not aware of discussions about balance metrics in the literature.  One straightforward option is to measure the KL-divergence between the actual distribution of the class labels and a uniform distribution over the class labels.  A KL-divergence of zero would indicate a perfectly balanced dataset.  The KL-divergence will grow to infinity as the actual distribution differs from the perfectly-balanced distribution. This metric would allow you to compare the degree of imbalance between two datasets.

Unfortunately, KL-divergence is pretty fragile.  It's technically undefined when one of the classes has zero examples.  In practice you could treat such datasets as if they're infinitely imbalanced. This is what scipy.stats.entropy does.

However, this treatment has one undesirable side-effect.  Any dataset in which there are zero examples of a class gets the same "imbalance score" (infinity). This doesn't match our intuition about imbalance.  Suppose we have two datasets with examples from the classes "dog", "cat", and "horse".  Let's say dataset A has 19 dogs, 1 cat, and 0 horses.  Dataset B has 10 dogs, 10 cats, and 0 horses.
Ideally our metric would say that Dataset A is more imbalanced than Dataset B, but KL-divergence will give the same result for both datasets.

We can correct this problem by adding a tiny epsilon to each of the class distributions before calculating KL-divergence.  This both solves the problem of being undefined when a class has zero examples, and it also corrects for the above scenario where two different datasets get the same imbalance score.


will increasing threshold always increase precision?

here precision at threshold 0.85 > precision at threshold 0.90. shouldnt it be the other way round? increasing threshold will reduce False positive and precision will be greater than before?

Precision is $\frac{\text{TP}}{\text{TP}+\text{FP}}$

Both $\text{TP}$ and $\text{FP}$ are reduced when you increase the threshold. If both decrease in proportion to the current precision (i.e. they are spread evenly at each confidence value), then precision will remain the same. Most models on most datasets will tend to increase precision as the threshold increases, at least initially (e.g. moving from 0.5 to 0.6) as false positives may commonly be found as uncertain edge cases with low confidence, i.e. false positives tend to occur more frequently at low confidence, so increasing threshold will exclude a higher ratio of false positives than true positives than the current precision. However, there is no guarantee of that.

The value of precision will vary in practice depending on what the model predicted for each example. If you have a cluster of highly confident false positives, they can cause precision to drop as threshold grows, until they get excluded. The most extreme example would be where the most confident classification is incorrect, in which case the highest possible threshold will score zero precision.


Graphs demonstrating the structure of neural networks are very unclear

You are making one mistake which cascades on towards other mistakes. The multiple inputs are different features of the same sample. Let's say we want to regress house prices based on some features of a house. These three nodes could represent the rating of the neighborhood, the surface area and the number of bedrooms for example. Then you take a weighted sum to every of the hidden nodes. Every hidden node takes different weights over these features. The order you mention is per layer, since we need the output of the previous layer as input for the next one, these dependencies force this order.

With regards to output, you can have a combined loss. In case of binary classification you would probably just have a sigmoid activation at the end, however if you would have multiclass classification you would have a softmax layer at the end, one output for every potential class. There are a lot of other things you could want to output, for example a bounding box so you would need coordinates of where the network thinks the box is, or even per pixel if it's part of the object or not, then you would have a m x n grid as output.

EDIT: All the things you mentioned about the GPU etcetera, that is way too far for what this image represents. It's one network for one sample. I do agree it would have been better to just have one output node because most more basic examples in supervised learning have just 1 target.


Adapting Neural Network to new domain without labels

I think those are one of the most cited papers:

https://arxiv.org/pdf/1409.7495.pdf
http://www.jmlr.org/papers/volume17/15-239/15-239.pdf
http://openaccess.thecvf.com/content_cvpr_2017/paper/Tzeng_Adversarial_Discriminative_Domain_CVPR_2017_paper.pdf
http://openaccess.thecvf.com/content_cvpr_2017/papers/Bousmalis_Unsupervised_Pixel-Level_Domain_CVPR_2017_paper.pdf


Why joint probability in generative models?

Its because joint distribution takes all of the values variable Y can take and conditional only one. Additionally it could be easier to compute/use approximative methods for joint.


What makes you confident in your results? At what point do you think you can present your work to tech illiterate superiors?

Hey Welcome to the Site!

What you are saying is right, Data Science din't reach to the stage where it has some standard methods for achieving this(standard procedures, don't know we would be able to reach that stage in near future). But we have some general standards like:

Forecasting: ETS,ARIMA,SARIMA etc
Prediction: Linear Regression,Random Forest, GLM, Neural Network etc
Classification: Logistic Regression, Random Forest etc

When you go to granular level it is hard to generalize, as every business problem is different and one single method cannot be used for solving all the business problems.

So, to answer the next question how do you get confidence that the outcome is good enough, I assume that you have heard about RMSE,MAPE and many more for predictions and confusion matrix for classification problem. We use these metrics to see access the models performance, for example if you are trying to classify whether the given cell is cancer cell or not, there are 100 records in which 90 are non-cancer cells and 10 are cancer cell, your model gives 99% accuracy but could classify 5 out of 9 literally 55% of the total in such scenarios you need to look cannot use accuracy, you need to use F1 score etc. As you were asking about a model right, all models are not useful. True not all models built are going to go for production level, you would choose the best one and productionize it. You can re-train your model on a basis(Daily, Weekly,Monthly based on business requirement). Would you call it a day off post completion of validation? I wouldn't, I would go to the Subject Matter Expert present him the results ask him/her for their insights, if they both are inline then I would do a Beta testing on some actual data and then productionize it.

Now to address your last question, There is no standard saying that this is good or bad, if it works for you, your Business then that is a Good Model. To convenience your mangers and subject matter(Data) experts, you need to dig deep into the data try all different scenarios ask as many questions as possible. Try understanding the data very well. So, you can answer Business Questions with data supportive answers(This is possible only when you are well-worse with data). As they are very good with business they would be asking questions with respect to business, you need to be ready with all such scenarios by understanding business and data well.

Finally, I do have a feeling like you do. I did alot of things but nothing worked but you shouldn't be unhappy as you understood that these are the ways which would lead you to Unsuccessful results(best example is Thomas Alva Edison has used 1000 diff metals before using Tungsten to make a bulb). Similarly all the methods which we have tried are different steps you have tried to get the solution. My funda is, did I try something different/new everyday or not. Crucial part of this process is, maintenance of clear documentation at each and every step. Which would come in handy in the near future.

Anything in R&D is never a waste it is just an other try or experiment, so your work is never waste. Your are trying to build a solid base for the bright future of your company.


To produce confidence intervals for xgboost model you should train several models (you can use bagging for this). Each model will produce a response for test sample - all responses will form a distribution from which you can easily compute confidence intervals using basic statistics. You should produce response distribution for each test sample.

this answer is provided here:
https://stackoverflow.com/questions/37418938/how-to-obtain-a-confidence-interval-or-a-measure-of-prediction-dispersion-when-u


what metrics to evaluate rank order results?

For comparing two rankings Spearman's rank correlation is a good measure. It's probably worth a try, but since your gold truth appears to be binary I would think that top-N accuracy (or some variant of it) would be more appropriate (advantage: easy to interpret). You could also consider using the Area Under the Curve (AUC), using the predicted rank as variable threshold (less intuitive to but doesn't require choosing any top N).


What makes the validation set a good representative of the test set?

The general idea of using a validation set is to analyse or control the training of the model. Usually, the model tends to under-fit or over-fit, and this can be observed using the validation split. In practical applications, we just have the visible training set and unknown test set. So, with some assumption on the test set, a simple assumption can be such that test set belongs to the similar distribution as of the training set, we use part of training set as validation set. Once the validation set is chosen, we train the system on the training set and at every required step we observe how the trained model performs on the validation set. (Here, we are still under an assumption that the test-set is not visible). Hence, we can use the validation to check if the model is being trained well and to control the training (early stopping) especially in the case of over-fitting. With this assumption a good validation set should be something which has similar distribution as that of the test set. However, most of the times since we split the available dataset into training, validation and test set, the validation set usually has similar distribution as that of test set.

As mentioned above, difference in the distributions of test set and validation set can cause some discrepancies in the results. As you pointed out, if we upsample the training set before obtaining the validation split, then the validation set will contain different distribution than that of the test set. The discrepancies also occur when the the test set is not provided before-hand, and if it has different distribution than that of training/validation set.

Generally, a random-uniform-split produces the similar distribution, i.e, the ratio of  imbalance reflects in all the splits. However, when the number of samples are small (either overall number of samples or number of samples in minority class), the ratio may not be proportional across the splits. Hence, given an option to have stratified-split, you can prefer the stratified split. (As mentioned, it may not make much difference if random-uniform-distribution produces similar ratios across the split).


Can the extent of variability within a dataset be reflected through clustering?

I think it would make sense to compare the distribution (and means) of standard deviations for each variable between the groups. So the mean of the standard deviation for each era would be something like:
$
\overline{\sigma}_{Era} = \frac{\sum_{i = (features)}  \sigma_{i}}{n}
$
where $n$ is the number of features for each era, and $ \sigma_{i} $ is the standard deviation for one feature of the era.
I think interpreting the results of this would be more straight forward and would be simple to compute. It would also be easy to visualize, as you could just make a box plot for the feature standard deviations for each architectural era and you'd be able to easily see which era has the greatest variance.


Modeling and Predicting Co-Occuring Values

I would recommend a Restricted Boltzmann machine, which is a type of neural network that can model probability distributions. What you are trying to do is to estimate the marginal distribution of the missing features. In a probabilistic model, this is done through marginalization. The beauty of this method is that it provides the full distribution over the missing features rather than mere point estimates.

Here are two tutorials.


Role of a known predictor in a disease prediction

Basically what I am trying to know is should we feed variables which we are confident that they will impact the outcome into the model?

  Or is the model to help us know something which we aren't aware of .

It depends, there's no general rule. The goal is of course to predict the target variable as well as possible, i.e. to maximize the performance. So the question is to find the right balance between:

Giving only a few features which are known to help predict the target variables. In this case we make things as easy as possible for the learning algorithm, so that it can perform optimally with the features provided. Disadvantage: maybe there are other more subtle indications that would have helped the algorithm make better predictions.
Feeding the learning algorithm with a very high number of features. In this case we let the algorithm find the relevant indications itself, hoping that it will catch even subtle clues that a human expert would easily miss. Disadvantage: too many features are susceptible of causing two problems: overfitting, when the model relies on very specific patterns which happened by chance in the training set; and redundancy, which can also decrease the performance with some algorithms.

The data-driven answer is to try various combinations of features on the spectrum between these two options, and evaluate the corresponding models on a validation set (preferably with cross-validation in order minimize the effect of chance). This can be done manually or using feature selection/extraction methods.

[added for clarity:] when selecting the best option between different sets of feature (or different algorithms as well), one needs to use a validation set which is distinct from the test set (and also from the training set of course), otherwise there is again a risk of overestimating the performance.


number of neurons for mnist dataset using mlp?

If you train long enough and have "too many" hidden layer units, then you will eventually have over-training. Usually, the goal is to find the smallest number of hidden units that can do reasonably well, as that will generally give you good generalization. In fact, there is the "bottle neck theory" that attempts to show this effect (https://openreview.net/forum?id=ry_WPG-A-)

For more on comparisons between flat hidden layer networks versus convolutional networks, see:

http://conx.readthedocs.io/en/latest/MNIST.html#A-multi-layer-perceptron-network-for-MNIST-classification


Preprocessing: StandardScaler() Do we really need mean to be zero?

The reason is the same.
 I assume you understand how the Features at a very different scale can create issue
 But just scaling will not always bring them on the similar scale because Standard Deviation is dependent on the Range of the Feature.
So, if a feature is very large but in a small range then simply scaling it will not help.
Let's check an example with two features
import numpy as np
np.set_printoptions(precision=2)

feat_A = np.array([1,2,3,4,5])
feat_B = np.array([10000000,10000001,10000002,10000003,10000005])

# Case I  - Only Std
print(feat_A/feat_A.std())
print(feat_B/feat_B.std())

# Case II - Mean and Std
print((feat_A-feat_A.mean())/feat_A.std())
print((feat_B-feat_B.mean())/feat_B.std())

Output
Case - I - [You may see, this doesn't solve the problem]

[0.71 1.41 2.12 2.83 3.54]
[5812381.94 5812382.52 5812383.1  5812383.68 5812384.84]

Case - II - [This is what we were looking for]

[-1.41 -0.71  0.    0.71  1.41]
[-1.28 -0.7  -0.12  0.46  1.63]


From a mix of real-world and back-calculated data how to remove the part that was back-calculated?

Try to fit several linear regression models, for very small number of observations from the data set (for instance, $n$ or $n+1$ randomly selected points, where $n$ is the number of dimensions). If there are enough perfectly linearly aligned points and enough models, it is likely that one of the models will be built from points that were only drawn from the back-calculus.

Then compare the models predictions to the actual data. For the models which perform very well on a large number of test cases (i.e. more than $m$ points are correctly predicted with an error of $\epsilon$ or less), these observations have probably been computed.

You will have to define $m$ and $\epsilon$ based on your knowledge of the problem / by adjustment.

If I may suggest, you can then keep these observations in your dataset, but give them a lower weight for the final model training.


Pros and cons of pandas or R for longitudinal data?

Let me answer this from the point of data in general. I understand you want the answer for longitudinal data but the reason I don't want to specifically answer that is because pretty much all these frameworks treat data in similar ways.

What you should be looking for?

1. How proficient are you in coding?

If you code well enough and you can tackle most problems by coding the I suggest Python + Pandas. The reason is simple since Python is a programming language you can use the language outside just the normal ecosystem of running your task. So if you want to integrate other components such as maybe a web server, store in a database etc. Python will be most useful.
            where as if you are not comfortable with coding then I suggest R is the way to go since it is much more easy to learn but the disadvantage comes when you are trying to do things other than just perform your daily data analysis. It might be limiting

2. In terms of efficiency

Now pretty much all these tools work pretty fast and the differences are hardly noticeable. Usually, you would have a better edge with R in just the way computations are performed. It is a bit more optimized and it utilizes CPU cores to perform a tad bit faster computation than python does. However these days with the heavy intensive RAM etc it is not really that big of a difference. Also, most libraries for heavy matrix calculations are present in both these toolkits. So not really much difference.

In my opinion

I would go for Python just because of the diverse uses it has. Also has a great supporting environment for libraries etc.


Why should re-sampling change the value of model's coefficients?

In each sampling, your data is going to be different from the previous sampling. For each sampling, you are going to find the best line which describes your sample with the least error value. Consequently, for each sample you are going to find a model which may be different due to the fact that it reduces the cost for each sample.


Normalization of possibly not fully representative data

A friend of mine answered on this question in different social media on different language, I'll post his answer here:

1. scaler should be saved in this case.
You do fit_transform in the example the second time you run it, but you should just transform. Scaler should be "fit" once on data train and not change it afterwards. Then you will get 5 in 1 in both cases. And when 10 appears, scaler will convert it to 2.
2. It's not a fact that this will break the model.
I think everything is clear here, but just in case:
Let us consider the simplest case. Suppose there is a linear dependence y = 3 * x, we don't know about it, but there is a dataset:
X Y
1 3
2 6
3 9

If we assume that the model has the form y = a * x, in the process of learning it turns out that
a = 3 gives the best score and when suddenly x = 100 appears, it has no problem mapping to 300, even though 100 is strongly out of dataset (1,2,3).
3. You can check if this breaks the model now.
Suppose there is a chip that takes values 0, 1, 2, 3, 4, 5, 6, 7, 8 ...
You are afraid that after some time samples with values greater than the current maximum (9, 10, 11, ...) will appear and the model will behave strangely on them. You can try to "simulate" this situation.
Split dataset into train and test so samples with values from 0 to 6 will go into train, and samples with values from 7 to 8 will go into test. If accuracy doesn't fail at the test stage, then we can assume that when all the 0-8 ranks get into train, then the appearance of 9-11 won't break anything.

Also, there is answer for the second question: https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value


What algorithms can be used to derive matching rules between known matches in datasets?

What you're trying to do is called "entity resolution" or "record linkage" (you can do a more thorough search in google). A typical approach is to treat it as classification, using as rows each combination of entries from one table matched against the other, label being whether they are a match or not; and any features that you can think about such as edit distance between the two, number of characters in common, whether they start and end with the same letter, largest common subsequence, number of characters in each, etc.

(a more computationally efficient approach is to treat it as one-class classification).


Scaling of variables considering the values of a single column or the whole dataset

Both can make sense depending on your data :

Example 1 : if you have a feature that varies between -100 and -50 on the whole dataset you would want to normalize that column based on the values on the whole dataset
Example 2 : if each sample is a list of values from different sensors and you only care about their relative values (for anomaly detection for instance) then you might want to normalize each sample

That being said, I think 9 times out of 10 you will want to normalize based on the statistics of the whole dataframe.
Note also that normalizing each sample based on the statistics of the whole dataset is an operation that can be reversed using the normalized data and the dataset statistics and therefore no information is lost : your data is basically the same but rescaled. A neural network could very well learn to reverse this operation.
Normalizing each sample based on this sample statistics can also be useful but this operation actually transforms the representation of your data. As an example, let's say you normalize each sample to a norm of 1, you can't recover the initial data without knowing the initial norm of that specific sample.


TF-IDF vectorizer doesn't work better than countvectorizer

There are a few possibilities. First, there is some variability in performance. It could have been only by chance that countvectorizer performed better than tf-idf. Did you use cross validation (with how many folds)? Is the superior performance of the countvectorizer reliable? I would compare performance across folds to make sure countvectorizer consistently performs better.

Second, if you find that countvectorizer reliably outperforms tf-idf on your dataset, then I would dig deeper into the words that are driving this effect. It may be that common words (words which will appear in multiple documents) are helpful in distinguishing between classes. There is substantial research that shows that use of some function words (e.g. first person singular pronouns, “I”) change depending on someone’s psychological state. Function words like pronouns are very common and would be down weighted in tf-idf, but given equal weight to rare words in countvectorizer. I’m not suggesting that first person singular pronouns in particular are driving your results, but it’s worth looking at what words are driving the effect. I would examine which words are important in both types of models, countvectorizer and tf-idf, and then think about whether the words that are most important for the countvectorizer make sense in the context of your text documents and labels. Also, are you removing stop words? You could also see how the models perform with and without stop words, which would be another way to test whether frequent words are actually helping you to distinguish between classes.


Working with Data which is not Normal/Gaussian

There are models that do not make assumption that the underlying data distribution is a normal distribution.

For example, support vector machine just cares about the boundaries of the separating hyperplane and do not assume the exact shape of the distributions. Decision tree models also do not make such assumption.

Gaussian distribution is popular and it has been analyzed frequently for its simplicity but there are other models as well.

If you know the distributions that you believed your data can follow, you can also build your own model, for example by maximizing the likelihood or posterior distributrion.

If you are able to convert your data to normal distribution and use a model that you are familiar with, you can try it and see how does it perform.


Why is sampling useful in machine learning?

TL:DR - If you know the posterior distribution of the complex process (i.e. the output distribution), and that distribution is one which can be modelled with reasonable accuracy, then sampling from it should reasonably represent the responses of the complex system.

An example could be a complex decision process, whereby many decisions are made consecutively, maybe with some conditional or temporal relationships along the way (basically any process that is considered to be complex). Now imagine, after all this complexity, there are e.g. two possible categorically outputs: a zero or a one. Well this something we might be able to model with a Bernoulli distribution, assuming we can estimate a reasonable parameter $p$ (specific to the Bernoulli distribution).

By sampling from the distribution, we would hope to draw samples, which are representative of the complex process. I hope this covers your main question.
I think point 4. is also clear to understand as well from my trivial example. I am unsure as to what cost refers exactly in point 3.

As for point 1., there are examples where (random) sampling, for example in a grid search of model/optimisation parameters can improve results, as it results in an improved exploration of the parameter space compared to other methods such as grid-search.


Is a multi-layer perceptron exactly the same as a simple fully connected neural network?

Yes, a multilayer perceptron is just a collection of interleaved fully connected layers and non-linearities.
The usual non-linearity nowadays is ReLU, but in the past sigmoid and tanh non-linearities were also used.
In the book, the MLP is described this way:

An MLP is composed of one (passthrough) input layer, one or more layers of TLUs,
called hidden layers, and one final layer of TLUs called the output layer (see
Figure 10-7). The layers close to the input layer are usually called the lower layers, and the ones close to the outputs are usually called the upper layers. Every layer except the output layer includes a bias neuron and is fully connected to the next layer.

("TLU" stands for threshold logic unit)


Does finetuning BERT involving updating all of the parameters or just the final classification layer?

Both approaches are reasonable. Updating the BERT weights will train for longer period of time, but should give more accurate results.


It's usually the denominator $p(x)$ (the "evidence") which is intractable. You could attempt to compute it by marginalizing over the latent variable $p(x) = \int p(x|z)p(z)dz$. However, you would need to evaluate all possible values of $z$ which would require exponential time. (That's why in maximium-likelihood estimation you have no intractability problem because you can treat the evidence as a constant.)


f_val is the F Statistic value. Mathematically it is

$ F = \frac{MS_{Between}}{MS_{Within}}$

The null hypothesis for your ANOVA is

$H_0: \mu_{Entire home/apt} = \mu_{Private room} = \mu_{Shared room} $ which means all means ($\mu_i$ s) are equal and there is no need of grouping using explanatory variable

vs

$H_A:$ At least one $\mu_i$ is different. There is a need for grouping

The p-value for this test was very very low, hence python returned 0. Anything less than 0.05 is considered low enough to reject the null hypothesis.

Independent variables are also called explanatory variables. I believe price is a dependent variable.


Choice of method for classification and/or duration prediction

If you are interested in predictions, I would do two different models to get the most out of each single step.

If you are interested in causal effects, you may consider so called „hurdle models“.
https://www.stata.com/stata14/hurdle-models/


You actually do not have a problem here. The way np.correlate is implemented, you get the negative delay values first. If you plotted your entire signal, it would look as expected. Run plt.plot(result) and you will get

The maximum is at 100 and represents zero delay. That the maximum is so high, is because np.correlate does not normalize the function, but just computes the sum, as you can see in the documentation

  This function computes the correlation as generally defined in signal
  processing texts:

  c_{av}[k] = sum_n a[n+k] * conj(v[n])

  with a and v sequences being zero-padded where necessary and conj
  being the conjugate.

That the acf decays for your signal and for delays larger or smaller than zero is due to its finite length. For an infinite signal, you would be able to see the periodicity without decay.


Neural Network data type conversion - float from int?

Technically with most languages you could pass in integer features for the input layer, since the weights will be floats, and multiplying a float by an integer will give you a float. Also, you don't usually care about partial derivatives of the input data, so it doesn't matter that the values are discrete.

However:

For all weights and neuron activations, if you are using a method based on backpropagation for training updates, then you need a data type that approximates real numbers, so that you can apply fractional updates based on differentiation. Best weight values are often going to be fractional, non-whole numbers. Non-linearities such as sigmoid are also going to output floats. So after the input layer you have matrices of float values anyway. There is not much speed advantage multiplying integer matrix with float one (possibly even slightly slower, depending on type casting mechanism). So the input may as well be float.
In addition, for efficient training, the neural network inputs should be normalised to a specific roughly unit range (-1.0 to 1.0) or to mean 0, standard deviation 1.0. Both of these require float representation. If you have input data in 0-255 range - float or not - you will usually find the network will learn less effectively.

There are exceptions to these rules in some architectures and learning algorithms, where perhaps an integer-based input would work, but for the most common NN types, including MLP and "deep" feed-forward networks, it is simpler and easier to use float data type.


Open source solver for large mixed integer programming task?

Never done stuff on that scale, but as no-one else has jumped in yet have you seen these two papers that discuss non-commercial solutions?  Symphony and COIN-OR seem to be the dominant suggestions.

Linderoth, Jeffrey T., and Andrea Lodi. "MILP software." Wiley encyclopedia of operations research and management science (2010). PDF version

Linderoth, Jeffrey T., and Ted K. Ralphs. "Noncommercial software for mixed-integer linear programming." Integer programming: theory and practice 3 (2005): 253-303. Compares performance


$p(s', r | s, a)$ is the probability of arriving at state $s'$ and obtain reward $r$  given that the environment was in state $s$ and the agent took action $a$. Therefore, this probability is defined assuming action $a$ is taken. There is no probability of taking $a$ included there.
The probability of the agent taking an action is provided by the policy $\pi$, and that is why we need it in the equation.
You can think of the interaction of these two terms with the law of total probability: $p(A)=\sum _{n}p(A\mid B_{n})p(B_{n})$, where $p(B_{n})$ is analogous to $\pi(a|s)$ and $p(A\mid B_{n})$ is analogous to $p(s', r | s, a)$.


How to reduce the resolution of a Image(276*276 --> 48*48) without affecting the features in it

At least as far as I know reducing the size of images is always lossy and you can not retrieve the original image due to the fact that you are throwing away information. Although there are solutions for dimensionality reduction like PCA, they are linear approaches. It depends on your task that what you want to do but there is a solution if you have a classification problem. Try to use a spatial transformer module before the input of the predefined network and send the output of that module to your predefined network. For your grid sampler, you can set the size of the grid to $48 * 48$. For more details take a look at here.

Concretely, the grid generator first creates a normalized meshgrid of the same size as the input image U of shape $(H, W)$, that is, a set of indices $(x^t,y^t)$ that cover the whole input feature map (the subscript $t$ here stands for target coordinates in the output feature map). Then, since we’re applying an affine transformation to this grid and would like to use translations, we proceed by adding a row of ones to our coordinate vector to obtain its homogeneous equivalent. This is the little trick we also talked about last week. Finally, we reshape our 6 parameter $θ$ to a $2x3$ matrix and perform the following multiplication which results in our desired parametrised sampling grid... since bilinear interpolation is differentiable, it is perfectly suitable for the task at hand. Armed with the input feature map and our parametrised sampling grid, we proceed with bilinear sampling and obtain our output feature map $V$ of shape $(H’, W’, C’)$. Note that this implies that we can perform downsampling and upsampling by specifying the shape of our sampling grid. (take that pooling!) We definitely aren’t restricted to bilinear sampling, and there are other sampling kernels we can use, but the important takeaway is that it must be differentiable to allow the loss gradients to flow all the way back to our localisation network.


Is XGBoost better with numeric predictors?

By default, feature importance in xgboost is given by how many times a given feature appears as a split feature across all trees in the ensemble.

When one-hot encoded, each newly created dummy variable can only take the values 0 and 1, and so can only appear once in each (sub)tree. However, when combining the values into one numeric feature by giving each category a different value, the feature can appear many more times on different levels in each tree, which brings up the importance score.


Multi Class Classification on large dataset with over 600 classes

Generally I have seen SVM perform well with Text Classification tasks. Why don't you first try with taking TF-IDF in place of Word2vec.
Let's say you have a sentence D1 with terms T1,T2 and T3. Represent it as TF-IDF first and classify. This link might help https://www.analyticsvidhya.com/blog/2018/04/a-comprehensive-guide-to-understand-and-implement-text-classification-in-python/
Then you will know if the problem is with memory issues or your word2vec approach.


Deep Learning Project to Predict Stock Prices

Essentially what I would love to do is create an A.I. app that will be fed the same data that the "experts" had and see if I can create something more accurate and beat them at it. Is this a viable approach?

Sure, you can use one or more supervised learning techniques to train a model here. You have features, a target variable and ground truth for that variable.

In addition to applying ML you have learned, all you need to do to test your application fairly is reserve some of the data you have with expert predictions for comparison as test data (i.e. do not train using it).

I would caveat that with some additional thoughts:

You haven't really outlined an "approach" here, other than mentioning use of ML.
Be careful not to leak future data back into the predictive model when building a test version.
Predicting stock and markets is hard, because they react to their own predictability and many professional organisations trade on the slightest advantage they can calculate, with experienced and highly competent staff both gathering and analysing data.

Not directly part of the answer, but to anyone just starting out and discovering machine learning, and finding this Q&A:

Please don't imagine rich rewards from predicting markets using stats at home, it doesn't happen. If you think that this is a route to "beating the market" be aware that you are far from the first to think of doing this, and such a plan can be summarised like this:

Market Data + ML
???
Profit

You can fill in the ??? by learning loads about financial markets - i.e. essentially by becoming one of the experts. ML is not a short-cut, but it might be a useful tool if you are, or plan to be, a market analyst.


How to determine feature importance while using xgboost in pipeline?


Loss function and data distribution for positive / negative examples

Your approach of adding negative examples to your data set, replacing the 10-output softmax final layer with an 11-output softmax, and retraining is certainly valid.

You need these additional examples because the network can only learn to classify categories that it has seen before.

Retraining the final layer only is a good first attempt because it is the cheapest, but if you don't get good results you can try fine tuning one or more additional layers at the end of the network.

All these techniques fall under the topic of Transfer Learning.

Regarding the second part of your question about how many negative examples to include - this is known as Class Imbalance Problem in Machine Learning.

Whether your model will always predict the negative class if you include many negative examples depends on the data and the model, but it is certainly a possibility.

The safest approach would be to undersample your negative examples and include only 7000 to match the frequency of other classes in the training set. This should be a good first try, but if it doesn't work you should consider a couple of questions:

What is the expected distribution of my target population?
What is the cost of misclassifying each type of example?

And then make an informed decision about how to proceed, which may involve either sampling negative examples differently, or adjusting your performance metric to something like AUC (which can guide early-stopping, for example).

Additional material:
A systematic study of the class imbalance problem in convolutional neural networks


Usually, the setup of having two conv. layer followed by one pooling layer works well for CIFAR. Since your batch size is relatively high and the learning rate relatively low these are probably not the source for the lack of convergence. For further analysis a plot of your train and validation loss (not accuracy) would be helpful.

Therefore, I suggest the following steps:

1. Plot train and valid loss curves to understand where you're standing in terms of convergence and overfitting/underfitting, e.g. see this article on how to read loss curves.

2. If your current model capacity is insufficient train for more epochs or add another stack of layers with the same structure (features should be increasing throughout the network by for example first having two conv. layers with depth 32 and the next two conv. layers with depth 64)

3. If overfitting becomes a problem add dropout

Moreover, I do not think you need multiple fully connected layers at the end but they might not do harm either. Adam might work well the standard learning rate here but if you have manually optimized yours that is fine too (the learning rate is, in fact, the most influential hyperparameter so it almost always makes sense to optimize it).

In deep learning it usually makes a lot of sense to check what networks have worked on the task you're working on (or similar tasks). For CIFAR you find a simple but well working network here.


What is the difference between sklearn's transform() and predict() methods?

Partial Least Squares Regression is a supervised learning technique that also does dimensionality reduction. The PLSRegression class needs both transform and predict interfaces to properly implement the behavior of Partial Least Squares Regression. The transform method is the interface for dimensionality reduction. The predict method is the interface for generating targets from a trained regression model.
Not all Estimators in scikit-learn do both feature manipulation and prediction (typically target), thus not all classes are going to have both transform and predict interfaces.


Speed up Keras Model Prediction Load Times

I was able to get to work like this without flask or django. Just using default http.server in python
from http.server import BaseHTTPRequestHandler, HTTPServer
import logging
import sys
import pandas as pd
from keras.models import load_model
from urllib.parse import urlparse
model = load_model('model.h5')

class S(BaseHTTPRequestHandler):
    def _set_response(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_GET(self):
        query = urlparse(self.path).query
        params = dict(qc.split("=") for qc in query.split("&"))
        X = pd.read_csv(params["input"]).values
        prediction = model.predict(X)
        pd.DataFrame(prediction).to_json(params["output"])
        self._set_response()
        self.wfile.write("Processed".encode('utf-8'))

def run(server_class=HTTPServer, handler_class=S, port=8080):
    logging.basicConfig(level=logging.INFO)
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    logging.info('Starting httpd...\n')
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()
    logging.info('Stopping httpd...\n')

if __name__ == '__main__':
    from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()

Trigger Server using
python3.6 predict_server.py 8000

API like
http://ip/localhost:8000/?input=predict_scaled.csv&output=prediction.json


How to predict when and why of hospitalization?

Why is potentially hidden in your feature explanations.
Given that your Features give some indication on why, like Patient demographics data, or some other Features that you can include here, you can use them to answer the why.
Like this using shapley and or eli5 you can integrate it with your Standard predictor classes and for the explanations with user friendly API.
BUT i have to prefice that even tough it might be hidden it must not be at the same time, since the predictiveness of time might have Nothing to do real why you are Looking for.


Showing standard deviation for training curve

One option is to plot fewer epochs. There is no useful information after 20 epochs because training and validation performance are the same after that point.