How can you estimate the value of a component part where you only know the value of a whole?

Clearly this problem doesn't always have a unique solution, but if you are interested in finding one possible solution you could try a simple genetic algorithm simulation:

Each individual gene represents an item from the list of all possible items.
Each gene/item is assigned a price randomly at first (gene expression)
When a mutation is applied to a gene/item, its price is slightly modified randomly.
A crossover causes a "child gene" to take as value the mean of its two "parents genes".

This setting means that every individual in a population consists of all the items being assigned a particular price. At each generation each individual/assignment is evaluated by applying the prices assignment to the actual data and then measuring the error compared to the actual prices. Finally the top N individuals/assignments which perform the best are selected as parents for the next generation. Eventually the population should converge to realistic prices assignments.

I think this is a perfect case for a genetic algorithm because the evaluation of a potential price assignment is a very simple calculation, so there is no major efficiency issue repeating the process over many generations (as opposed to many problems where evaluation is prohibitively expensive).


Adding a custom constraint to weighted least squares regression model

The model you are looking for is this:

$Y=\frac{A}{1+e^{-(\beta_0+\beta_1x+\beta_2log(x))}}$, this could not be obtained but a very similar was obtained.

This code in R might work:

R=data.frame(X=c(1,2,3,4,5,6,7,8,9),Y=c(1,2,3,3,3,3,3,3,3)) # Data in which X is a line, and Y has an still unknown limit.
model=nls(formula = Y~A/(1+exp(-(b0+b1*X))),data=R)
summary(model)

In the result you can see how $A$ says that the limit of 3 (previously unknown) is calculated.

There is a limitation to take into account, is explained in this link, is summarized in the impossibility for all possible models to exist, the "most inside" model should be linear.

The model $\beta_0+\beta_1x+\beta_2log(x)$ could not be used, the model $\beta_0+\beta_1x$ could be used, take this into account.

First steps with Non-Linear Regression in R

Singular Gradient Error in nls with correct starting values


Pandas - how to make further sub-groups after groupby( ) and get_group( )

You can do that by using a combination of shift to compare the values of two consecutive rows and cumsum to produce subgroup-ids.

So the code looks like this:

# define a function that assigns subgroups
def get_time_group(ser):
    # calculate the time difference between
    # each time and the time of the previous
    # time
    # the backfill has the effect, that the first
    # row gets diff=0
    time_diff= ser - ser.shift(1).fillna(method='backfill')
    # now assign the id which is increased any time the
    # time difference between two records is greather than 60
    return (time_diff > 60).cumsum()

# group by the IP to compare the times only for the same IP
# and call the get_time_group from transform to assign the
# new group to each row (the row will be unique in combination
# with the IP
grouped= df.groupby('IP')
df['sub_group']= grouped['Unix_time'].transform(get_time_group)

After execting this code the df now looks like this:

           IP   Unix_time  sub_group
No.
1     1.1.1.1  1563552000          0
2     1.1.1.1  1563552030          0
3     1.1.1.1  1563552100          1
4     1.1.1.1  1563552110          1
5     1.1.1.1  1563552150          1
6    1.2.3.10  1563552120          0

Now you can use the sub_group as required. E.g. if you want to have the start-end of each subgroup and the number of records per subgroup, you can do:

result= df.groupby(['IP', 'sub_group']).agg({'Unix_time': ['first', 'last', 'count']})
result.columns= ['Group_start', 'Group_end', 'Num_entries']

# the result looks like this:
                    Group_start   Group_end  Num_entries
IP       sub_group
1.1.1.1  0           1563552000  1563552030            2
         1           1563552100  1563552150            3
1.2.3.10 0           1563552120  1563552120            1

The test dataframe for the output above is from your sample data, you get it by executing:

import io
import pandas as pd

raw=\
"""| No.|   IP     |      Unix_time     |    Comment
| 1  | 1.1.1.1  |     1563552000     |    # equivalent to 12:00:00 AM
| 2  | 1.1.1.1  |     1563552030     |    # equivalent to 12:00:30 AM
| 3  | 1.1.1.1  |     1563552100     |    # equivalent to 12:01:40 AM
| 4  | 1.1.1.1  |     1563552110     |    # equivalent to 12:01:50 AM
| 5  | 1.1.1.1  |     1563552150     |    # equivalent to 12:02:30 AM
| 6  | 1.2.3.10 |     1563552120     |    #"""

df= pd.read_csv(io.StringIO(raw), sep='\s*\|\s*', dtype={'Unix_time': 'Int64'}, engine='python', index_col=0, usecols=['No.', 'IP', 'Unix_time'])


key parameter in max function in Pyspark

You pass a function to the key parameter that it will virtually map your rows on to check for the maximum value. In this case you pass the str function which converts your floats to strings. Since '5.0' > '14.0' due to the nature of string comparisons, this is returned. What is usually a more likely use is using the key parameter as follows:

test = sc.parallelize([(1, 2), (4,3), (2,4)])
test.max(key = lambda x: -x[1])
(1,2)

Because of the - we sort descending and the x[1] means we use the second entry in our tuples as the key.


How can I fix this error : Python Error ""ValueError: could not broadcast input array from shape (4) into shape (1000)""

Your data generator retrieves your labels as categorical and based on the error, I assume you have 4 classes. However, in your extract_features function, you are first initializing a labels array of shape=(sample_count) instead of shape=(sample_count, 4). In your for loop, you are getting a labels_batch from the generator which apparently has a shape of (16, 4) and you are trying to save it as part of your initialized labels array which has a shape of (sample_count) instead of (sample_count, 4) and that's why it complains.

In short, you should change:

labels = np.zeros(shape=(sample_count))

to:

labels = np.zeros(shape=(sample_count, 4))


How to predict churn events that may happen within a period of time?

The problem is similar to modeling defaults of a subset of companies in a specific basket, big or small. The problem is very well-researched, drawing from methods developed in asset pricing, actuarial science and survival analysis. For a good reference, you may look into
Duffie & Singleton (2003). Credit Risk: Pricing, Measurement, and Management.
The most popular and flexible solution is modeling the churn evens as correlated counting processes with intensities that depend on the predictors that you have mentioned above + some other predictors. Counting process is an extension of Poisson process.


Finding TN,FN, TP, and FN for arrays using confusion matrix

1) Your code :

Expected inputs for confusion matrix

sklearn.metrics.confusion_matrix expect $y_{pred}$ and $y_{true}$ to be $(n\_samples,)$ shape. :

  y_true array-like of shape (n_samples,)
    Ground truth (correct) target values.

  y_pred array-like of shape (n_samples,)
    Estimated targets as returned by a classifier.

Results of argmax on your matrix :

The argmax function on your test and prediction array is :
for each row, where is the first (max element) of the row.
So let's take the first four rows of your test set :

TestArray = np.array(
[
[1,0,0,1,0,1,1,0,1,0,1,1,0,0,1,1,1,0,0,1],
[0,1,1,0,1,0,0,1,0,0,0,1,0,1,0,1,1,0,1,1],
[1,0,1,1,1,1,0,0,1,1,1,1,0,0,1,0,0,0,0,0],
[0,1,1,1,0,0,0,0,0,1,0,0,1,0,0,1,0,1,1,1],...]

Argmax returns :
[0, 1, 0, 1, ...]
Wich is the locations of the first 1 in each row. I imagine this is not what you expected.

2) What I propose :

If you want to get a confusion matrix that sums up to $(n_{rows}$ x $n_{columns})$ you have to ravel the inputs :

cm=confusion_matrix(TestArray.ravel(), PredictionArray.ravel())
TN = cm[0][0]
FN = cm[1][0]
TP = cm[1][1]
FP = cm[0][1]

print(TN,FN,TP,FP)

returns (sum up to 200) :

74 28 73 25


Neural network is not giving the expected output after training in Python

It looks like your loss is improving (the loss is decreasing every epoch), so I wouldn't say the network isn't training. Try looking at a training plot (which would have epoch on the x-axis and loss on the y-axis). See if there are any jumps or strange patterns in the training. If you don't see a roughly decreasing training loss, then you have a problem.

It's also possible that this model is just completely overfitting your data, leading to poor prediction on the test set. How many inputs do you have? Why are you using three hidden layers? Try reducing the size of your model and see if your prediction improves.


What does it mean: "Everything looks OK but loss won't decreases!"

IF EVERYTHING ELSE IS OKAY then ideally, your loss should decrease every epoch which means your model is learning and updating weights accordingly.

Steady loss might mean that your model has over-fitted data. Your model has memorized data and there is no room for improvement. Maybe

1) You have less data with very complex LSTM. Try having very simple LSTM.
2) Apply regularization layer in between.


What are the cases in which Isomap fails to do a good job?

Here is the visualization of COIL-20 data set in t-SNE paper:

The data-set consists of images of 20 objects (clusters). In all cases provided in the paper and some cases I found on the Internet (6000 MNIST data set, slide no. 31), the quality of IsoMap (2001) is not comparable to t-SNE (2008).


RandomForestClassifier OOB scoring method

In general, the performance of classifiers are compared using accuracy, this is a measure of the number of correctly classified instances divided by the total number of instances. However, from the training data we can get a better approximation of the expected error from our classifier when we are using ensemble learning or bagging techniques.

Out-of-bag error

This metric is the accuracy of examples $x_i$ using all the trees in the random forest ensemble for which it was omitted during training. Thus it kind of acts as a semi-testing instance. You can get a sense of how well your classifier can generalize using this metric.

To implement oob in sklearn you need to specify it when creating your Random Forests object as

from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 100, oob_score = True)

Then we can train the model

forest.fit(X_train, y_train)
print('Score: ', forest.score(X_train, y_train))

  Score:  0.979921928817

As expected the accuracy of the model when evaluating the training set is very high. However, this is meaningless because you can very well be overfitting your data and thus your model is rubbish. However, we can use the out-of-bag score as

print(forest.oob_score_)

  0.86453272101

This is the accuracy whilst evaluating our instances in the training set using only the trees for which they were omitted. Now let's calculate the score on the testing set as

print('Score: ', forest.score(X_test, y_test))

  Score:  0.86517733935

We see that the accuracy measured by oob is very similar to that obtained with the testing set. It thus follows through the theory that the oob accuracy is a better metric by which to evaluate the performance of your model rather than just the score. This is a consequence of bagging models and cannot be done with other types of classifiers.

Calculating oob using different metrics

Yes, you can do this! However, it depends how exactly your code is structured. I am not sure how you can include the oob and AUC all together with the cross_val_score function. However, if you are doing the cross validation folds manually you can do the following, the random forests algorithm in sklearn provides you the decision function of the oob as

print(forest.oob_decision_function_)

The class can then be obtained using

from sklearn import metrics
pred_train = np.argmax(forest.oob_decision_function_,axis=1)

Then we can calculate the AUC by using the following

metrics.roc_auc_score(y_train, pred_train)

  0.86217157846471204


Replace values that starts with "<" in a data frame in r


Text classification using deep learning with removing name, place address etc

Normally if you have enough training data this shouldn't be an issue, because the model will realize that these named entities are not relevant for classifying the text.
Anyway if you want to remove them you would need to first detect them with Named Entity Recognition, there are many implementations. Then you can simply remove them.
Using DL classification as a first option is questionable imho:

it usually requires larger training data
it takes much longer to train than traditional models
it's harder to understand what happens when things go wrong

I'm old school so I'd recommend starting with simple methods, for instance TFIDF vectors with Naive Bayes or Decision Trees.


For imbalanced classification, should the validation dataset be balanced?

Both test and validation datasets should have the same distribution. In such a case, the performance metrics on the validation dataset are a good approximation of the performance metrics on the test dataset. However, the training dataset can be different. Also, it is fine and sometimes helpful to balance the training dataset. On the other hand, balancing the test dataset could lead to a bias estimation from the performance of the model because the test dataset should reflect the original data imbalance.
As I mentioned at the beginning the test and validation datasets should have the same distribution. Since balancing the test dataset is not allowed, the validation dataset can not be validated too.

Additionally, I should mention that when you balance the test dataset, you will get a better performance in comparison to using an unbalanced dataset for testing. And of course, using a balanced test set does not make sense as explained above. So, the resulted performance is not reliable unless you use an unbalanced dataset with the same distribution of classes as the actual data.


How to identify sentiment of a given word from a sentence

One of the ways to do it is to build bag of words and apply a linear model for sentimental analysis. Then you can look at model's coefficients - positive coefficient means word is positive, negative coefficient - negative word.


Linear regression is easy.  Or you could build some self training AI NN and do it the hard way.

Engine vs fuel efficient trade off  would not be linear and there are so many other factors that go into a final trade off.

You must trade off ALL factors to optimise the system.


Check Accuracy of Model Provided by Consultant

Cross validation can be used in parameter tuning or model selection, but it does not evaluate the performance of a model.

When developing a model, you divide your data between train, validation and testing. In the best case scenario, testing is only used once at the end to score the model. You should definitely keep the 2016 data.

If you give all your data, it is easy to have a model learning "by heart" your expected data but it will not generalize well to future years. This is overfiting. The only way to know is by testing it on unknown data, here, 2016

Employed for measuring model performance, cross validation can measure more than just the average accuracy and you can select your features to answer the best accuracy score


How to propagate error back to previous layer in CNN?

Since calculating the backward gradients that too in CNN is very math heavy and it's a bit different from normal MLP as in CNN'S we have Tensors not scalar values to multiply so People use Correlation to Calculate the Convolution Operation...

Just adding some links which I refer to from time to time thinking that someday I will be able to do this from scratch (seems impossible with deepening of Layers though...)

Reference 1
Reference 2
CS231N Slides (Lecture 4-6 specially, the computational graphs construction are very helpful to understand the Back-propagation Algorithm..)

It would be great if Someone can transfer the Equations into Latex here
So that the question can be answered formally..

Based on the above Links, here's what I got,(Came to know that Convolutions are equivalent to Correlation)

Image 1

Image 2

Stack doesn't allow me uploading image so using gdrive..(do rotate one of the images, the first one actually)

Hope it will give you an idea...


What would cause a Hierarchical Cluster to look like skewed (if this is the right term to use!)?

This happens if:

There is no clear way to separate the clusters, i.e. there is only one large group of instances and the instances which are far from this group are too far from each other to form their own cluster.
The linkage criterion plays a big role: this is more likely to happen with single linkage clustering , because as the main cluster grows it becomes more likely to be the closest point to any other instance, therefore all the instances are "attached" to this cluster one by one. This is less likely to happen with complete-linkage clustering, but this option has other biases.


If you would skip this line:

X_train, X_test , y_train,y_test = train_test_split(X, Y, test_size = 0.3, random_state = 100)

and change the method parameters to train your model like this:

clf_entropy.fit(X, Y)

it should work as expected.


What are the relationships/differences between Bias, Variance and Residuals?

I think your confusion arises from mixing two different kinds of terms together here. Bias and variance are general concepts which can be measured and quantified in a number of different ways. A residual is a specific measurement of the differences between a predicted value and a true value.

Bias, loosely speaking, is how far away the average prediction is from the actual average. One way to measure it is in the difference of the means. You could also use difference of medians, difference in range, or several other calculations. To a get a complete picture of the bias of a model, you will want to look at several different measures.

Variance, when used informally in data science, is a property of single sets (whether predictions or true values). The variance of a model, loosely speaking, is how far from the average prediction a randomly selected prediction will be. It's very often assessed using cross-validation. You construct multiple models using slightly different training sets but the same algorithm and tuning parameters. You then calculate an evaluation metric for each model, and calculate the standard deviation of this evaluation over all your models. This gives you a sense of the "stability" of a given algorithm/parameter set when exposed to different training and testing sets.

(N.B. This can be confusing because there is a specific definition of "variance" used in statistics, $v = \sigma^2$. In data science, it's usually used more informally.)

Residuals are a specific quantity associated with a single prediction/true value set pair. You've got the right definition there. This makes it not a general concept, but instead a measurement that you can use to assess either bias or variance. They're also frequently used in fitting regression models and otherwise performing gradient descent-style optimization. The mean or median of a residual set can be a way to assess bias, while the standard deviation of a residual set can be used to assess a variance.


Interpreting high precision and very low recall score

All problems due to "a very imbalanced dataset". Please look at the definition of recall and precision. Based on your score I could say that you a very small set of values labeled as positive, which are classified correctly($precision=\frac{TP}{TP+FP}$). But you have a very big set of values labeled as negative, which have influence on $recall=\frac{TP}{TP+FN}$, in that way that $TP$ stays the same like in precision, but have you a lot of $FN$ values which leads to small value of recall.

At least you should resample of the dataset to make better results.


Identifying users based on smartphone data (Google)

Even I was unable to find a paper on sucha topic by Google. But, I can discuss some features which Google might be using to uniquely identify its users based on smartphone data.

Fingerprint scanning: One of the preliminary methods of scanning users as the fingerprint is unique to every individual. Also, most devices are now equipped with a fingerprint scanner.
Home location: The Google account also keeps a track of the user's home location which can't be directly used to identify a user, but could be an important feature.
Data Personalization: Suppose a user has habit of viewing videos, blogs and websites related with the top tag "artificial intellgence". This personalized information could be used to track down a user.

  Data persoanlization could be applied to apps, videos and websites with which the user interacts. This data will be super complex and probably unique to each individual.

Device Usage: Every individual likes to personalize and use his/her device in their own way. Android 9 had this feature of tracking apps which are most used by the user and also at which times. This is another great factor for identification.
Preliminary device details: IP address, the device's model, Android version, account password are also stored with the Google account.

The above features could be brought together to uniquely identify users as they provide information in different dimensions regarding the user.

The weaker features ( Data personalization, device usage ) can be used with the powerful features ( IP address, passwords ) to uniquely identify a particular user.

All the above information was not mentioned in any paper.

Hope this helps. :-)


When I selected a cell beneath my xgboost training cell, then selected: Cells --> Run All Above, the kernel would always die on the xgboost training cell. This happened probably 40-50 times in a row.

When I ran the same cells one-by-one, it completed fine on the first time I tried this and every time after.


Merging three different customer segmentation systems into one

You are supposedly supposed to merge the existing three systems.

I think it is a fairly safe bet they don't want you to add a fourth, that is unstable in the sense that as data changes, segments will change.


Pecularities of classification of Hidden Markov Models?

What you are looking for is called a metric (or distance, or similarity measure) for HMMs (some people would say SVM kernel for HMMs).

Somehow, if you have a distance, you can cluster the HMMs.

The most classical distance, defined by Rabiner and Juang is a Monte-Carlo approximation of a Kullback-Leibler divergence for HMM. It requires a long observation sequence $O$ generated from the first HMM $\lambda_1$ and is then computed as:
$$ D_{KL}(\lambda_1||\lambda_2) = 1/T * (log(P(O|\lambda_1)-log(P(O|\lambda_2))$$

The advantage of this is that it does not imply any constraint on the number of hidden state. However, $O$ needs to be a long sequence if you want accurate results.
There is some research on the topic but not too much, and even less for HMMs with different topology (which is another reason why Google does not give much results...)

Edit: This paper proposes a distance measure that does not require the number of hidden states to be the same but is only valid for uni-dimensional observations.
A new distance measure for hidden Markov models by Zeng, Duan, and Wu in Experts Systems with Applications 37, p. 1550-1555, 2010. It is rather easy to implement.


Is there any other probability distribution model than gaussian for multivariate data

There are quite a number of multivariate distributions, in theory. At the outset, we categorize them as multivariate discrete distributions and multivariate continuous distributions.

To start with, you may explore the following link for :
multivariate discrete distributions and the following for multivariate continuous distributions.

There are entire volumes printed on the subject, for example, books by:  N. Balakrishnan, Norman L Johnson, and Samuel Kotz and these books are published by John Wiley and Sons, Inc.


Should a beginner in ML use libraries like TF and Keras?

Start of by reading up on math/theory. For neural networks I can really recommend Michael Jensens free online book. All along he gives good python examples using only numpy.
For DNNs, TF and Keras are IMO the easiest approach. For linear models, scikit learn would probably cover most.


Nearest neighbors search for very high dimensional data

I hope that the following resources might get you additional ideas toward solving the problem:

1) Research paper "Efficient K-Nearest Neighbor Join Algorithms for High Dimensional Sparse Data": http://arxiv.org/abs/1011.2807

2) Class project paper "Recommendation System Based on Collaborative Filtering" (Stanford University): http://cs229.stanford.edu/proj2008/Wen-RecommendationSystemBasedOnCollaborativeFiltering.pdf

3) Project for the Netflix Prize Competition (k-NN-based): http://cs.carleton.edu/cs_comps/0910/netflixprize/final_results/knn/index.html

4) Research paper "Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data" on the curse of dimensionality phenomenon and its relation to machine learning, in general, and k-NN algorithm, in particular: http://jmlr.org/papers/volume11/radovanovic10a/radovanovic10a.pdf

5) Software for sparse k-NN classification (free, but appears not to be open source - might clarify with authors): http://www.autonlab.org/autonweb/10408.html

6) Several discussion threads on StackOverflow:

https://stackoverflow.com/questions/20333092/knn-with-big-sparse-matrices-in-python
https://stackoverflow.com/questions/18164348/efficient-nearest-neighbour-search-for-sparse-matrices
https://stackoverflow.com/questions/21085990/scipy-sparse-distance-matrix-scikit-or-scipy
https://stackoverflow.com/questions/10472681/handling-incomplete-data-data-sparsity-in-knn
https://stackoverflow.com/questions/5560218/computing-sparse-pairwise-distance-matrix-in-r (unlike all previous discussions, which refer to Python, this one refers to R ecosystem)

7) Pay attention to GraphLab, an open source parallel framework for machine learning (http://select.cs.cmu.edu/code/graphlab), which supports parallel clustering via MapReduce model: http://select.cs.cmu.edu/code/graphlab/clustering.html

You might also check my answer here on Data Science StackExchange on sparse regression for links to relevant R packages and CRAN Task View pages: https://datascience.stackexchange.com/a/918/2452.


Have you tried without the Tf-Idf weighting? What about a bi-gram analysis?

You can also review the pre processing, maybe without removing the low frequency words (wich the tf idf already does). Another idea is to remove one very high frequency word that appear on your corpora.


Seemingly good results with training a CNN but bad when testing

You model might just overfit. This is a classical case with neural networks. It means that your model is very good on training data but performs poorly on testing data, i.e. it is very bad at generalizing...

You can monitor the performance on a validation set and use dropout to circumvent overfitting...

[EDIT] As @Jan van der Vegt said, overfitting on validation set is unlikely. Something similar happened to me in the past and it turned out that I was using a very small validation set. I'm still not sure why but I had very similar performance on both train and validation sets. As soon as I increased the size of the validation set, this behaviour disappeared. Although data leakage was indeed my first guess, I could not find any error in the way datasets were created. So my advice is: make sure that you don't have data leakage between training and validation set and make sure that your validation scores can be trusted by having a large enough validation size...


Information Extraction/Semantic Search for long, unstructured documents

If you think filtering certain POS-tag words or "bad" topics from topic modelling doesn't suffice to distinguish between relevant text and junk, you can implement your custom filter. Try regex rules and use your prodigy licence to annotate junk to fit a classifier based on tf-idf or other text feature extractor.

But don't forget, there isn't a preprocessing framework that fits all needs.


Data set size versus data dimension, is there a rule of thumb?

In this video by Caltech prof. Yaser Abu-Mostafa, he explains the relationship between dimension of a dataset and it's size required for any learning model to work.

As a general rule of thumb, size of dataset should be at-least about 10x it's dimension and should be independent of the model used.

Also, this link has summaries from some of the relevant papers, viz.

  For a finite sized data with little or no a priori information, the ratio of the sample size to dimensionality must be as large as possible to suppress optimistically biased evaluations of the performance of the classifier.

This says, the ratio of size of dataset (sample) to dimension should be as large as possible to reduce classifier bias towards a particular class.

  The ratio of the sample size to dimensionality should vary inversely
  proportional to the amount of available knowledge about the class
  conditional densities.

In a classifier setting, the more knowledge we have for each class' probability density, lesser can be the sample-size to dimension ratio.

In simpler terms, we can say we should be including as much data as possible and if we are not able to do that, include as much information as possible in the small dataset itself, because for any model to work we need to feed it with high variance dataset.


Fit Decision Tree to Gradient Boosted Trees for Interpretability

Gradient boosting learns multiple decision or regression trees after each other. The difference with random forests is that the trees correct each other. Each new tree is fitted on the residual produced by the predictions from the earlier trees.

The explain method shows for each prediction (i.e. record) why a particular decision was made. This results in a set of decisions. Each tree makes several decisions. The decisions of all trees are stacked to get the set of rules that justifies a single decision. This information is specific for the gradient boosting model and the specific record.

If you want to know more about the gradient boosting model, you may want to consider feature importance. Feature importance expresses how important the features are. You can use print model.get_feature_importance() for this.


"Understanding Machine Learning: From Theory to Algorithms" the universal approximation theorem

Yes, it is applicable to a Boolean function and you can modify the result to compact set.

A result where the codomian is $[-1, 1]$ would cover regression and not restricting ourself to boolean.

You might or might not have a result for $f: \mathbb{R}^n \to \mathbb{R}$. A theorem states what conditions it requires in order to hold, for conditions that are not promised by a theorem, you have to check carefully whether the same conclusion can still apply. In particular, if the proof uses compactness, then you can't replace $[-1,1]$ with $\mathbb{R}$.


How much of a problem is each member of a batch having the same label?

If you are interested in convergence, then most probably it shouldn't converge. I used "should" because although we have an intuition of how the convergence work but no one is 200% sure of everything.
 Let's understand a few of the aspects of the whole learning process.
 - Every data point has a distinct Loss space
 - Overall Loss space will be the average of all individual Loss surface
 - We can safely assume, that the overall Loss space of the data points in one class will be closer to each other but very different when compared to the other class.
 - Lastly, the Gradient is calculated at the end of a Batch
When we have well-shuffled batch -
In this case, the Gradient after each batch will be directed towards the overall Gradient because the Gradient will be averaged. It means a smooth Loss reduction with each batch
When we have each batch of one Class -
 In this case, every alternate batch will move on two different Loss space(One of each Class).
The issue will be that we will calculate Loss for one space, but after the weight update at the end of a batch, the next batch will continue in its own space but using the weight from the previous batch. This will make learning very unpredictable.
 Also, many logic applied in optimizers will not help because Gradient will randomly change with each batch.
I tried this on MNIST digit (just using 0,1 digit) and got this Loss per Batch. I have not out enough effort on LR optimization, LR decay, etc. So, can't conclude that result will be always similar.
$\hspace{3cm}$


Can learning algorithms take in data along with their uncertainty? (chaining ML algorithms along with errors)

One option is to use Bayesian methods in machine learning.
Assume the input to any stage in a machine learning pipeline is a prior distribution and the output of any stage is a posterior distribution. Then it would be possible to chain stages together with uncertainty propagated through the system through the distributions.


Is epsilon error a standard known error or custom created by this paper?

What is that equation and is it standardly used?

I havn't seen exactly that before, so I guess it's not "standard", but it's not so unusual.  The equation is "just"  1 - (normal distribution).  If you think of the normal distribution as a measure of "how similar a sample is to the mean", then that equation turns the similarity (big is good) into a distance (small is good).
It looks related to "robust loss functions";  See below.

What is x in this equation? I thought it was some type of feature
scaling of the image but that makes no sense because the results are
all different in the table and completely out of context anyway. Is x
supposed to be the predicted age?

Your second thought is right.  $x$ is predicted age.  They use that set up since they don't have a single correct answer for their data, but rather, many guesses/votes as what the correct age is.
From the paper:

The LAP challenge evaluation employs fitting a normal distribution with the mean  μ and standard deviation σ of the votes for each image.

Say you have two images. For one, every guess you have for the age is the same.  For another, you get a wide range of guesses for the age.  The second image must be "harder" somehow.  This equation gives you an error measure that tells you how the algorithm performed relative to the many people that guessed the true age.
Robust losses
Check out this work
The "epsilon-error" equation from the paper you linked to will look sort-of like the bottom (orange) trace in Figure 1:


Is vanilla RNN suitable for time series prediction?

Absolutely yes, Vanilla RNNs can be very good predictors for time series data. You can check this detailed official TensorFlow on time series forecasting, for example.

As they said in part 1:

  the model will be given the last 20 recorded temperature observations, and needs to learn to predict the temperature at the next time step

This also gives you information about how to structure your data for training. Given an input sequence of given length, predict the next n steps of the series - where n can be 1 or more. The choice is very task-specific.

Assuming you are working with univariate time series (i.e. single series with no other variables) then you can shape your X_train as:

( number of observations , length of input series , 1 )

If you are working with multivariate datasets, then change 1 with your number of explanatory variables.

At each training iteration, the model will look at input data, and the actual outcome, and try to "learn" a mapping between x and y.


If you would have posted this in quantF SE site the most people there would have argued that it is not possible with machine learning techniques to do trading. - One can verify this because of the thread you linked in your comment. - The underlaing argument they are making have something to do with Efficient Market Hypothesis (Fama 1970) and the CAPM. You can read more about that in the original article or on wikipedia.
The point is that the EMH is a scientific hypothesis, that holds under certain circumstances. Nothing more and less. The research in the past decades proved the hypothesis in several markets and configuration. But the problem - which has been fixed by Fama later - is the size premium and the value premium. Because of this, a lot of people (e.g. Buffett and Graham) relayed on value investing (or fundamental) strategies.
The EMH additionally builds the ground for the ETF hype. - I'm not saying that this isn't working. It is just the only reliable product we have to invest in stock markets with sustainable return. - If you think about how ETFs work, you will quickly understand that one ETF build a large selection of stocks. In statistical terms: You widen your sample to get less bias, but you also get lower variance.
What AI practitioners try to do, and deeply believe is: It could be that a human is not able to beat the market by picking stocks, but a computer, which is a excellent machine for pattern recognition - would be able to do it; be able to pick a sample what represents the winners in every market situation.
For deeper diving in to this topic you can read the Article by Hsu et al. (2016). They did research about if and how this could be possible.
The beautifully that Deep Reinforcement Learning brings us is, that it learns itself to recognize the patterns hidden in the data, extract the features and learn how to take the actions. It is forecasting and action taking in one step. For a deeper understanding of these algorithms and underlaying theory I would recommend you to read the book by Sutton and Barto (2018). It is like the bible in (Deep) RL.
Additionally you should watch the Deep RL course by David Silver and the updated one by other researchers of DeepMind.
A good starting point to understand DRL for trading with code is the ICAIF 2020 paper and Github Repo by Yang et al. (2020), which I relay on for my masters thesis.
References
Fama, E. (1970). Efficient Capital Markets: A Review of Theory and Empirical Work. The Journal of Finance, 25(2), 383-417. doi:10.2307/2325486
Hsu, Ming-Wei; Lessmann, Stefan; Sung, Ming-Chien; Ma, Tiejun; Johnson, Johnnie E. V. (2016): Bridging the divide in financial market forecasting: machine learners vs. financial economists. In: Expert Systems with Applications 61, pp. 215-234.
Sutton, Richard S.; Barto, Andrew (2018): Reinforcement learning. An introduction. Second edition. Cambridge, MA, London: The MIT Press (Adaptive computation and machine learning).
Yang, Hongyang and Liu, Xiao-Yang and Zhong, Shan and Walid, Anwar, Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy (September 11, 2020). Available at SSRN: https://ssrn.com/abstract=3690996 or http://dx.doi.org/10.2139/ssrn.3690996


Can RNN be replaced with non-recurrent classifier for Sequence Classification problem?

Conceptually, the sound of dropping a metal chain on the floor is different from the sound of dropping the separated links of that chain.
Feedforward NN
In a feed forward neural network all of sequential features would be consumed independently:
$f(x)=WX+b=w_1x_1+..+w_nx_n+b$
This is all good, as far as you’re ready to sacrifice the step-wise dependency between x1, x2 etc.
Recurrent NN
To be able to utilise the temporal or sequential signal in your dataset you need a method that “chains” each feature with its past/future state, right?
We "connect" these sequential events with the alleged "hidden state":
$a_n = f(W_n, a_{n-1}, x_n)$
By exploding the above equation you can see how past information is accumulated in the subspace of the hidden state:
$a_n = f(W_n, a_{n-1}, x_n) = f(W_n, f(W_{n-1}, a_{n-2}, x_{n-1}), x_n)$, since $ a_{n-1}=f(W_n, a_{n-2}, x_n)$.
You may be able to see some issues with the above especially when it comes to sequences of larger length, usually tackled with LSTM and attention pooling architectures, but this is a different discussion!
Hope it helps.


Yarn timeline recovery not enabled error upgrading via ambari

Run the following from the ambari server's shell:

/var/lib/ambari-server/resources/scripts/configs.sh set mace gamma yarn-site "yarn.timeline-service.recovery.enabled" "true"

Where mace is the ambari server's host name and gamma is the cluster name


As RELU is not differentiable when it touches the x-axis, doesn't it effect training?

A step function is discontinuous and its first derivative is a Dirac delta function. The discontinuity causes the issue for gradient descent. Further the zero slope everywhere leads to issues when attempting to minimize the function. The function is essentially saturated for values greater than and less than zero.
By contrast RELU is continuous and only its first derivative is a discontinuous step function. Since the RELU function is continuous and well defined, gradient descent is well behaved and leads to a well behaved minimization. Further, RELU does not saturate for large values greater than zero. This is in contrast to sigmoids or tanh, which tend to saturate for large value. RELU maintains a nice linear slope as x moves toward infinity.
The issue with saturation is that gradient descent methods take a long time to find the minimum for a saturated function.
Summarizing:

Step function: discontinuous and saturated at +/- large numbers.
Tanh: Continuous and well defined, but saturated at +/- large numbers.
Sigmoid: Continuous and well defined, but saturated at +/- large numbers.
Relu: Continuous and well defined. Does not saturate at + large number.

Hope this helps!


Normalization and Outlier on Target variable which is continuous

You should do Outlier Analysis of your target variable to prepare your training data for the model. Most model would perform better on noiseless data, as Outlier might skew the findings of your model in one direction.

Generally, there is no need to perform normalization on target variable for model performance or accuracy. (Though it might be useful to do some analysis on target variable to get some useful business insights out of it)

Reasons behind performing normalization on input variables are as follows:
1) Feature scaling improves convergence of steepest descent algorithms
2) Helps to avoid a situation when several variables dominate other variables in magnitude

While if you normalize target variable, it, in turn, will normalize MSE and there will be no impact on results.

Cases when you might choose to do Normalization of Target Variables:

Neural Networks update weights of nodes based on the error generated by the cost function via back propagation, large errors might cause drastic change in the weights and could make learning process unstable. Results of which optimizer might not be able to settle on Optimal Minima.
The only time when you might want to normalize target is the case of floating point overflow. Sometimes the number is too large or too small that CPU memory can't handle it and will turn into INF or wrap-around to the other extreme representation.


You do not need a custom function to do this. Let dplyr handle it. Assuming your data is in a dataframe named df, here is what it might look like:

df %>%                                       # Set up the pipe
subset(complete.cases(df)) %>%               # Removes rows with NA values
group_by(Name) %>%                           # Groups by the Name column
count(Color) %>%                             # Counts each Color by Name, creates a new column n
mutate(max = max(n)) %>%                     # Creates a new column for the max(n) by Name
subset(n == max(n)) %>%                      # Keeps only those rows where n equals max(n)
mutate(Keep == case_when(                    # Creates a dummy logical column named 'Keep'
   n > 1 ~ TRUE,                             # That is TRUEfor n > 1 to keep ties
   n == 1 & Color == head(Color, 1) ~ TRUE,  # That is TRUE for the first row of n = 1
   TRUE ~ FALSE)) %>%                        # That is FALSE for all other cases
subset(Keep) %>%                             # Keeps only those rows where Keep is TRUE
select(Name, Mode = Color, n)                # Keeps only the Name, Color, and n columns and
                                             # renames Color as Mode

Here is the output

 # A tibble: 3 x 3
 # Groups:   Name [3]
   Name  Mode   Count
   <fct> <fct>  <int>
 1 Bob   Green      3
 2 Drew  Blue       1
 3 Jim   Blue       2
 4 Jim   Red        2

If you want a function, then wrap this up in a function definition:

my_mode_func <- function(df){
df %>%
   subset(complete.cases(df)) %>%
   group_by(Name) %>%
   count(Color) %>%
   mutate(max = max(n)) %>%
   subset(n == max) %>%
   mutate(Keep = case_when(
      n > 1 ~ TRUE,
      n == 1 & Color == head(Color,1) ~ TRUE,
      TRUE ~ FALSE)) %>%
   subset(Keep) %>%
   select(Name, Mode = Color, Count = n)
}


You are right, neg_mean_squared_error is simple -1 * mean_squared_error. This is because a convention in the Scikit-learn api that all the scorers follow.
According to scikit-learn documentation (some emphasis added):

For the most common use cases, you can designate a scorer object with the scoring parameter; the table below shows all possible values. All scorer objects follow the convention that higher return values are better than lower return values. Thus metrics which measure the distance between the model and the data, like metrics.mean_squared_error, are available as neg_mean_squared_error which return the negated value of the metric.


Binary classification: how to transform features in real numbers?

The term you are looking for is text classification. There exists a huge number of tutorials and papers out there, for example this tutorial  and this survey.


how can i fix this problem ? I tried to train a simple linear VSM classifier

Check the structure of letters_train.
SVM is for continuous and categorical variable analysis, and surely one of your variables (letter) is likely still a character variable. Change it to factor and try again.


How to identify topic transition in consecutive sentences using Python?

One solution could be to:

Get sentence embeddings from FastText
Compute Euclidean Distance between the consecutive sentences
If the distance between the consecutive sentences is close to 1, then, you may say the two sentences are talking about different topics.

See here how to compute sentence embeddings for the English language: https://github.com/facebookresearch/fastText/blob/5b5943c118b0ec5fb9cd8d20587de2b2d3966dfe/python/fasttext_module/fasttext/FastText.py#L127

fasttext.util.download_model('en', if_exists='ignore')  # English
ft = fasttext.load_model('cc.en.300.bin')
fasttext.util.reduce_model(ft, 20)
def get_fasttext_sentence_embedding(sentence, ft):
    if pd.isna(sentence):
        return np.zeros(20)
    return ft.get_sentence_vector(sentence)

Then, compute the euclidian distance between fast text embeddings of consecutive sentences.

The same can be done using LDA (topic model), but, that would require a lot of text to model the topics.


Mostly this is avoided if reasonably possible in my experience, but:

One scenario where you may want to consider binning is if you are doing modeling with a logistic model.  The coefficients indicate that per unit increase of the IV, the log odds of the dependent variable will increase by ___.  The important distinction is that we are assuming a linear relationship in the logit by saying "per unit increase, we see ___ increase", when in fact it may not be linear.  To fix this, one possible solution is two bin/discretize your IV to make the relationship more linear, and this interpretable.  This is only important if you are interested in interpretability, not just prediction.  There are also other techniques to account for this that may be preferable (like generalized additive models)

Another scenario is for certain kinds of "big" data where the signals can shift a little bit on an axis due to measurement/machine error, and binning techniques allow you to make sure that the correct signal is matched up closely to other runs where the signal may be in a slightly different position. like below:
you may bin 65,000 x-axis values into 3200 x-axis values and average their appropriate y-values.  This reduces dimensionality and accounts for the minor shifts in x-axis position that can occur.  e.g. bin 1324 will likely capture the same signal across all other runs, where if you specificed bin 23,950, it might correspond to bin 24,000 on another run of the machine/technique, and your columns wouldn't match up correctly between samples.  In the example below, by binning you are making sure each respective compound is correctly captured, considering slight x-axis variation.

Last but not least, among several other factors seen here and here, you are purposefully dropping information for the model, which means you will lose power, as you mentioned.  Also, when you are deciding a cutpoint to (for example) categorize a continuous variable as high vs low, how do you choose the cutpoint?  Can you defend your choice, or is it somewhat arbitrary?  Did you choose multiple cutpoints to see which one favored your results and thus committed the tempting garden of forking paths/multiple testing?


pre-process your documents (some of the steps may be skipped)

tokenize
remove stop words
stem or lemmatize
do normalization (e.g. U.S.A. -> USA, météo -> meteo, etc) and orthographic correction
perform phonetic normalization (e.g. with Soundex)
find equivalence classes (using thesauri, e.g. WordNet)

use a Vector Space model to represent documents (you may use TF, aforementioned TF-IDF or other models)
do the same with the query: preprocess and represent it in the vector space
find the most similar documents by computing the vector similarity (e.g. using the cosine similarity)

That's an outline of the Information Retrieval process

Introduction to Information Retrieval by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze is a very good book to get started in IR.

Or just use Apache Solr to get everything you need out of the box (or Apache Lucene, that is used by Solr, to build your own application)


The toolbox only manage the sampling so this is slightly different from the algorithm from the paper.

What it does is the following: it creates several subset of data which are balanced. These subsets are created by randomly under-sampling the majority class. That is what you are getting from the toolbox.

To obtain what in the paper, you need to train an AdaBoost classifier for each subset. Thus, what you get is an ensembles of ensembles.

Hope that's help.


Generating a sequence based on value in another column in Python

I don't know how you have tested it the first time, here is my logic. It supposes the first element in flag is 0!

df = pd.DataFrame({'memberid': [1]*11,
                   'flag': [0,0,1,1,0,1,0,0,0,1,1],
                   })
df['seq'] = ""
for i in range(0, len(df)):
    df.loc[i, 'seq'] = 1 if df.loc[i, 'flag'] == 0 else  df.loc[i - 1, 'seq'] + 1
print(df)

Another solution using lambda:

df = pd.DataFrame({'memberid': [1] * 11,
                   'flag': [0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1]
                   })

def f(flag):
    global previous_seq
    previous_seq = 1 if flag == 0 else previous_seq + 1
    return previous_seq

previous_seq = 0
df['seq'] = df[['flag']].apply(lambda x: f(*x), axis=1)
print(df)

I am not sure if it is faster than the first solution....


Does gradient boosting algorithm error always decrease faster and lower on training data?

You should worry about overfitting when the test error rate starts to go up again. Until then I would set it aside. Overfitting is rather about the number of parameters, e.g. When two models with the same performance have different number of parameters you would prefer the one with less parameters to preserve the generalisation power of the model.
Generally the training error will be lower than the test error for obvious reasons (the model specifically sees the training data but not the test data), but even this can be reversed in some cases. For example when there is very little test data, or the test data has a different distribution than the training data.
A "perfect" learning curve doesn't exist, the final performance is what counts.


What are the individual models within a machine learning ensemble called?

I am not aware of a specific definition. Wikipedia does not mention such a term either. I would prefer "components", "individual/constituent models", or something like that.

If you definitely want to find a "correct" term, one way to discover it (if it exists) is to look into an ensemble learning early paper.
To arrive at one, a good way is to search for a survey paper (e.g. here) and go on a journey to the past to find that original study.

From a quick skim of the above, you might want to look at Exploratory Data Analysis (Tukey 77), if you manage to find the pdf.


Why use both validation set and test set?

Let's assume that you are training a model whose performance depends on a set of hyperparameters. In the case of a neural network, these parameters may be for instance the learning rate or the number of training iterations.

Given a choice of hyperparameter values, you use the training set to train the model. But, how do you set the values for the hyperparameters? That's what the validation set is for. You can use it to evaluate the performance of your model for different combinations of hyperparameter values (e.g. by means of a grid search process) and keep the best trained model.

But, how does your selected model compares to other different models? Is your neural network performing better than, let's say, a random forest trained with the same combination of training/test data? You cannot compare based on the validation set, because that validation set was part of the fitting of your model. You used it to select the hyperparameter values!

The test set allows you to compare different models in an unbiased way, by basing your comparisons in data that were not use in any part of your training/hyperparameter selection process.


Machine Learning Method For Creating Chemical Formula

When picking an ML method, a great place to start is the sklearn algorithm cheat sheet:

You are predicting quantities, so regression class of variables is what you're looking for. Specifically you have multiple continuous inputs and multiple continuous outputs, which is a multivariate (linear) regression family.
As highlighted in the comments, start with a simple baseline model and build up in complexity to attempt to improve performance, towards CNNs or other.


What machine learning algorithms to use for unsupervised POS tagging?

There are no unsupervised methods to train a POS-Tagger that have similar performance to human annotations or supervised methods.

The current state-of-the-art supervised methods for training POS-Tagger are Long short-term memory (LSTM) neural networks.


Understanding Locally Weighted Linear Regression

Locally weighted linear regression is a non-parametric method for fitting data points. What does that mean?

Instead of fitting a single regression line, you fit many linear regression models. The final resulting smooth curve is the product of all those regression models.
Obviously, we can't fit the same linear model again and again. Instead, for each linear model we want to fit, we find a point x and use that for fitting a local regression model.
We find points that closest to x to fit each of our local regression model. That's why you'll see the algorithm is also known as nearest neighbours algorithm in the literature.

Now, if your data points have the x-values from 1 to 100: [1,2,3 ... 98, 99, 100]. The algorithm would fit a linear model for 1,2,3...,98,99,100. That means, you'll have 100 regression models.

Again, when we fit each of the model, we can't just use all the data points in the sample. For each of the model, we find the closest points and use that for fitting. For example, if the algorithm wants to fit for x=50, it will put higher weight on [48,49,50,51,52] and less weight on [45,46,47,53,54,55]. When it tries to fit for x=95, the points [92,93,95,96,97] will have higher weight than any other data points.

Do you see the pattern? The points closer the where you want to fit have higher weight and the points further have lower weight (zero if too far). That's what the weight function is for.

To answer your question directly:

  x are the data points for each local regression model. They are usually (but not always) the data points in your sample.


How should classification be done for a very small data set?

I don't think you need some classification algorithm, you can use your basic understanding on data/ Business Knowledge to do the classification. As the number of data points are too low, the model cannot give you good/generalised results.

Even if you try applying some complex algorithm like SVM/NN, it is of no use as the data is too low.

If you still want to apply some machine learning algorithm and then you can apply Naive Bayes, Decision Tree as these are the basic algorithms, can do the job.


How will a decision tree cope if it cannot find a suitable feature to choose as root node from which to split further?

In scikitlearn at least the code is in this file: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/_splitter.pyx.

Basically the determination of the root node is the same as the determination of any other node, it's just the first one that's done. What I think it's doing is either random splits (in which case your question is irrelevant) or "best" split, where best is determined by the evaluation of some criterion like decrease of gini impurity for each feature. In the case of a "Best" split it's evaluating the criterion for each feature in turn and keeping track of the best one, only overwriting that where a new feature's criterion evaluation is strictly better than the currently stored best feature. That strictly better evaluation means that, where two features have the same value for the criterion, the model will select the first feature it encounters as the one to split.

In other words, if Feature A and Feature B would result in an equally good split, it will set the root node to be Feature A purely because it evaluated that first.


unsupervised anomaly detection on sparse data

1- You better start with Isolation forest Isolation Forest This is a very simple algorithm where you can control the contamination rate of your data.
2- For visualization you can plot the anomalous points in red, and you can distinguish them using the Isolation Forest predict(X) function that returns -1 for outliers and 1 for the rest.
3- You can use LSTM by comparing the predicted values of your model, with the real values of your dataset, then use a KPI ( if/else depending on the max value of the difference between the predicted values and the real ones), in simple words you will need to define your own contamination rate.
4- For visualization with LSTM you can use the same function described above with Isolation Forest, but instead of the predict function that return [-1,1], you will use the function described in step 4.


What are common problems around HADOOP storage?

You can ask following questions:

Hadoop Cluster Size (No. of nodes/machines allocated for data storage)
Hardware configuration (storage capacity) for nodes/machines
Data Block Replication factor for fault tolerance
Data Compression technique used for file storage
Data Archiving process


how print f1-score with scikit´s accuracy_score or accuracy of confusion_matrix?

Use f1_score instead of the classification report:
from sklearn.metrics import f1_score

...

print('f1_score', f1_score(xgb_results, y_test))


How would you optimize this python/pandas code?

It seems you just need to merge two DFs:

df = df.merge(indicadores[['estado','cidade','population']],
              leftt_on=['address_locality','address_region'],
              right_on=['estado','cidade','city'],
              how='left')

NOTE: usually you don't need to specify left_on and right_on if the "joining" columns have the same names in both DFs (Pandas will use all common (with same names) columns
 from both DFs for joining/merging). But as i've seen from comments and from your question before you've edited it - it's not your case. So you would need to specify correct column names for left_on and right_on.

UPDATE: demo (using updated sample data sets):

In [7]: df
Out[7]:
   uf            city
0  mg        araguari
1  mg      uberlandia
2  sp       sao-paulo
3  go         goiania
4  mg  belo-horizonte

In [8]: indicators
Out[8]:
              city  uf  population
0     abadia-goias  go        6876
1  abadia-dourados  mg        6704
2         araguari  mg       19000
3           abaete  mg       22690
4       uberlandia  mg      700000

In [9]: df = df.merge(indicators,
   ...:               left_on=['city','uf'],
   ...:               right_on=['city','uf'],
   ...:               how='left')
   ...:

In [10]: df
Out[10]:
   uf            city  population
0  mg        araguari     19000.0
1  mg      uberlandia    700000.0
2  sp       sao-paulo         NaN
3  go         goiania         NaN
4  mg  belo-horizonte         NaN


How to quantify the performance of the classifier (multi-class SVM) using the test data?

Machine Learning process usually follows three steps:

Training (on training data): this data is used in the learning stage. The model optimally adjusts the weights in order to minimize the loss function over this dataset.
Evaluation (on evaluation data): it serves to fine-tune the model, for instance you can decide if your model overfits
Testing (on test data): tells you about the inference power of your model

The intersection of these three datasets has to be zero, and they should come from the same disttibution.

Recalling that, the first thing is the naming convention for your two datasets:

If training and test datasets come from the same distribution (I guess yes, since they are all Belgian sign traffics) you can name it Training and Validation datasets.
If the same condition holds, you can perform cross-validation over your Training dataset, and carry on naming it Training and Tets datasets.

  I am particularly interested detecting 8 signs. Question 1: Am I suppose to train the model using only the training set of these 8 signs or train the model with the entire training dataset and ignore the signs that I am not interested in at the detection stage?

It depends on whether the practical application you are designing the algorithm for will encounter those remaining signs. If so, I will give these signs a label, for examle 'Don't care'. The algorithm should have the capability of recognizing your 8 signs and to know that the others are of non importance, but do not fall on your desired targets. On the contrary, if the algorithm will be always fed with these 8 signs, you can use only those images. There would be other approaches: you can train a previous SVM in which you set two groups of signs (binary classifier or Support Vector Data Description), one containing the target 8, and other with the remaining ones. Then, another SVM can recognize the sign only if it has been previously classified among the 8 desired one (this SVM is only trained with the 8 signs).

  Upon training the multi-class SVM, I want to test the classifier performance using the test data. Question 2: Can someone guide me through how to do this? I found something similar to what I am looking for available here. It would be great if I can obtain quantitative data such as a confusion matrix in percentage form.

Taking it into account, I guess you have to name your datasets as Training and Evaluation. You first train the model with the training data, and then you evaluate how it generalizes with the evaluation data. First you have to decide which metrics you will use, I recommend precision and recall, per class. Once you have your trained model, you perform predictions over the evaluation test and you'll get the metrics for this dataset. For instance, if your results over the evaluation data are worse than on over the training data, your model is overfitting the training dataset.

However, I strongly recommend you to use cross-validation on the Training-dataset, visit this link


Evaluation method for multi-class classification problem modeled as binary classification problem

Unbalanced data will definitely be a problem and should be addressed. In particular, "accuracy" will not be dependable metric any more so if you decide the use unbalanced data directly, so you should use other metrics that are more reliable for such scenarios, but that also can depend on the data distribution you have. Here is a discussion of how each metric perform  for different situations.

Apart from the choice of a proper metric, There are other ways to deal with imbalanced data. Here you can find some of the methods but probably most well known way to deal with it is oversampling and undersampling, in which you equate the number of samples in each class.

Regarding the individual evaluation vs general evaluation, this probably depends on your own preference and the classes you have in the problem. But individual classification will be useful at least to fine tune your model: you would not prefer a model which performs very well on some set of samples but very badly on some other set. Such a situation might indicate that the features used might not be as useful for each class and require getting more data.

And finally, relatively small differences in the size of classes might not be a problem and ignored; but this might be more of a personal choice. What are the relative sizes of the classes?


What is the meaning of the bubbles / spikes in the shap values plot?

That's to help indicate the volume of points in that region, which would be lost due to overlap if all plotted along the horizontal line.  This kind of plot is referred to as a "bee-swarm", and is somewhat similar to violin plots, strip plots, or scatter plots with jitter.


Why ReLU is better than the other activation functions

The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by Krizhevsky et al).

But it's not the only advantage. Here is a discussion of sparsity effects of ReLu activations and induced regularization. Another nice property is that compared to tanh / sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply thresholding a matrix of activations at zero.

But I'm not convinced that great success of modern neural networks is due to ReLu alone. New initialization techniques, such as Xavier initialization, dropout and (later) batchnorm also played very important role. For example, famous AlexNet used ReLu and dropout.

So to answer your question: ReLu has very nice properties, though not ideal. But it truly proves itself when combined with other great techniques, which by the way solve non-zero-center problem that you've mentioned.

UPD: ReLu output is not zero-centered indeed and it does hurt the NN performance. But this particular issue can be tackled by other regularization techniques, e.g. batchnorm, which normalizes the signal before activation:

  We add the BN transform immediately before the nonlinearity, by
  normalizing $x = Wu+ b$. ... normalizing it is likely to produce
  activations with a stable distribution.


Negative range for binary cross entropy loss?

One thing you can do, is forcing your labels (${-1,1}$) to be ${0,1}$ using this simple linear transformation:

\begin{equation*}\hat{y} = (y + 1) / 2\end{equation*}

This way, -1 maps to 0, and 1 maps to 1.

For practical purposes, you can either change the outputs and labels of your model directly (before applying the original BCE), or slightly change to BCE loss function according to the linear transformation:

\begin{equation}H_p(q) = -\frac{1}{N} \sum_{i=1}^N \frac{(y_i+1)}{2} \log\left(p\left(y_i\right)\right) + (1-\frac{(y_i+1)}{2})\log\left(1-p\left(y_i\right)\right)\end{equation}

Notice that I changed only the coefficients, assuming that the probability function $p$ only returns meaningful values for ${-1,1}$

NOTE: obviously, all of this is relevant only for the binary cross entropy case.


Decision boundary in a classification task

For those data points, a threshold just on x1 axis would perfectly separate the two distributions. You could fit a decision stump to calculate the single parameter of the decision boundary.


Yes.
Because decision trees divide items by lines, so it does not difference how far is a point from lines.


What is the differences in the Gini Index, Chi-Square, and Information Gain splitting methods?

As I understand it, all three want to minimize the false classified data points in your data set. (Logically, if you look for what decision trees are used)

But each of them comes from another side to this problem.

gini impurity wants "better as random"

It compares the "I label random data with random labels" against the labeling after possible split by decision tree (Wish is, that you can split the tree with better outcome than "random random random")

information gain wants small trees

It uses knowledge from information theory. It models the difference between "good" and "bad" split with criteria "simple/small trees preferred". As a result of this, it want to split the data in a way, that the daughters are "pure as possible".

For the chi-square ... I have found two things: CHAID, a (seemingly complex) decision tree technique and the chi square to prune decision trees after their building.

The chi square in general has its roots in biological statistics. It gives a characteristic number how the observed distribution conform with the null hypothesis one have about this distribution. (Biology have to act like this a lot. "I observe something, I search for an explanation, I form a hypothesis, I probe if this is statistical confirmable")

For formulas please look in Wikipedia and other sources.


Welcome to the site!

I think it is called as Flow Visualization. I remember seeing something similar when I was running a Campaign using Google Analytics. That graph is used for showing the Click-Path Analysis, Visit Flow, Navigation Flow etc. You can go through this link-1, link-2 for better understanding.

The outcome of this graph was to see which path was mostly used and where can we find some room to improve it the ones which are not used mostly, finding the root cause and provide best experience to the users.


Finding and ranking best semantic matches between two sets of phrases

I'm experienced primarily in recommender systems, but I've done enough work in NLP to have some ideas on how to approach this problem.
I don't know of any formal name for the problem that you are proposing, but I do know that it's going to be really hard to train a model to learn from just that data, even if you had dense labels. There's just too much unstated human context embedded into the problem to train a model on that from scratch.
Like for your example of Fruits traditionally grown in Germany. You have to find some Knowledge graph or embedding that has learned the relationship of fruits and where they grow geographically. There is a limited set of things out there that would be capable of this.
So what you need to do is apply another model or language embedding to the data, and try to engineer a solution from that.
The first thing I thought of is any large-scale pre-trained skip-gram or CBOW embeddings (these embeddings are often called "word-vectors" or "thought vectors"). A resource to the basics of that here.
The idea here is to use some pre-trained language model and calculate embeddings for your inputs and outputs. Then you just do cosine similarity between the embeddings for each input and output, and see if you can get good matches.
Because you're using sentences you're going either going to have to use a model that embeds sentences or documents (doc2vec is an example), or you're going to have to find some way to aggregate token embeddings. I would pick the former if you try this approach.
But to build on that, once you have these pretrained embeddings, you could also train your own neural net on top of those embeddings to classify for matches. I would read up on the task of question-answer for neural networks for inspiration, as I imagine you might take some ideas from that for how to associate the queries and matches for a neural net (Google "QANet" for some leads on this).
Another approach, which depending on the data you're working with, would be a knowledge graph. This solution would be more complicated, but basically you would break up the documents into different pieces (POS/NER tagging) and then search for semantic equivalents in a knowledge graph. Some example KGs can be found here.
Another thing to google would be the field of "Ontology". Very much related to knowledge graphs but may get you some more niche results that could prove to be helpful.
Some more context about the exact problem you're trying to solve or what kind of dataset you're working with may help illuminate other solutions. Hope that some of the terms I gave you pushes you into the right direction. Good luck!


Finding the most important questions from a questionary-> results

If selecting the most important and relative questions is the question and you choose the option:

option 1.) find the closest document to each answer(30 docs) and after the documents are validated relative or not. You will have the results of each person's evaluation (Xpersons x 30 scores). Now order the questions in descending order from most scored question to less scored one.

option 2.) Summing up all the answers and finding 20 docs. After the validation of relativeness by the persons, it is not possible to find the important features(questions) because they are summed up and the information is already lost. From this result you could only find which person validates closer to your model among all, which is not the aim of the work.


Alternatives with better GPU than Google Colab Pro

Here are a few options:

Commercial cloud provider: AWS is the go-to cloud provider for virtually any need, they have options for everything. Check out their options for Deep Learning.
You could also buy your own hardware
Finally from a data science perspective there is also the option of modifying the parameters of the model and/or data so that the training fits with the hardware limitations.


What happens when you have highly correlated columns in a dataset?

Having highly correlated features is a type of redundancy in features. And yes, it effects a regression model if you are having highly correlated features. A very nice explanation is given here.

PCA is a nice choice when it comes to dimensionality reduction.


Standardization/Normalization test data in R

Check out the preProcess function from the caret library.  You can choose the parameters you want to scale/center the training data, and it also saves the transformations it makes so then you can normalize the test set with the same specifications that you normalized the training set with.  Could go something like this:
library(caret)
trainData <- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))
testData <- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))
normParam <- preProcess(trainData)
norm.testData <- predict(normParam, testData)

now your norm.testData is scaled and centered according to the training data set parameters.
Another way to do this without using caret:
## set up data
trainData <- data.frame(v1 = rnorm(15,3,1), v2 = rnorm(15,2,2))
testData <- data.frame(v1 = rnorm(5,3,1), v2 = rnorm(5,2,2))

## find mean and sd column-wise of training data
trainMean <- apply(trainData,2,mean)
trainSd <- apply(trainData,2,sd)

## centered
sweep(trainData, 2L, trainMean) # using the default "-" to subtract mean column-wise
## centered AND scaled
norm2.testData <- sweep(sweep(testData, 2L, trainMean), 2, trainSd, "/")


Fast and/or memory efficient classification solutions

LightGBM is the answer you are looking for. It uses less memory than xgboost, catboost and the fastest GBM library. Also it has a RF mode which is faster than sklearn implementation. It works on both R and Python. Also it has some parameters to control memory management.
You can check this benchmark repo.


Reducing text input size into word2vec without affecting performance too badly?

I don't think there really is a right or wrong answer to the "removing stopwords" question.  Some people will argue that throwing away information will reduce model performance, while others argue that it'll increase noise.

I personally follow a simple rule of thumb.  If my model depends on sentence structure, then I keep stopwords.  If i'm modeling topics and more interested in important phrases, then I remove them.  This seems to work well.

If you're looking for ways to reduce you matrix, then yes removing stopwords is a perfectly acceptable idea.  Another thing you can do is apply common feature reduction techniques like LSA or chi2 to find the most important words and reduce your input space to the most meaningful words.  However, doing this may dramatically effect the performance or your word2vec model.  But if it is your only choice, then why not give it a go.


Evaluate clustering by using decision tree unsupervised learning

That's a good question.

Is there such a thing? Can we use the tree to identify which combination will give us higher cluster purity?

Clustering and simple decision tree fitting are used together in many cases such as:

First, like you mentioned quality of clustering can be measured by using decision tree leafs. I heard this calculation first time (I know some other measures very similar to it) but it makes sence since it still measures how are clusters are distinct from each other and dense.
Second (and most used one), fitting a decision tree by assigning cluster labels as class labels. The fit in here should be overfit (training error should be close nearly 0). This let you when you have a new customer (let's say segmentation in e-commerce) you don't have to calculate all distances and find clusters, you just predict the new customer with the tree and assign cluster = segment label to her/him.

Also if we can do that can we just use a simple tree without even evaluate how good or bad tree is?

No, you cannot. The tree fit well (nearly overfit). Since you want to obtain cluster separation rules in your data rather than obtain a model.

And finally, as every single method is giving us different answers how can we decide and pick which one to use to pick the right combinations?

In that stage you should think why are we doing this segmentation? With all possible cluster models try to simulate your business approach and compare results.

Hope it helps!


Time series plot using netcdf data in python

You may want to try these out and see if they work for your use-case.
Plotting geospatial data in Python:
https://towardsdatascience.com/geopandas-101-plot-any-data-with-a-latitude-and-longitude-on-a-map-98e01944b972
Using matplotlib for simple animations:
https://www.geeksforgeeks.org/using-matplotlib-for-animations/


The jumps in the CDF tell you that this is a Discrete random variable as opposed to a continuous random variable. The points on the x-axis where the jumps happen are the values that the discrete random variable take. The quantum of the jump represents the probability of the random variable taking on that value.


Text categorization: combining different kind of features

If I understand correctly, you essentially have two forms of features for your models. (1) Text data that you have represented as a sparse bag of words and (2) more traditional dense features. If that is the case then there are 3 common approaches:

Perform dimensionality reduction (such as LSA via TruncatedSVD) on your sparse data to make it dense and combine the features into a single dense matrix to train your model(s).
Add your few dense features to your sparse matrix using something like scipy's hstack into a single sparse matrix to train your model(s).
Create a model using only your sparse text data and then combine its predictions (probabilities if it's classification) as a dense feature with your other dense features to create a model (ie: ensembling via stacking). If you go this route remember to only use CV predictions as features to train your model otherwise you'll likely overfit quite badly (you can make a quite class to do this all within a single Pipeline if desired).

All three approaches are valid and have their own pros and cons. Personally, I find (1) to typically be the worst because it is, relatively speaking, extremely slow. I also find (3) to usually be the best, being both sufficiently fast and resulting in very good predictions. You can obviously do a combination of them as well if you're willing to do some more extensive ensembling.

As for the algorithms you use, they can essentially all fit within that framework. Logistic regression performs surprisingly well most of the time, but others may do better depending on the problem at hand and how well you tune them. I'm partial to GBMs myself, but the bottom line is that you can try as many algorithms as you would like and even doing simple weighted ensembles of their predictions will almost always lead to a better overall solution.


Why is the accuracy of a LinearSVC not the same as the SDGClassifier?

When used with loss="hinge" The SGDClassifier gives a LinearSVM, so they should be the same. This is matter of choosing the same hyperparameters for both. Can you check that you using the exact same parameters?
As a side note (I don't know your dataset), 41.1 and 41.5 looks pretty similar, this also might be about splitting the training/testing data exactly the same and stopping the training at the correct time.


You can try to build some kind of "sliding window table". Let's say you have following attributes:

call duration (x1)
time of call (x2)
picked the phone (x3)

Let's further assume that you have data from past 3 weeks, which allows us to set following table. The rows contain individual calls, the columns the attributes. The appendix _1 tells us the time. So for example x1_1 is call duration previous week, x2_2 is time of call two weeks before etc.

client | x1_1 | x2_1 | x3_1 | x1_2 | x2_2 | x3_2 | ... | x3_3

You can train your model using historical data, where x3_3 is last week. Then, you will feed the model with current data (_3 is current week*) and try to predict x3_3 - whether the customer will pick the phone.

*I am assuming that you know who you are going to call, hence you have _3 attributes, but you don' know yet whether they respond or not

The aim is to give the model the opportunity to learn the time dependencies - maybe time of call week before together with call duration strongly correlate with future chance of picking up the phone again.

What can also help is perform feature selection. The assumption is that some attributes are strongly correlated with others, whereas others are not. You can simply use x1_1, x2_1 and see the relationship to x3_1. But I'd suggest recalculate these often as the preferences might change in time.


Simple Q-Table Learning: Understanding Example Code

My question is, why are we multiplying by 1/(i+1)? Is this supposed to be an implementation of epsilon annealing?

The code looks like a relatively ad-hoc* adjustment to ensure early exploration, and an alternative to $\epsilon$-greedy action choice. The 1/(i+1) factor is similar to decaying $\epsilon$, but not identical.

$\epsilon$-greedy with the same decay factor might look like this:

a = np.argmax(Q[s,:])
if epsilon/(1+math.sqrt(i)) > random.random():
    a = random.randrange(0, env.action_space.n)

The math.sqrt(i) is just a suggestion, but I feel that epsilon/(1+i) is probably too aggressive and would cut off exploration too quickly.

It is not something I have seen before when studying Q-Learning (e.g. in David Silver's lectures or Sutton & Barto's book). However, Q-Learning is not predicated on using any specific action choice, it just needs enough exploration in the behaviour policy. For the given problem adding some noise to a greedy selection obviously works well enough. Technically for guaranteed convergence tabular Q-Learning needs infinite exploration over infinite time steps. The code as supplied does indeed do that because the noise is unbound from the Normal distribution. So there is always some small finite chance of selecting an action with a relatively low action-value estimate and refining that estimate later.

However, the fast decay (1/episode number) and initial scaling factor for the noise are both hyperparameters that need tuning to the problem. You might prefer something more standard from the literature such as $\epsilon$-greedy, Gibbs sampling or upper-confidence-bound action selection (the example is quite similar to UCB, in that it adds to the Q-values before taking the max).

* Perhaps the approach used in the example has a name (some variation of "Noisy Action Selection") but I don't know it, and could not find it on a quick search.


scipy.spatial.distance.mahalanobis return null values for some vectors in python

It is important to include your test cases when you encounter a potential bug.

Without further information, looking at the formula of Mahanologis distance.
$$\sqrt{(u-v)^TV^{-1}(u-v)}$$
It is possible to obtain an error if $V$ is not positive definite as the input for the square root might be negative.


BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling.
BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of the masked word).
This way, with BERT you can't sample text like if it were a normal autoregressive language model. However, BERT can be seen as a Markov Random Field Language Model and be used for text generation as such. See article BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model for details. The authors released source code and a Google Colab notebook.
Update: the authors of the MRF article discovered their analysis was flawed and BERT is not a MRF, see this
Update 2: despite not being meant for next word prediction, there have been attempts at using BERT that way. Here you can find a project that does next word prediction with BERT, XLNet, RoBERTa, etc.


Setting batch size: training requires twice as much memory as validating

Aside: Tensorflow-GPU currently grabs 100% of GPU memory by design -- so that may may change the challenge a bit.

To add to @hssay on point answer -- with images, it is helpful to look at the entire pipeline.  Optimize the input pipeline, then you will use less memory.  May want to start here: TensorFlow

Test, test test.  You may find that your results do not really change much with smaller image files (less resolution, color, etc).  In some cases, results may even get better (the overtraining/undertraining balance).

[this is really should be a comment, but my rep does not transfer here :( ]


How accurate does my machine learning model need to be?

The accuracy level that should satisfy you depends on the reliability that your customer would have on it and the costs matrix of false predictions.

He should put into consideration the costs of the unpredicted failures (low recall of your model) causing him production delays and the costs of over/false maintenance alarms (low precision of your model). If you know these costs you can minimize the expected cost equation which considers the precision recall and accuracy of your model:

Expected cost
  = p(true positive) × "maintenance cost"
  + p(false positive) × "inspection cost"
  + p(true negative) × 0 (no action)
  + p(false negative) × "mechanical failure cost"


Text classification: Choosing training and testing set for each category

The most commonly used option is 2/3rd of the data as training and 1/3rd as testing sets respectively. What kind of software or tool are you using for this categorization process? For your classifier to work efficiently at the end of all the hard work, have you considered cross validating it using a multiple fold cross validation before using and assigning the training and testing sets? As in for the skewed data categories, try finding instances for the classes/ categories which have fewer samples (~10%).


Problems with Concatenating Embedded Categorical and Numerical variables for LSTM use

You should reshape your numerical layers so that they have a shape (None,4,4,1). To concatenate, you need the all but one axis to be equal.
If you use:
tf.keras.layers.Reshape((4,4,1))(num_input)

this should work.


python: why add new features of image to be worse result with xgboost

Guessing by your reported accuracies and the 93 sample points, it looks like you have just 19 samples in your validation set (the three accuracies you report then are 11/19, 13/19, and 15/19).  So the smallest change in accuracy you can get is more than 5 percentage points, and your reported swings don't seem so drastic.

However, continuous variables can lead to overfitting, if your model learns each particular value separately (which would also lead to your continuous variables getting high feature importance scores).  The default XGBClassifier parameter max_depth=3 makes that a little less likely, but the default n_estimators=100 for only 74 samples might well be at fault.  I'd agree with bradS that you should tune hyperparameters a bit.  Also, it's probably worth at least throwing something else like logistic regression at the data.  But again I'm not sure how much fine-tuning you can realistically do with just 19 validation points.


Maximum Entropy Policy Gradient Derivation

Writing the objective in a different way may be helpful.
If we only focus on the reward part and ignore the entropy part since the focus in the question is about the second summation, the original objective is
$$
J(\theta)=\sum_{t=1}^T\mathbb{E}_{(s_t,a_t)\sim q(s_t,a_t)}[r(s_t,a_t)]
$$
If we look at it from the perspective of the full trajectory $\tau$, just like the ELBO in the paper Eq.19, the objective is
$$
J(\theta)=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^Tr(s_t,a_t)\right]
$$
Then
$$
\begin{aligned}
\nabla_\theta J(\theta)&=\nabla_\theta\sum_\tau q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&=\sum_\tau \nabla_\theta q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&=\sum_\tau q_\theta(\tau)\nabla_\theta \log q_\theta(\tau)\sum_{t=1}^T r(s_t,a_t)\\
&=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\nabla_\theta \log q_\theta(\tau)\sum_{t=1}^Tr(s_t,a_t)\right]
\end{aligned}
$$
Since $q_\theta(\tau)=q(s_1)\prod_{t=1}^T q(s_{t+1}|s_t|a_t) q_\theta(a_t|s_t)$,
$$
\nabla_\theta \log q_\theta(\tau)=\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t)
$$
Then
$$
\nabla_\theta J(\theta)=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t) \sum_{t=1}^Tr(s_t,a_t)\right]
$$
That is where the second summation comes from.
Since policy at time $t'$ cannot affect reward at time $t$ when $t<t'$ (causality),
$$
\begin{aligned}
\nabla_\theta J(\theta)&=\mathbb{E}_{\tau\sim q_\theta(\tau)}\left[\sum_{t=1}^T \nabla_\theta \log q_\theta(a_t|s_t) \sum_{t'=t}^Tr(s_t',a_t')\right]\\
&=\sum_{t=1}^T \mathbb{E}_{(s_t,a_t)\sim q(s_t,a_t)}\left[\nabla_\theta \log q_\theta(a_t|s_t) \sum_{t'=t}^Tr(s_t',a_t')\right]
\end{aligned}
$$


while re-training a pre-trained model, I'm facing this issue RuntimeError: You must compile your model before using it

There 2 techniques while using the pre-train model

1. Transfer learning: In the case of transfer learning, we use the pre-train model to train on new kind of dataset so generally, remove the last layer and add a new layer into the network. At this point, as you are adding a new layer the model should be compiled. But don't worry if you are compiling it then weights will not be lost.

2. Retraining (I guess this is your case): As you are not modifying the architecture but just training it on additional dataset then also it is not going to loose weights.


How to obtain original feature names after using one-hot encoding

Consider using the one-hot encoder in category_encoders module for your encoding. It has an inverse_transform method which I believe will transform your one-hot encoded data back to its original form.


Meaning of equation for CNN probabilities

I agree the equation might not be clear, but you can decompose it into something like the following:

First, the term  $\operatorname{argmax}_k p^i (y=k|\mathbf{x})$ tells you which label has the higher probability from model $i$ given the input object $\mathbf{x}$.
Then, this "iterates" over all models in the Committee, computing for each the label that is most likely.
Finally finding which label is the most common one (that $\operatorname{argmax}_j$) at the end.

Also, it helps to think about it in pseudo-code

def get_label(CNNs, x):
    labels = [0, 0, 0, 0, 0]  # each position refers to that last $j$
    for pCNNi in CNNs:
        predictions = pCNNi(x)
        label_i = predictions.index(max(predictions))  # this is the argmax_k
        labels[label_i] += 1
    return labels.index(max(labels))  # this is the argmax_j


What classifier is the best to determine if object was detected in the correct position?

It might actually be called Intersection Over Union.

  So how do you tell if your object detection algorithm is working well? In this video, you'll learn about a function called, "Intersection Over Union". And as we use both for evaluating your object detection algorithm, as well as in the next video, using it to add another component to your object detection algorithm, to make it work even better. Let's get started. In the object detection task, you expected to localize the object as well. So if that's the ground-truth bounding box, and if your algorithm outputs this ...


How can I detect partially obscured objects using Python?

You can try CSRNet model.  The "official" repo is here.
There are a number of other implementations floating around GitHub.


They should really be more clear about what they mean, but I expect they're using a Laplacian pyramid.  As more evidence, they cite: "Denton et al "Deep generative image models using a laplacian pyramid of adversarial networks."

The idea is, store a very low resolution copy of your image, and a series of "difference" images.  Each difference image tells you what to add to the lower resolution copy to get the next higher resolution version of the image.  You can imagine that lots of values of the difference will be close to zero.

The "mean intensity" and therefore "illumination" is only really stored at that lowest resolution copy, and doesn't really (usually, they hope) affect the gradient of the image, which is what the laplacian pyramid stores.  That's why those authors say it's not sensitive to illumination changes.  Does that make sense?


Book keeping of experiment runs and results

I recently ran into a similar problem: How to manage extracting a variety of features from a large dataset, without knowing up front what all of them would be. (Even calculating mean values repeatedly would be computationally expensive.) Further, how would I manage predictions based on different feature sets? Meaning, if I added a new feature, how would I know which models to train on new features? It could quickly snowball into a huge mess.

My current solution is to track it all in a local NoSQL database (MongoDB). For example, I might have a collection features, each entry of which has a name, a description of how the feature was was calculated, the python file that ran the extraction, etc.

Likewise, a collection models includes models run on the data. Each entry might have a name, a list of features that were used to train the model, its eventual parameters, predicted values on a held-out test set, metrics for how the model performed, etc.

From my vantage point, this has a number of benefits:

By saving predictions, I can use them later in ensemble predictions.
Because I keep track of which features were used, I know which ones need retraining as I extract more features.
By saving model descriptions, I ensure that I always know what I've tried. I never have to wonder, "Have I tried LASSO with regularization parameters set by grid-search CV?" I can always look it up, and see how successful it was.

From your question, it sounds like you could adapt this approach to your problem's workflow. Install Mongo or another database of choice, and then save each experimental run, its inputs, its results, and anything else you might wish to track over the course of the project. This should be much easier to query than a spreadsheet, at the least.


An answer to the third question: "How to build trust to make use of data science for the business operations?"

It heavily depends on the type and structure of data that you want to get access and the application that you want to provide.

For example, if data owner has a concern about the privacy of its users, and you just want to execute some statistical queries over their data, you should design and propose a privacy-preserving method for accessing to their data. There are some techniques like Differential Privacy (or Randomized Response) which provide a well-defined privacy guarantee for data owner.