Instead of looping through the predictions why don't you import mean_squared_error from scikit learn's metrics module?
I'm guessing you are optimising using the mean squared log error objective.
Firstly, Log transform the Actual and predicted values and find the mean squared error
Example:
from sklearn.metrics import mean_squared_error
mean_sq_error = mean_squared_error(np.log1p(yval), np.log1p(pred))
np.log1p(x) does the same as np.log(x+1)


On which data evaluate an object detection model ? (similar or real life data ?)

The model would perform better when given similar data as its trained on.i think you should try mixing the two datasets and check the results.
To the client, you can show the three results and explain how it is a great product.
In place of mAp, show a chart of precision and recall to explain the accuracy.


The OpenCV kmeans function is equivalent to fit(). It returns the cluster centers, and you can use this to implement your own predict()function. For a new sample, calculate the distance to each of the cluster centers. The cluster with the lowest distance is the predicted class.


I'm not really sure if if P(cat) and P(dog) ~0.5 then it's an anomaly would be sound.

But what you are referring to is not a strange idea: A CNN forms a conceptual description of inputs based on patterns in previously seen data. New data that is less like the train data doens't compress and reconstruct well. The approach usually used for this is called autoencoding.

Roughly: If you would build an autoencoder by reconstructing images of cats and dogs, the in and out images would eventually (after enough data) approach each other (the distance you'd define would decrease). A new image from another domain would reconstruct less well (The distance between reconstruction and input would be larger than average).


Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

It goes like this: In the case of fine-tuning, you may want to use huge batch size, which may lead to the Out of Memory (OOM) issues. In such case, you may find the embeddings of BERT useful than fine-tuning on BERT itself. Maybe you can think of using BERT embeddings to train CNN or RNN classifier, in such case you can try to obtain the embeddings of BERT using a small batch size (it can be as low as 1), and then use these embeddings to further train your CNN or RNN classifier.


What is the relationship between the accuracy and the loss in deep learning?

There is no relationship between these two metrics.
Loss can be seen as a distance between the true values of the problem and the values predicted by the model. Greater the loss is, more huge is the errors you made on the data.
Accuracy can be seen as the number of error you made on the data.
That means:

a low accuracy and huge loss means you made huge errors on a lot of data
a low accuracy but low loss means you made little errors on a lot of data
a great accuracy with low loss means you made low errors on a few data (best case)
your situation: a great accuracy but a huge loss, means you made huge errors on a few data.

For you case, the third model can correctly predict more examples, but on those where it was wrong, it made more errors (the distance between true value and predicted values is more huge).
NOTE:
Don't forget that low or huge loss is a subjective metric, which depends on the problem and the data. It's a distance between the true value of the prediction, and the prediction made by the model. It depends also on the loss you use.
Think:

If your data are between 0 and 1, a loss of 0.5 is huge, but if your data are between 0 and 255, an error of 0.5 is low.
Maybe think of cancer detection, and probability of detecting a cancer. Maybe an error of 0.1 is huge for this problem, whereas an error f 0.1 for image classification is fine.


image classification with training set with 4 classes and test set with 3 classes

In principle there's nothing wrong with that, since every instance in the test set is predicted individually. You will have a 4 x 3 confusion matrix, because the model might predict some false positives on the fourth class. Of course you won't be able to know if the model can correctly identify a true instance from the missing class.

It depends what is the goal:

If the model is meant to be able to predict any of the 4 classes, then it should be trained on the 4 classes and it would be preferable to also test it on the 4 classes, but testing it only on 3 should already gives a good indication of its performance.
If the model only needs to predict 3 classes ever, then the instances of the 4th class should be removed from the training set since they just make things more complex.


Calculating an estimate of KL Divergence using the samples drawn from distributions

Check this article. They use k-NN to interpolate the values of P(x) and Q(x), so that you can use the KL-divergence formula with 'approximated histograms'.


Predicting on real test set gives only very high probability for 1 for a very unbalanced data

In unbalanced scenarios such as this, often the probability estimates are skewed massively in favour of the majority class, but the model is still learning to assign lower probabilities to the minority class, so it can be a good idea to choose a threshold higher than 0.5 to determine your final classification. It would be a good idea to look at a precision-recall curve (implemented in scikit-learn) which considers all thresholds between 0 and 1 and plots the precision and recall against each other for each threshold. You should pick the threshold that gives the best trade-off for your problem.


Implementation of Stochastic Gradient Descent in Python

There is only one small difference between gradient descent and stochastic gradient descent. Gradient descent calculates the gradient based on the loss function calculated across all training instances, whereas stochastic gradient descent calculates the gradient based on the loss in batches. Both of these techniques are used to find optimal parameters for a model.

Let us try to implement SGD on this 2D dataset.

The algorithm

The dataset has 2 features, however we will want to add a bias term so we append a column of ones to the end of the data matrix.

shape = x.shape
x = np.insert(x, 0, 1, axis=1)

Then we initialize our weights, there are many strategies to do this. For simplicity I will set them all to 1 however setting the initial weights randomly is probably better in order to be able to use multiple restarts.

w = np.ones((shape[1]+1,))

Our initial line looks like this

Now we will iteratively update the weights of the model if it mistakenly classifies an example.

for ix, i in enumerate(x):
   pred = np.dot(i,w)
   if pred > 0: pred = 1
   elif pred < 0: pred = -1
   if pred != y[ix]:
      w = w - learning_rate * pred * i

This line is the weight update w = w - learning_rate * pred * i.

We can see that doing this process continuously will lead to convergence.

After 10 epochs

After 20 epochs

After 50 epochs

After 100 epochs

And finally,

The code

The dataset for this code can be found here.

The function which will train the weights takes in the feature matrix $x$ and the targets $y$. It returns the trained weights $w$ and a list of historical weights encountered throughout the training process.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def get_weights(x, y, verbose = 0):
    shape = x.shape
    x = np.insert(x, 0, 1, axis=1)
    w = np.ones((shape[1]+1,))
    weights = []

    learning_rate = 10
    iteration = 0
    loss = None
    while iteration <= 1000 and loss != 0:
        for ix, i in enumerate(x):
            pred = np.dot(i,w)
            if pred > 0: pred = 1
            elif pred < 0: pred = -1
            if pred != y[ix]:
                w = w - learning_rate * pred * i
            weights.append(w)
            if verbose == 1:
                print('X_i = ', i, '    y = ', y[ix])
                print('Pred: ', pred )
                print('Weights', w)
                print('------------------------------------------')

        loss = np.dot(x, w)
        loss[loss<0] = -1
        loss[loss>0] = 1
        loss = np.sum(loss - y )

        if verbose == 1:
            print('------------------------------------------')
            print(np.sum(loss - y ))
            print('------------------------------------------')
        if iteration%10 == 0: learning_rate = learning_rate / 2
        iteration += 1
    print('Weights: ', w)
    print('Loss: ', loss)
    return w, weights

We will apply this SGD to our data in perceptron.csv.

df = np.loadtxt("perceptron.csv", delimiter = ',')
x = df[:,0:-1]
y = df[:,-1]

print('Dataset')
print(df, '\n')

w, all_weights = get_weights(x, y)
x = np.insert(x, 0, 1, axis=1)

pred = np.dot(x, w)
pred[pred > 0] =  1
pred[pred < 0] = -1
print('Predictions', pred)

Let's plot the decision boundary

x1 = np.linspace(np.amin(x[:,1]),np.amax(x[:,2]),2)
x2 = np.zeros((2,))
for ix, i in enumerate(x1):
    x2[ix] = (-w[0] - w[1]*i) / w[2]

plt.scatter(x[y>0][:,1], x[y>0][:,2], marker = 'x')
plt.scatter(x[y<0][:,1], x[y<0][:,2], marker = 'o')
plt.plot(x1,x2)
plt.title('Perceptron Seperator', fontsize=20)
plt.xlabel('Feature 1 ($x_1$)', fontsize=16)
plt.ylabel('Feature 2 ($x_2$)', fontsize=16)
plt.show()

To see the training process you can print the weights as they changed through the epochs.

for ix, w in enumerate(all_weights):
    if ix % 10 == 0:
        print('Weights:', w)
        x1 = np.linspace(np.amin(x[:,1]),np.amax(x[:,2]),2)
        x2 = np.zeros((2,))
        for ix, i in enumerate(x1):
            x2[ix] = (-w[0] - w[1]*i) / w[2]
        print('$0 = ' + str(-w[0]) + ' - ' + str(w[1]) + 'x_1'+ ' - ' + str(w[2]) + 'x_2$')

        plt.scatter(x[y>0][:,1], x[y>0][:,2], marker = 'x')
        plt.scatter(x[y<0][:,1], x[y<0][:,2], marker = 'o')
        plt.plot(x1,x2)
        plt.title('Perceptron Seperator', fontsize=20)
        plt.xlabel('Feature 1 ($x_1$)', fontsize=16)
        plt.ylabel('Feature 2 ($x_2$)', fontsize=16)
        plt.show()


Neural networks: which cost function to use?

This answer is on the general side of cost functions, not related to TensorFlow, and will mostly address the "some explanation about this topic" part of your question.

In most examples/tutorial I followed, the cost function used was somewhat arbitrary. The point was more to introduce the reader to a specific method, not to the cost function specifically. It should not stop you to follow the tutorial to be familiar with the tools, but my answer should help you on how to choose the cost function for your own problems.

If you want answers regarding Cross-Entropy, Logit, L2 norms, or anything specific, I advise you to post multiple, more specific questions. This will increase the probability that someone with specific knowledge will see your question.

Choosing the right cost function for achieving the desired result is a critical point of machine learning problems. The basic approach, if you do not know exactly what you want out of your method, is to use Mean Square Error (Wikipedia) for regression problems and Percentage of error for classification problems. However, if you want good results out of your method, you need to define good, and thus define the adequate cost function. This comes from both domain knowledge (what is your data, what are you trying to achieve), and knowledge of the tools at your disposal.

I do not believe I can guide you through the cost functions already implemented in TensorFlow, as I have very little knowledge of the tool, but I can give you an example on how to write and assess different cost functions.

To illustrate the various differences between cost functions, let us use the example of the binary classification problem, where we want, for each sample $x_n$, the class $f(x_n) \in \{0,1\}$.

Starting with computational properties; how two functions measuring the "same thing" could lead to different results. Take the following, simple cost function; the percentage of error. If you have $N$ samples, $f(y_n)$ is the predicted class and $y_n$ the true class, you want to minimize

$\frac{1}{N} \sum_n \left\{
\begin{array}{ll}
1 & \text{ if } f(x_n) \not= y_n\\
0 & \text{ otherwise}\\
\end{array} \right. = \sum_n y_n[1-f(x_n)] + [1-y_n]f(x_n)$.

This cost function has the benefit of being easily interpretable. However, it is not smooth; if you have only two samples, the function "jumps" from 0, to 0.5, to 1. This will lead to inconsistencies if you try to use gradient descent on this function. One way to avoid it is to change the cost function to use probabilities of assignment; $p(y_n = 1 | x_n)$. The function becomes

$\frac{1}{N} \sum_n y_n p(y_n = 0 | x_n) + (1 - y_n) p(y_n = 1 | x_n)$.

This function is smoother, and will work better with a gradient descent approach. You will get a 'finer' model. However, it has other problem; if you have a sample that is ambiguous, let say that you do not have enough information to say anything better than $p(y_n = 1 | x_n) = 0.5$. Then, using gradient descent on this cost function will lead to a model which increases this probability as much as possible, and thus, maybe, overfit.

Another problem of this function is that if $p(y_n = 1 | x_n) = 1$ while $y_n = 0$, you are certain to be right, but you are wrong. In order to avoid this issue, you can take the log of the probability, $\log p(y_n | x_n)$. As $\log(0) = \infty$ and $\log(1) = 0$, the following function does not have the problem described in the previous paragraph:

$\frac{1}{N} \sum_n y_n \log p(y_n = 0 | x_n) + (1 - y_n) \log p(y_n = 1 | x_n)$.

This should illustrate that in order to optimize the same thing, the percentage of error, different definitions might yield different results if they are easier to make sense of, computationally.

It is possible for cost functions $A$ and $B$ to measure the same concept, but $A$ might lead your method to better results than $B$.

Now let see how different costs function can measure different concepts. In the context of information retrieval, as in google search (if we ignore ranking), we want the returned results to

have high precision, not return irrelevant information
have high recall, return as much relevant results as possible
Precision and Recall (Wikipedia)

Note that if your algorithm returns everything, it will return every relevant result possible, and thus have high recall, but have very poor precision. On the other hand, if it returns only one element, the one that it is the most certain is relevant, it will have high precision but low recall.

In order to judge such algorithms, the common cost function is the $F$-score (Wikipedia). The common case is the $F_1$-score, which gives equal weight to precision and recall, but the general case it the $F_\beta$-score, and you can tweak $\beta$ to get

Higher recall, if you use $\beta > 1$
Higher precision, if you use $\beta < 1$.

In such scenario, choosing the cost function is choosing what trade-off your algorithm should do.

Another example that is often brought up is the case of medical diagnosis, you can choose a cost function that punishes more false negatives or false positives depending on what is preferable:

More healthy people being classified as sick (But then, we might treat healthy people, which is costly and might hurt them if they are actually not sick)
More sick people being classified as healthy (But then, they might die without treatment)

In conclusion, defining the cost function is defining the goal of your algorithm. The algorithm defines how to get there.

Side note: Some cost functions have nice algorithm ways to get to their goals. For example, a nice way to the minimum of the Hinge loss (Wikipedia) exists, by solving the dual problem in SVM (Wikipedia)


What criteria use in order to select the best internal validation for clustering?

There is no catch-all metric that can be used for evaluation (internal or otherwise) of the clustering achieved. This is why machine learning is also art. There are no hard limits, many things depend on application, domain, and data themselves.
TL;DR
The purpose of the homework is to familiarise yourself with the problem of clustering, but also with the fact that there is no definite best method nor evaluation metric as panacea for all cases.
As such, you can try various criteria (see below references) on your data and algorithm results and ponder on their effectiveness for your problem at hand.
An Impossibility Theorem for Clustering

Although the study of clustering is centered around an intuitively
compelling goal,it  has been very difficult to develop a  unified
framework for reasoning about it at a technical level, and profoundly
diverse approaches to clustering abound in the research community.
Here we suggest a formal perspective on the difficulty in  finding
such a unification, in the form of an impossibility theorem: for a set
of three simple properties, we show that there is no clustering
function satisfying all three. Relaxations of these properties expose
some of the interesting (and unavoidable) trade-offs at work in
well-studied clustering techniques such as single-linkage,
sum-of-pairs, k-means, and k-median.

Wikipedia has a nice summary of internal evaluation metrics:

Therefore, the internal evaluation measures are best suited to get
some insight into situations where one algorithm performs better than
another, but this shall not imply that one algorithm produces more
valid results than another. Validity as measured by such an index
depends on the claim that this kind of structure exists in the data
set. An algorithm designed for some kind of models has no chance if
the data set contains a radically different set of models, or if the
evaluation measures a radically different criterion. For example,
k-means clustering can only find convex clusters, and many evaluation
indexes assume convex clusters. On a data set with non-convex clusters
neither the use of k-means, nor of an evaluation criterion that
assumes convexity, is sound.
More than a dozen of internal evaluation measures exist, usually based
on the intuition that items in the same cluster should be more similar
than items in different clusters. For example, the following methods
can be used to assess the quality of clustering algorithms based on
internal criterion:
Davies–Bouldin index
The Davies–Bouldin index can be calculated by the following formula:
$$DB={\frac {1}{n}}\sum _{i=1}^{n}\max _{j\neq i}\left({\frac {\sigma _{i}+\sigma _{j}}{d(c_{i},c_{j})}}\right)$$
where $n$ is the number of clusters, $c_{x}$ is the centroid of cluster $x$, $\sigma _{x}$ is the average distance of all elements in
cluster $x$ to centroid $c_{x}$, and $d(c_{i},c_{j})$ is the distance
between centroids $c_{i}$ and $c_{j}$. Since algorithms that produce
clusters with low intra-cluster distances (high intra-cluster
similarity) and high inter-cluster distances (low inter-cluster
similarity) will have a low Davies–Bouldin index, the clustering
algorithm that produces a collection of clusters with the smallest
Davies–Bouldin index is considered the best algorithm based on this
criterion.
Dunn index
The Dunn index aims to identify dense and well-separated clusters. It is defined as the ratio between the minimal inter-cluster distance
to maximal intra-cluster distance. For each cluster partition, the
Dunn index can be calculated by the following formula:
$$D={\frac {\min _{1\leq i<j\leq n}d(i,j)}{\max _{1\leq k\leq n}d^{\prime }(k)}}\,,$$
where $d(i,j)$ represents the distance between clusters $i$ and $j$, and $d'(k)$ measures the intra-cluster distance of cluster $k$.
The inter-cluster distance $d(i,j)$ between two clusters may be any
number of distance measures, such as the distance between the
centroids of the clusters. Similarly, the intra-cluster distance
$d'(k)$ may be measured in a variety ways, such as the maximal
distance between any pair of elements in cluster $k$. Since internal
criterion seek clusters with high intra-cluster similarity and low
inter-cluster similarity, algorithms that produce clusters with high
Dunn index are more desirable.
Silhouette coefficient
The silhouette coefficient contrasts the average distance to elements in the same cluster with the average distance to elements in
other clusters. Objects with a high silhouette value are considered
well clustered, objects with a low value may be outliers. This index
works well with k-means clustering, and is also used to determine the
optimal number of clusters.

Furthermore:
An Evaluation of Criteria for Measuring the Quality of Clusters

An important problem in clustering is how to decide what is the best
set of clusters for a given data set, in terms of both the number of
clusters and the member-ship of those clusters.  In this  paper we
develop four criteria for  measuring  the quality  of different  sets
of clusters. These criteria are designed so that different criteria
prefer cluster  sets that generalise at different levels  of
granularity.  We  evaluate  the  suitability  of  these  criteria  for
non-hierarchical  clustering  of  the  results returned by a search
engine. We also compare the  number of clusters  chosen by  these
criteria with the  number of clusters  chosen by a  group of human
subjects.  Our  results  demonstrate  that  our  criteria  match  the
variability  exhibited  by  human  subjects,  indicating there is no
single perfect criterion. Instead, it is  necessary to select the
correct criterion to match a human subject's generalisation needs.

Evaluation Metrics for Unsupervised Learning Algorithms

Determining the quality of the results obtained by clustering
techniques is a key issue in unsupervised machine learning. Many
authors have discussed the desirable features of good clustering
algorithms. However, Jon Kleinberg established an impossibility
theorem for clustering. As a consequence, a wealth of studies have
proposed techniques to evaluate the quality of clustering results
depending on the characteristics of the clustering problem and the
algorithmic technique employed to cluster data.

Understanding of Internal Clustering Validation Measures

Clustering validation has long been recognized as one of the vital
issues essential to the success of clustering applications. In
general, clustering validation can be categorized into two classes,
external clustering validation and internal clustering validation. In
this paper, we focus on internal clustering validation and present a
detailed study of 11 widely used internal clustering validation
measures for crisp clustering. From five conventional aspects of
clustering, we investigate their validation properties. Experiment
results show that _ is the only internal validation measure
which performs well in all five aspects, while other measures have
certain limitations in different application scenarios.


Why logistic regression example code does not port to linear regression example?

I suspect that you are missing this mapping:

# Map predictions to outcomes (only possible outcomes are 1 and 0)
predictions[predictions > .5] = 1
predictions[predictions <=.5] = 0

Basically, what linear regression produces after a call to predict is a set of numbers (scores), then we need to manually map it back to class label. cross_validation.cross_val_score doesn't do the mapping for us, so what it ends up doing is comparing scores with class labels, which turns out to be bad.

On the other hand, predict of logistic regression gives us the class label (Reference)


News outlets tend to use "Big Data" pretty loosely.  Vendors usually provide case studies surrounding their specific products.  There aren't a lot out there for open source implementations, but they do get mentioned.  For instance, Apache isn't going to spend a lot of time building a case study on hadoop, but vendors like Cloudera and Hortonworks probably will.

Here's an example case study from Cloudera in the finance sector.

Quoting the study:

  One major global financial services conglomerate uses Cloudera and Datameer to help
  identify rogue trading activity. Teams within the firm’s asset management group are
  performing ad hoc analysis on daily feeds of price, position, and order information. Having
  ad hoc analysis to all of the detailed data allows the group to detect anomalies across
  certain asset classes and identify suspicious behavior. Users previously relied solely on
  desktop spreadsheet tools. Now, with Datameer and Cloudera, users have a powerful
  platform that allows them to sift through more data more quickly and avert potential
  losses before they begin.

.

  A leading retail bank is using Cloudera and Datameer to validate data accuracy and quality
  as required by the Dodd-Frank Act and other regulations. Integrating loan and branch data
  as well as wealth management data, the bank’s data quality initiative is responsible for
  ensuring that every record is accurate. The process includes subjectingthe data to over
  50 data sanity and quality checks. The results of those checks are trended over time to
  ensure that the tolerances for data corruption and data domains aren’t changing adversely
  and that the risk profiles being reported to investors and regulatory agencies are prudent
  and in compliance with regulatory requirements. The results are reported through a data
  quality dashboard to the Chief Risk Officer and Chief Financial Officer, who are ultimately
  responsible for ensuring the accuracy of regulatory compliance reporting as well as
  earnings forecasts to investors

I didn't see any other finance related studies at Cloudera, but I didn't search very hard.  You can have a look at their library here.

Also, Hortonworks has a case study on Trading Strategies where they saw a 20% decrease in the time it took to develop a strategy by leveraging K-means, Hadoop, and R.

These don't answer all of your questions.  I'm pretty sure both of these studies covered most of them. I don't see anything about tool selection specifically.  I imagine sales reps had a lot to do with getting the overall product in the door, but the data scientists themselves leveraged the tools they were most comfortable with.  I don't have a lot of insight into that area in the big data space.


Smart sentence segmentation not splitting on abbreviations

Neural tools trained on Universal Dependencies corpora use learned models for tokenization and sentence-spliting. Two I know of are:

UDPipe – developed at Charles University in Prague. Gets very good results (at least for parsing), but has a little unintuitive API.

Stanza – developed at Stanford University. The API is quite similar to Spacy.

However, they are quite slow compared to regex-based sentence-spliting.


In your random forest, this is due to the fact that your final model is overfitting. Sklearn's GridSearchCV has a default argument refit = True, that takes the model with the best performance based on cross-validation and retrains it in the whole dataset. Your accuracy score is very high due to the fact that it is only measured on your training data, and the best_score is a measure that incorporates how your model performs in models that it has not seen.

To wrap up, in your random forest you are overfitting very badly, as there is a big gap between your validation and training error. Try to use refit = False and you will not see this gap anymore (but you still have a problem since you are still overfitting your training set with this model).


How do i cluster binarized categorial data, without knowing the number of clusters?

For categorical data, robust hierarchical clustering algorithm ( ROCK) will work better that employs links and not distances when merging clusters, which improves quality of clusters of categorical data. Boolean and categorical are two types of attributes that are most suited in this algorithm.

ROCK is a static model that combines nearest neighbor, relocation, and hierarchical agglomerative methods. In this algorithm, cluster similarity is based on the number of points from different clusters that have neighbors in common.

You can use CBA Package in R to perform the ROCK clustering.

Algorithm Steps:

Data----->Draw Random Sample----->Cluster with Links----->Label Data in DIsk

A random sample is drawn from the database
A hierarchical clustering algorithm employing links is applied to the samples
This means: Iteratively merge clusters Ci, Cj that maximise the goodness function
merge(point1,point2) = total number of crosslinks /expected number of crosslinks
Stop merging once there are no more links between clusters or the required number of clusters has been reached.
Clusters involving only the sampled points are used to assign the remaining data points on disk to the appropriate clusters

Hope it helps!!

For more details with examples, refer the following links:
https://www.cis.upenn.edu/~sudipto/mypapers/categorical.pdf
https://en.wikibooks.org/wiki/Data_Mining_Algorithms_In_R/Clustering/RockCluster


Getting kinds on features that has high confidence on classfication

Yes you can use a decision tree for example. These trees can have a stop criterion. For example if the leaf is too small in sample size or, and this is interesting for you, is above a pre-defined purity threshold. By discarding all the final leafs that were created due to not purity related conditions you should end up with a set of rules like you described.


Predicting complete time series for given parameter sets

Check the literature on functional data. Your function here comprises the sensor values over a complete path of the robotic arm. From a random sample of such functions you are wanting to predict a new function with specified start and target positions. A good intro is Ramsay and Silverman (2005).


How does class_weight work in Decision Tree

When deciding on a split at a node, the algorithm basically calculates some metric (entropy or gini impurity) for the given node and for the two resulting left and right nodes after the split. Comparing them tells you how much the split would improve the result.

The statistics for the child nodes are weighted by the number of samples in the left and right node, respectively.

When you use sample_weight this adjusts the count and replaces it with the sum of the sample weights. class_weight gives equal sample weights for each sample based on its class according to its class proportion.

For example, the improvement in impurity is calculated as

$$ \frac{N_{parent}}{N_{total}} * (impurity_{parent} - \frac{N_{right}}{N_{parent}} * impurity_{right-child}
                                - \frac{N_{left}}{N_{parent}} * impurity_{left-child}) $$

Without class_weight or sample_weights, the $N$s are just counts. With class_weight you replace them with the corresponding weights.

The idea is the same for entropy, even though calculated differently.

source code


Sequence forecasting in keras not possible for variable-length sequence forecasts

No, it is not possible with Vanilla LSTM. It's not a Kera problem, but the fundamental structure of Vanilla LSTM.

I think you have a bit of misunderstanding of how LSTM works. You are assuming given $x_0$...$ x_{19}$ the LSTM is predicting $x_{20}$...$x_{39}$. This is NOT the case. In LSTM if your input sequence is $x_0$...$ x_{19}$ then your output sequence is $x_1$...$ x_{20}$.

To achieve what you want, you can use something like Seq2Seq model.


What is the model architectural difference between transductive GCN and inductive GraphSAGE?

The main novelty of GraphSAGE is a neighborhood sampling step (but this is independent of whether these models are used inductively or transductively). You can think of GraphSAGE as GCN with subsampled neighbors.

In practice, both can be used inductively and transductively.

The title of the GraphSAGE paper ("Inductive representation learning") is unfortunately a bit misleading in that regard. The main benefit of the sampling step of GraphSAGE is scalability (but at the cost of higher variance gradients).


What is the use of the np.ndarray() function and how is it used?

numpy.ndarray is not a function but a class, as you can see in the documentation.

An ndarray is meant to store data, it is just a multidimensional matrix. You store your matrix data in an ndarray and then you operate on it, adding to other ndarrays, or any other matrix operation you want to compute. From the documentation of ndarray:

  An ndarray is a (usually fixed-size) multidimensional container of items of the same type and size. The number of dimensions and items in an array is defined by its shape, which is a tuple of N non-negative integers that specify the sizes of each dimension. The type of items in the array is specified by a separate data-type object (dtype), one of which is associated with each ndarray.

While you can create yourself instances of numpy.ndarray by invoking its constructor, most of the times, arrays are created with the convenience function numpy.array, which ends up creating and returning an instance of numpy.ndarray. For instance, let's create a 3x3 identity matrix manually:

import numpy as np

a = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]])

And now we can compute operations on it, e.g.:

b = a + a
print(b)

The result is:

[[2 0 0]
 [0 2 0]
 [0 0 2]]


It depends what question you are trying to answer. You are looking at the rate of change of a time series, and it sounds like you are trying to show how that changed over time. The mean gives the reader one intuitive insight: they can trivially estimate the number of followers at any date $d$ days since the start by multiplying by the mean rate of change.

The downside to this single metric is that it doesn't illustrate something which is very common in series such as this: the rate of change is not fixed over time. One reasonable metric for giving readers an idea of whether the rate of change is static is giving them the median. If they know the minimum of the series (presumably zero in your case), the current value, the mean and the median, they can in many cases get a "feel for" how close to linear the increase has been.

There is a great cautionary tale in Anscombe's quartet - four completely different time series which all share several important statistical measures. Basically it always comes back to what you are trying to answer. Are you trying to find users which are likely to become prominent soon? Users which are steadily accruing followers year by year? One hit wonders? Botnets?

As you've probably guessed, this means it's not possible to universally call mean or median "better" than the other.


Can you choose a binary feature matrix for a binary classification model

I'm not aware of any literature specific to the case of classification based on binary features since it's just a subset of the general case, but it's definitely possible.
A very common example is traditional text classification, where the document is represented as a bag of words: there are different options but each word in the vocabulary can be represented as a boolean variable, representing whether it belongs to the document or not. For example (among many others), a Bernoulli Naive Bayes classifier can be trained on such data.


Fetching rules from rpart using caret package

You can use the rpart.rules function if you have used rpart to build the original tree.

library(rpart)
fit<-rpart(Reliability~.,data=car.test.frame)
rpart.rules(fit)


Best way to combine two similar document

Your description could corresponds to the NLP task of summarization. This is an active field of research: https://scholar.google.com/scholar?q=text+summarization

A much simpler option is to only extract sentences from both: in this case the goal is not to produce a text which reads like a story, just an enumeration of sentences.

Even in this case you will have to define how to measure "information value", this is not easy afaik.


Combining Machine Learning classifier with NLTK Vader for Sentiment Analysis

Interesting approach, but the whole purpose of NLTK Vader is to have a pre-trained model. After all, NLTK Vader was manually (!) labeled.

I just tested Google vs. NLTK Vader on "I did not hate this movie" (negations are notoriously hard to catch for an algorithm) and NLTK Vader did much better than Google. NLTK Vader scored it kind of positive (0.45) while Google scored it negatively (-0.6).

Interesting ... :-)

strong text

Good luck!


Some parts of the code are missing that might contain clues (e.g. you plot method would only plot random noise, as it stands). What might be happening is that you are showing the image in matplotlib in a figure that is much smaller than the image, so matplotlib will automatically scale the image to the pixel space.

The plt.imshow() method can take an argument interpolation, which can produce dramatically different effects. Here are them all compared side-by-side:

Have a look at the relevant documentation

I personally stick to using OpenCV and cv2.imread() to load images. You just have to be aware (in the context of using Tensorflow/Keras) that the channel are the first dimension. i.e. (Channels, Height, Width) - whereas e.g. plt.imread() will return (Height, Width, Channels).


Q learning Neural network Tic tac toe - When to train net

This update scheme:

Q(s,a) += reward * gamma^(inverse position in game state)

has a couple of problems:

You are - apparently - incrementing Q values rather than training them to a reference target. As a result, the estimate for Q will likely diverge, predicting total rewards that are impossibly high or low. Although in your case with a zero sum game and initial random moves, it may just random walk around zero for a long time first.
Ignoring the increment, the formula you are using is not from Q-learning, but effectively on policy Monte Carlo control, because you use the end-of-game sum of rewards as the Q value estimate. In theory, with a few tweaks this can be made to work, but it is a different algorithm than you say you want to learn.

It is worth clarifying a few related terms (you clearly know these  already, but I want to make sure you have them separated in your  understanding of the rest of the answer):

Reward. In RL, a reward (a real number) can be returned on every increment, after taking an action. The set of rewards is part of the problem definition. Often noted as $R$ or $r$.
Return (aka Utility). The sum of all - maybe discounted - rewards from a specific point. Often noted as $G$ or $U$.
Value, as in state value or action value. This is usually the expected return from a specific state or state, action pair. $Q(S_t, A_t)$ is the expected return when in state $S_t$ and taking action $A_t$. Note that using $Q$ does not make your algorithm Q-learning. The $Q$ action value is the basis for several RL algorithms.

Your formula reward * gamma^(inverse position in game state) gives you the Return, $G$ seen in a sampled training game, $G_t = \gamma^{T-t} R_T$ where $T$ is the last time step in the game. That's provided the game only has a single non-zero reward at the end - in your case that is true. So you could use it as a training example, and train your network with input $S_t, A_t$ and desired output of $G_t$ calculated in this way. That should work. However, this will only find the optimal policy if you decay the exploration parameter $\epsilon$ and also remove older history from your experience table (because the older history will estimate returns based on imperfect play).

Here is the usual way to use experience replay with Q learning:

When saving experience, store $S_t, A_t, R_{t+1}, S_{t+1}$ - note that means storing immediate Reward, not the Return (yes you will store a lot of zeroes). Also note you need to store the next state.
When you have enough experience to sample from, typically you do not learn from just one sample, but pick a minibatch size (e.g. 32) and train with that many each time. This helps with convergence.
For Q-learning, your TD target is $R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$, and you bootstrap from your current predictions for Q, which means:

For each sample in the minibatch, you need to calculate the predicted Q value of all allowed actions from the next state $S_{t+1}$ - using the neutral network. Then use the maximum value from each state to calculate $\text{max}_{a'} Q(S_{t+1}, a')$.
Train your network on the minibatch for a single step of gradient descent, with NN inputs $[S_t, A_t]$ and training label of the TD target from each example.
Yes that means you use the same network to first predict and then learn from a formula based on those predictions. This can be a source of problems, so you may need to maintain two networks, one to predict and one that learns. Every few hundred updates, refresh the prediction network as a copy of the current learning network. This is quite common addition to experience replay (it is something that Deep Mind did for DQN), although may not be necessary in your case for a game as simple as Tic Tac Toe.
The TD target is a bootstrapped and biased estimate of expected $G$. The bias is a potential source of problems (you may read that using NNs with Q-learning is not stable, this is one of the reasons why). However, with the right precautions, such as experience replay, the bias will reduce as the system learns.

In case you are wondering, it is the use of both $S_t$ (as NN input) and $S_{t+1}$ (to calculate TD target) in the Q-learning algorithm, which effectively distributes the end-of-game reward back to the start of the game's Q value.
In your case (and in many episodic games), it should be fine to use no discount, i.e. $\gamma = 1$

From your previous question, you noted that you were training two competing agents. That does in fact cause a problem for experience replay. The trouble is that the next state you need to train against will be the state after the opponent has made a move. So the opponent is technically viewed as being part of the environment for each agent. The agent learns to beat the current opponent. However, if the opponent is also learning an improved strategy, then its behaviour will change, meaning your stored experiences are no longer valid (in technical terms, the environment is non-stationary, meaning a policy that is optimal at one time may become suboptimal later). Therefore, you will want to discard older experience relatively frequently, even using Q-learning, if you have two self-modifying agents.


Measuring quality of answers from QnA systems

The ranking of the answers is part of the ML process, i.e. a system should be trained to rank the answers according to their relevance. Heuristic measures such as the ones mentioned in your question may offer decent approximations, but as you noticed they are very limited.

You may be interested in datasets and methods used in shared tasks about QA, for instance https://mrqa.github.io/shared.


Your intuition about the algorithm is about right and is a known technique from six-sigma.

As you may expect from six-sigma, take 3 times deviation. You could also take the running average and deviation to compare the new value to if you expect trends.

You need to inspect the spike but can consider removing it, which will reduce variations again.


Neural Networks: Predicting probabilities of the possible values of y, instead of just predicting y

Here are two approaches that use the idea of havng your prediction be a distribution over your continuous variables rather than a single value.

"Easy" but computationally expensive

Independently train K regressors.  You now have K samples from a distribution, and you can

Big downside - you have to train lots of networks, and run lots of networks at inference time.

For some details "Confidence and prediction intervals for neural network ensembles"

Harder

Have one network explicitly predict a parametrized (e.g. Gaussian/ mixture of gaussians) distribution.  For details see "Mixture density network", for example

So in the case of a single Gaussian, the network would output a mean and standard deviation, and so you have an estimate of the uncertainty.

A possible danger here is that you might need to be clever about how to train the variance parameter.  You could imagine a case where a network does a good job on the means but is either over- or under- confident in its predictions.


How to adapt the softmax layer for multiple labels?

I don't know what's going on in your code, but you seem to be close: to get multiple labels, simply replace the softmax output layer with a logistic layer (or something else that maps a real number to a probability), then optimize the cross entropy. That way you will have a probability associated with each label such that their sum across labels no longer need to add to unity.


How to load large csv file from web in python

Using requests

You can read the file line by line iteratively. You don't need to store the whole dataset into memory. This will only maintain a single row in memory at a time. This is very fast and memory efficient.

import requests
from contextlib import closing
import csv

url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv"

with closing(requests.get(url, stream=True)) as r:
    f = (line.decode('utf-8') for line in r.iter_lines())
    reader = csv.reader(f, delimiter=',', quotechar='"')
    for row in reader:
        print(row)

Using pandas

You can also do this with pandas, however this will store an entire chunk of the dataset in memory. However, this library is very powerful and can make your processing way easier.

import pandas as pd

url = "http://samplecsvs.s3.amazonaws.com/SalesJan2009.csv"
df = pd.read_csv(url)

print(df.head())


Multi-class classification metrics in R and Python

How about the MLmetrics package in R? It has multi class log loss function that you can use to assess the performance of your model.


Clustering with custom criterion (minimum cluster weight)

This does not work, and it's not how hierarchical clustering works.

If you stop at $n_\min$, no cluster will be larger than $2n_\min-2$ but there will either be plenty of badly clustered points, or unclustered points.

Consider the data set 0 2 3 5 with nmin=2. The first merge is (2,3) and fulfills the stopping criterion. So either you cluster this as (0), (2,3), (5) or as (0,5), (2,3) neither of which is convincing: either nmin is not a minimum size, or the clusters can be arbitrarily bad (and still may be below the minimum size).

The same concern applies to a weighted version.


semantic relation or semantic relatedness between terms or phrases

Other-than Ontologies do check word2vec, this tool provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words and can be used to find the closest words for a user-specified word


Is the output size of the last layer of a standard fully connected neural network the same as the input size?

In a Dense NN,

Input to a Layer depends on the output of the previous layer and its Neuron count
Previous Layer for the first layer is the Input Data Features
Output of a Layer is equal to the Layer's neuron count. A copy of each goes to all the Neuron of the Next Layer

It means for your case,
- First Layer will have *M(Input features)3 inputs going in i.e. M to each Neuron and 3 coming out.
 - The Second layer will have *3(Previous Layer)5 inputs going in and 5 coming out
 - The Last layer will have 5(Previous Layer)*2 inputs going in and 2 coming out
A Neural Network is a complex Tensor operation. Arrow and Circle are logical representations. Each arrow got a weight and each Circle got the Activation function and the Bias term. So, you may count this way too.
$\hspace{3cm}$
 Validate this with Keras Model.summary result which shows the parameter count.


The model of LSTM with more than one unit

The last image that you draw, where (at a given timestep) each cell of the second layer receives input from all cells in the first layer, is the right one.

You can think of recurrent layers as fully connected layers which receive sequences of input and apply their transformations at each timestep. They just receive other inputs depending on their previous inputs but the connectivity between layer is exactly the same as a fully connected : each unit of a given layer performs a weighted average of the activations of all the units in the previous layer (or of the inputs if it's the first layer).


How to assess that a cross entropy based model has converged

Pure, hardcore Stochastic Gradient Descent (when you feed just one observation at a time) is not advisable at all. The descent of the Gradient is so noisy that after a certain minimal loss reduction it will stop learning anything. It will wander around the loss function in unpredictable ways. In this case, you're right: there's no way to assess final convergence. Even if the Gradient hits the global minimum, it will jump away from it at the following iteration.

The most powerful technique is a robust version of SGD: Mini-Batch Gradient Descent. In this case, you feed a batch of data at each training iteration (usually of size between 30 to 250, but it's up to you). Batch size is another hyperparameter, therefore you must find a trade off between robustness to noise on one side, and on the other side to avoid get stuck in local minima and lower computational times.


How would you quantify an experience into a score without labeled data

To continue with your example, a domain expert (or business expert) would have to determine which of these is more important and therefore should have higher weight.

They may care about ETAs for the order because their customers appreciate getting a fast delivery, but may not care as much about the cancellations due to a high amount of orders they have anyways. Now this is purely guessing.

In order to build a model that will suit the business's needs - this should ideally be discussed with someone who understands the domain well and knows what they want to achieve.


Based on a quick read of the paper linked in the comment:

I just don't understand if your model (BERT or not) is only trained on these weak labels then you are treating them as "ground-truth",

Correct, but only for the training: the model is trained to recognize the labels obtained with a "quick and dirty" method.

and more importantly, don't you already know how to create "ground-truth" (by the rules-based system) ??? What's the point of the 2nd step?

No, because the real ground truth they are interested in is not those from the "quick and dirty" method. If they were, it would indeed be sufficient to run their rule-based system. The goal is to predict the labels obtained in what the authors call the "Gold Standard Corpus", which was manually annotated and never seen by the model.
Typically the quick and dirty method will result in some classification errors. The point of tuning the model with these labels is to see if the model can extrapolate from these low-quality labels to high-quality labels. This ability to generalize beyond the training data is based on the underlying semantic information contained in the original BERT-like model. For example this model might be able to associate a specific sport like swimming with "physical activity", even though the weak supervision doesn't contain this association.


How is hinge loss related to primal form / dual form of SVM

Hinge loss for sample point $i$: $$l( y_i, z_i) = \max(0, 1-y_iz_i)$$

Let $z_i=w^Tx_i+b$.

We want to minimize

$$\min \frac1n \sum_{i=1}^nl(y_i, w^Tx_i+b)+\|w\|^2$$

which can be written as

$$\min \frac1n \sum_{i=1}^n\max(0,1-y_i (w^Tx_i+b))+\|w\|^2$$

which can be written as

$$\min \frac1n \sum_{i=1}^n \zeta_i + \|w\|^2$$

subject to $$\zeta_i \ge 0$$

$$\zeta_i \ge 1-y_i (w^Tx_i+b)$$

The constraint comes from hinge loss. It is a reformulation of a minimax optimization problem.


One hot encoding error "sort.list(y)..."

I may be missing the point, but you don't need to do this one variable at a time in a loop and then bind the columns together - that's not the R way.  Here's an alternative:

# create some simulated data
n <- 10000
X <- data.frame(f_a_ = factor(sample(1:400, n, replace = TRUE)),
                f_b_ = factor(sample(1:60, n, replace = TRUE)),
                f_c_ = factor(sample(1:6, n, replace = TRUE)),
                f_d_ = factor(sample(1:5, n, replace = TRUE)),
                y = rep(1, n))

# use R's inbuilt understanding of formulas to do the "one hot encoding" for you
x_onehot <- model.matrix(y ~ ., data = X)


Estimate the location of an object in a field using computer vision and math

Your problem is focused on the object detection perspective. For a intro into the topic with some code you could look into this - https://medium.com/analytics-vidhya/detecting-objects-on-ip-camera-video-with-tensorflow-and-opencv-e2c25297a75a.
General object detection is quite easy for the second picture but for the first it would be a bit difficult. You could use dlib and CNN networks such as YOLO and RCNN (family of models).
Yes you could do it with two cameras. The configuration depends on you whether you use selected area for both cameras or both cameras cover the entire field.
https://www.geeksforgeeks.org/detect-an-object-with-opencv-python/ - The code here detects stop signs. You could find a dataset which deals with footballs/sports and do the exact same.
This link provides your objective for close range (zoomed in) photos - https://towardsdatascience.com/yolov2-to-detect-your-own-objects-soccer-ball-using-darkflow-a4f98d5ce5bf


Please help select an Algo based on Accuracy and Confusion Matrix

There must be definitely some leakage in the data, which is why the Logistic Regression is predicting almost all as the positive class. You may want to check on that first before you proceed. The only reason Random Forest may seem better in that case is that it, by default, only takes a subset of variables.


It is true that for training a lot of the parallalization can be exploited by the GPU's, resulting in much faster training. For Inference, this parallalization can be way less, however CNN's will still get an advantage from this resulting in faster inference. Now you just have to ask yourself: is faster inference important? Do I want this extra dependencies (a good GPU, the right files installed etc)?
If speed is not an issue, go for CPU. However note that GPU's can make it an order of magnitude faster in my experience.


clipping the reward for adam optimizer in keras

Gradient clipping takes two main forms in Keras: gradient norm scaling (clipnorm) and gradient value clipping (clipvalue).

1. Gradient Norm Scaling

  Gradient norm scaling involves changing the derivatives of the loss function to have a given vector norm when the L2 vector norm (sum of the squared values) of the gradient vector exceeds a threshold value.
  For example, we could specify a norm of 1.0, meaning that if the vector norm for a gradient exceeds 1.0, then the values in the vector will be rescaled so that the norm of the vector equals 1.0.

2. Gradient Value Clipping

  Gradient value clipping involves clipping the derivatives of the loss function to have a given value if a gradient value is less than a negative threshold or more than the positive threshold. For example, we could specify a norm of 0.5, meaning that if a gradient value was less than -0.5, it is set to -0.5 and if it is more than 0.5, then it will be set to 0.5.

3. Keras Syntax

The gradient clipping syntax for Adaptive Moment Estimation (Adam) is very simple and follows the same syntax as for Stochastic Gradient Descent (SGD) shown above:

opt_adam = optimizers.adam(clipnorm=1.)
opt_adam = optimizers.adam(clipvalue=0.5)


Clarification on the Keras Recurrent Unit Cell

The notation used for LSTM is quite confusing, and this took me some time to get my head around this as well. When you see this graphic (often used to explain RNNs):

You need to consider that X is a sequence of data the timestep t, it's not just a single scalar input value (as we're used to with feed-forward networks); it's an array / tensor of data. The diagram shows how there is an output at time-step t, but that it then also feeds into the next time-step, t+1, when the next array/tensor is then fed in.

A better/clearer way (in my opinion) is to look at it is like this:

So a LSTM 'cell' is actually what you might consider a layer. And a unit (one of the circles) is one of these:

which you can consider a neuron in a hidden layer.

Initially, I thought this was a cell, but it isn't, it's a single unit. So when you specify 32 units, for example, you're actually saying how many of these units (neurons) you want in the cell (layer).

This is what gives the model its capacity to learn the data that's presented to it. And just like hidden layers/neurons in a feed-forward, it's a hyperparameter that you'll need to experiment with; too low and the model will under-fit, too high and it will over-fit.


Do you have any information about your data set? Is it sparse, will most similarities be zero? Is the total dictionary small? You could consider a inverted index. For example

word  query_id
W1    [1, 3, 6]
W2    [2, 5]
W3    [1, 3, 4]
W4    [2, 3, 4]
W5    [2, 3, 6]

query_id  query
1         W1 W3
2         W2 W4 W4
3         W1 W3 W4 W5
4         W3 W4
5         W2
6         W1 W5

Here W_i is a word, e.g. birthday and query_id is the id of the query in the database. e.g. {how, are, you} might have id 22. Now you get a query {W1 W3 W5}. Aggregate counts on the inverted index. W1 was seen in queries 1, 3, and 6. W3 in 1, 3, and 4, etc.

query_id  count
1         2
2         1
3         3
4         1
6         2

The count will the number of words in common with the incoming query, this is the numerator of the jaccard similarity. So, to find the top k you can start with the queries with the highest count. query_id 3 has the highest count and its similarity is 3/4.

If you have a massive database there are techniques like locality sensitive hashing which will basically reduce the search space into a small bucket. The incoming query gets hashed and lands in a bucket. You can then do a linear search with all the queries in this bucket to find the nearest k.


If the characters in x are separated by spaces, they work the same way. If there are multiple whitespaces between the chacacters, they work differently as str(x).split(' ') splits specifically by each space and str(x).split() splits by consecutive whitespace.

Example:

test_cases = ["a b c", "a   b   c"]
for x in test_cases:
    splitted_by_space = x.split(" ")
    splitted_by_consecutive_ws = x.split()
    print("x =", '"' + x + '"')
    print('x.split(" ") -->',  splitted_by_space)
    print('x.split()    -->',  splitted_by_consecutive_ws)
    print("-" * 50)

Output:

x = "a b c"
x.split(" ") --> ['a', 'b', 'c']
x.split()    --> ['a', 'b', 'c']
--------------------------------------------------
x = "a   b   c"
x.split(" ") --> ['a', '', '', 'b', '', '', 'c']
x.split()    --> ['a', 'b', 'c']
--------------------------------------------------

From the official documentation:

  str.split(sep=None, maxsplit=-1)

  If sep is not specified or is None, a different splitting algorithm is applied: runs of consecutive whitespace are regarded as a single separator, and the result will contain no empty strings at the start or end if the string has leading or trailing whitespace. Consequently, splitting an empty string or a string consisting of just whitespace with a None separator returns [].


Input layer is incompatible even when dimensions (apparently) match

You set the input shape to (1500, 2) whereas your data only contains a single feature. You should therefore change the shape to (1,) or (None, 1) to match the shape of the input data.


Is there an encoder which can automatically detect the intrinsic order of an ordinal variable and assign values accordingly?

Yup, it should be target encoding. By calculating the target mean for each category, you are ordering the categories based on the target, I don't think of a best way to order them. Of course, this is only valid in a supervised learning setting. If not, I can't think of a way of automatically ordering categories.
See this question and this blogpost to dig deeper in target encoding.


1 and 2: If you rescale you images, you should do it on all partitions: training, validation and test. If you only rescale your images on the training set, then your network will see very different values (0~255, vs 0.0~1.0) on validation/test set and therefore give poor accuracy. That's your case 2.
I don't see any obvious problem.


After running K-means on 12 features, I get an array containing clusters for each row. What is the next step after this?

You can run your K-Means for different values of K (e.g. k=2, 3, ..., N) and for each iteration you can compute the Silhouette metric, which provides a measurement on "how compact" your clusters are. You can plot your Silhouette score and find what is the K that makes the score minimum, or stable. Since this is a clustering operation and K-Means is a simple method (nothing bad about that), I would recommend using a second clustering method (e.g. t-SNE, DBSCAN) and apply a similar procedure to see whether there is some agreement between the models. This is a general framework in data science, just applied to a clustering problem.


Is it possible to make tensorflow print out everything it see in a given image and not just the top five results?

Sounds like you want to see the predicted probabilities of a softmax function. You can assign the values to a list during training so you can see probabilities of each epoch if you'd like. As is written below, in_top_k will select the single top prediction of the softmax cross entropy function but if you have multiple targets in each picture you will want to change that "1" to the desired amount.

xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits = logits,
           labels = y)

logit = tf.nn.in_top_k(logits, y, 1)

y_one_prob = tf.sigmoid(logit)


How to create a stacked bar chart with gaps between values of each variables in Python

You may apply Wolfram Language to your project. There is a free Wolfram Engine for developers and with the Wolfram Client Library for Python you can use these functions in Python.

from wolframclient.evaluation import WolframLanguageSession
from wolframclient.language import wl, wlexpr

Start the session.

wolfSession = WolframLanguageSession();
A=[3,5,7];
B=[4,5,7];
C=[2,3,4,5,6,7];

Either the ArrayPlot or MatrixPlot functions can be applied. These functions take 2D matrices so first need to convert the lists.

With SparseArray a matrix can be specified without explicitly setting values for all positions. Empty positions are set to '' below.

pos=wolfSession.evaluate(
    wl.Flatten(
        wl.MapIndexed(
            wl.Function(wl.Rule([wl.First(wl.Slot(2)), wl.Slot(1)], wl.First(wl.Slot(2)))),
            [A,B,C],
            [2]
        ),
        1
    )
);
sparse = wolfSession.evaluate(wl.SparseArray(pos, wl.Automatic, ''));

print(wolfSession.evaluate(wl.Normal(sparse)))

[['', '', 1, '', 1, '', 1],
 ['', '', '', 2, 2, '', 2],
 ['', 3, 3, 3, 3, 3, 3]]

Using custom ColorRules and FrameTicks to get a basic representation of your plot.  You can use these and many other options to get you precise look you

stacked=wolfSession.evaluate(
    wl.ArrayPlot(
        sparse,
        AspectRatio = 1/4,
        ColorRules = wl.Flatten(
            [wl.Thread(wl.Rule(wl.Range(3),[wl.Green,wl.Orange,wl.Blue])),
            wl.Rule(wl.Blank(),wl.White)]
        ),
        FrameTicks = [
            [wl.Thread([range(1,4), ['A','B','C']]), wlexpr('None')],
            [
                wl.Thread([
                    wl.Range(7),
                    list(map(lambda r: str(r)+'k', range(1,8)))
                ]),
                wlexpr('None')
            ]
        ]
    )
);

Then Export in one of the supported Raster Image Formats or Vector Graphics Formats.

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        stacked
    )
)

Terminate session

wolfSession.terminate()

Hope this helps.


How to apply .replace() inside a column in a pandas data frame to clean data

EDITED to include a working example.

I have made an example with string type numeric data. I have also include a poorly formatted string number 1,002,*8320.

The code below will convert the numbers in the format of digits and digits with commas to floats, and convert all other strings to NaN. I have used a regular expression to replace commas with blanks.

The NaN must be used over "" since the comparison operator will not work on strings

import pandas as pd
import numpy as np
import re

# example data frame
df2 = pd.DataFrame([['1', '2000000000'],
                    ['2', '1,002,*8320'],
                    ['3', '1,000,000']],
                   columns = ['idx','product_ranking'])
# Remove Commas
df2['product_ranking'] = df2['product_ranking'].map(lambda x: re.sub('[,]*' , '', x))

# Remove Convert strings that are numeric into floats.
df2['product_ranking'] = df2['product_ranking'].map(lambda x: float(x) if x.isnumeric() else np.nan)

#Comparison is working
df2['product_ranking'] > 100000000


Tools for automatic anomaly detection on a SQL table?

If you need SQL code that runs various outlier detection methods against any arbitrary table, check out my series of articles and code samples geared towards SQL Server. I provide some preliminary code for Grubb's Tests, Z-Scores and Modified Z-Scores, Interquartile Range, Dixon's Q-Test, GESD, the Tietjen-Moore Test, Pierce's Criterion, Chauvenet's Criterion, the Modified Thompson Tau Test, Cook's Distance, Mahalanobis Distance, Benford's Law and various visual plots. Please note that I'm an amateur at this and that this is some of my older code, so I'm sure corrections will be needed; this at least provides a starting point so that you can run these tests against any database table you have permissions for. It's also in T-SQL, so you'll need to adjust the code for PL/SQL or whatever other brand of SQL you might be using, if you happen to be on some other platform like Oracle or MySql. This should get you started though. Just works out the kinks and set the stored procedures to run periodically on a schedule and you're good to go. Let me know if you have any feedback on the routines, I'm always looking to improve my SQL writing.


Doesn't over(/under)sampling an imbalanced dataset cause issues?

Yes, the classifier will expect the relative class frequencies in operation to be the same as those in the training set.  This means that if you over-sample the minority class in the training set, the classifier is likely to over-predict that class in operational use.
To see why it is best to consider probabilistic classifiers, where the decision is based on the posterior probability of class membership p(C_i|x), but this can be written using Bayes' rule as
$p(C_i|x) = \frac{p(x|C_i)p(C_i)}{p(x)}\qquad$ where $\qquad p(x) = \sum_j p(x|C_j)p(c_j)$,
so we can see that the decision depends on the prior probabilities of the classes, $p(C_i)$, so if the prior probabilities in the training set are different than those in operation, the operational performance of our classifier will be suboptimal, even if it is optimal for the training set conditions.
Some classifiers have a problem learning from imbalanced datasets, so one solution is to oversample the classes to ameliorate this bias in the classifier.  There are to approaches.  The first is to oversample by just the right amount to overcome this (usually unknown) bias and no more, but that is really difficult.  The other approach is to balance the training set and then post-process the output to compensate for the difference in training set and operational priors.  We take the output of the classifier trained on an oversampled dataset and multiply by the ratio of operational and training set prior probabilities,
$q_o(C_i|x) \propto p_t(x|C_i)p_t(C_i) \times \frac{p_o(C_i)}{p_t(C_i} = p_t(x|C_i)p_o(C_i)$
Quantities with the o subscript relate to operational conditions and those wit the t subscript relate to training set conditions.  I have written this as $q_o(C_i|x)$ as it is an un-normalised probability, but it is straight forward to renormalise them by dividing by the sum of $q_o(C_i|x)$ over all classes.  For some problems it may be better to use cross-validation to chose the correction factor, rather than the theoretical value used here, as it depends on the bias in the classifier due to the imbalance.
So in short, for imbalanced datasets, use a probabilistic classifier and oversample (or reweight) to get a balanced dataset, in order to overcome the bias a classifier may have for imbalanced datasets.  Then post-process the output of the classifier so that it doesn't over-predict the minority class in operation.


Since your data is sequential, you could try with sequential models (LSTM, RNN, GRU) etc.
With which you can you predict what the user will select after the set of books as recommendation. In this way the input sequence length can be anything. (like, 3345, 33456, 334567 etc).
But to answer your question with KMeans.
I assume, all the rows are in same length.

Cluster all data with KMeans
Optimize number of K with Elbow method.
For each cluster take UNIQUE books.
Foe each User,
a. Predict Cluster for the User
b. From the Cluster's Unique books, Find books which are not the user already read. (Cluster books - User books).
c. You can suggest these books to the User.

This is just one the starting method to do it. Depends upon the business you can find many alternatives.
One such I can suggest is create similarity embeddings for each books, and suggest most similar (with distance measures) book to the user.
Hope this helps.


[Wow.A Great problem]

Possible solution:(Personal opinion)

First Opinion is: Use SVM or KNN instead of a neural net as your data is trivial like 2D vector.

Second Opinion is:

As you aren't trying to predict the next text possibility of a particular patient so in this case RNN(LSTM,GRU) might not help you. So a simple solution is Autoencoder to compress the data and then run a simple logistic regression or svm or may be a neural net.

How?

1.As your Patient numbers are not same in all cases so you should represent his/her all medical test data into a single vector.That is where Autoencoder comes into play.example:

Suppose we have 3 patient: First-3 test , second-2 test ,three-5 test. Our goal is to compress each patient data into 2 dimension vector(as your features are RBC and WBC , 2 features , so it will learn well if we compress it into 2D vector).

(inside asterik sign(*) your data goes)

First Patient=[[*],[*],[*]] # three test
Second Patient=[[*],[*]] # two test
Third Patient=[[*],[*],[*],[*],[*]] # five test

Now you compress the data via Variational autoencoder.

scale your data as wbc>>rbc

then use variational autoencoder to get the latent space vector

use dimesion=2 in latent space via Dense(2)

now when you get the vectors just add the vectors of test for each patient

Like : For First patient you got three vector in latent space like-

[2.33,4.2]+[5.11,9.22]+[0.21,6.32]=result..and You got the representaion for First patient.

after you got all of them a trivial svm/polynomial regression would be just fine.

Let me know how you solved it.


Understanding how convolutional layers work

What are the filters?

A filter/kernel is a set of learnable weights which are learned using the backpropagation algorithm. You can think of each filter as storing a single template/pattern. When you convolve this filter across the corresponding input, you are basically trying to find out the similarity between the stored template and different locations in the input.

But how are they getting initialized? Do they have random initial value or there are standard image filters that are getting used?

Filters are usually initialized at a seemingly arbitrary value and then you would use a gradient descent optimizer to optimize the values so that the filters solve your problem.
There are many different initialization strategies.

Sample from a distribution, such as a normal or uniform distribution
Set all values to 1 or 0 or another constant
There are also some heuristic methods that seem to work very well in practice, a popular one is the so-called glorot initializer named after Xavier Glorot who introduced them here. Glorot initializers also sample from distribution but truncate the values based on the kernel complexity.
For specific types of kernels, there are other defaults that seem to perform well. See for example this article.

If they are getting initialized with random value then the values should get changed on the training process of the network. If that's the case then a new question is created, how does someone backpropagate the filter of the convolutional layer? What is the algorithm behind this process?

Consider the convolution operation just as a function between the input image and a matrix of random weights. As you optimize the loss function of your model, the weights (and biases) are updated such that they start forming extremely good discriminative spacial features. That is the purpose of backpropogation, which is performed with the optimizer that you defined in your model architecture. Mathematically there are a few more concepts that go into how the backprop happens on a convolution operation (full conv with 180 rotations). If you are interested then check this link.

Is the entire matrix of the output getting passed through the activation function? How does the usage of an activation function change the learning process of the convolutional layer?

Let's think of activation functions as just non-linear "scaling" functions. Given an input, the job of an activation function is to "squish" the data into a given range (example -> Relu 'squishes' the input into a range(0,inf) by simply setting every negative value to zero, and returning every positive value as is)
Now, in neural networks, activations are applied at the nodes which apply a linear function over the input feature, weight matrix, and bias (mx+c). Therefore, in the case of CNN, it's the same. Once your forward-pass takes the input image, does a convolution function over it by applying a filter (weight matrix), adds a bias, the output is then sent to an activation function to 'squish' it non-linearly before taking it to the next layer.
It's quite simple to understand why activations help. If I have a node that spits out x1 = m0*x0+b0 and that is then sent to another node which spits out x2 = m1*x1+b1, the overall forward pass is just x2 = m1*(m0*x0+b0)+b1 which is the same as x2 = (m1*m0*x0) + (m1*b0+b1) or x2 = M*x0 + B. This shows that just stacking 2 linear equations gives another linear equation and therefore in reality there was no need for 2 nodes, instead I could have just used 1 node and used the new M and B values to get the same result x2 from x0.
This is where adding an activation function helps. Adding an activation function allows you to stack neural network layers such that you can explore the non-linear model space properly, else you would only be stuck with the y=mx+c model space to explore because all linear combinations of linear functions is a linear model itself.

Does a convolutional layer have weight and biases like a dense layer?

Yes, it does. Its added after the weight matrix (filter) is applied to the input image using a convolution operation conv(inp, filter)

Do we multiply the output matrix after the convolution process with a weight matrix and add some biases before passing it through the activation function?

A dot product operation is done between a section of the input image and the filter while convolving over the larger input image. The output matrix, is then added with bias (broadcasting) and passed through an activation function to 'squish'.

If that's true, then do we follow the same process as we do with the dense layers to train these weights and biases?

Yes, we follow the exact same process in forward pass except that there is a new operation added to the whole mix, which is convolution. It changes the dynamics especially for the backward pass but in essence, the overall intuition remains the same.

The crux for intuition is -

Do not confuse a feature and a filter. A filter is what helps you to extract features (basic patterns) from the input image using operations such as dot, conv, bias and activations
Each filter allows you to extract a 2D map of some simple pattern that exists over the image (such as an edge). If you have 20 filters, then you will get 20 feature maps for a 3 channel image, that are stacked as channels in the output.
Many such features, which capture different simple patterns, are learnt as part of the training process and become the base features for the next layer (which could be another CNN or a dense)
Combinations of these features allow you to perform your modeling task.
The filters are trained by optimizing towards minimizing a loss function using backprop. It follows the backward reasoning:

      - How can I minimize my loss?
      - How can I find the best features that minimize the loss?
      - How can I find the best filters that generate the best features?
      - What are the best weights and biases which give me the best filters?

Here's a good reference image to keep in mind whenever working with CNNs (just to reinforce the intuition)

Hope that answers your questions.