Evaluate document similarity / content-based recommender system

When you don't have label/categories then it's called Unsupervised Learning. You can solve this problem via Latent Dirichlet Allocation (LDA) model and then evaluate your model by splitting the texts in half and compare the topic assignment for each half using cosine similarity. The more similar the topic assignment, the better.
Example


Can k-means clustering get shells as clusters?

You cannot just use arbitrary distance functions with k-means.

because the algorithm is not based on metric properties but on variance.

https://stats.stackexchange.com/q/81481/7828

Fact is that k-means minimizes the sum of squares. This does not even give you the "smallest distances" but only the smallest squared distances. This is not the same (see: difference between median and mean) - if you want to minimize Euclidean distances, use k-medians or if you want other distances PAM (k-medoids).

You can generalize k-means to use a few more distances known as "Bergman divergences" and you can do some variant of the kernel trick. But that is not very powerful, because you don't have labels for optimizing the kernel parameters! Still, that could be what this exercise question is up to... If your "shells" are indeed centered at 0, then you can transform your data (read: kernel trick done wrong) to angle+distance from origin, and k-means may be able to cluster the projected data (dependent on the not well defined scaling of the axes).
Or the textbook did not realize that a kernel k-means has been proposed long ago. Then the argument is probably this: the mean of each shell is 0, and thus the shells cannot be distinguished. This clearly holds for unmodified k-means.


100 items 100 baskets divisor association analysis problem

a) The items that are frequent are $\{1,2,3,4,5,\dots,20\}$ because these all appear in at least 5 baskets.

b)

(1) $\{5, 7\} → 2$. Then $Confidence=\frac{1}{2}$

hence 5 and 7 will appear together in basket no. 35 and 70 and 2 will appear along with 7 and 5 in basket no. 70 so:

$$ Confidence = \frac{support(\{5,7\}\cup\{2\})}{support(\{5,7\})}= \frac{1}{2} $$

(2) $\{2, 3, 4\}→ 5$. Then $Confidence=\frac{1}{8} $

$\{2,3,4\}$ appear in baskets having basket number multiple of 12 ($LCM\{2,3,4\}$) i.e in basket no $\{12,24,...,60,.....,96\}$ and $\{2,3,4,5\}$ appear together only in basket no 60.

$→ Confidence=\frac{1}{8}$


Standard NER is going to extract individual entities, which in this case is time (3 months, 1 month) and currency ($3000, etc). You'll also want to think about Relationship Extraction, which identifies how two pieces of text relate to one another. For example, from a Euclidean distance measure, "contract" is related to "valid" and "executed", and "valid" is related to "3 months" while "executed" is related to "1 month". Based on what you said your desired output should be, you'll want to train your model to calculate the shortest distance between "contract" and "3 months", which in this case means teaching it to look for "valid" while ignoring "executed". There are different ways of doing this which you'll want to think about in terms of what works best for your corpus of text.
Here's a link to get you started (also includes links to other resources).


Ground-truth and feature extraction for predictive modelling

Your approach is a good one.
This way to extract features can lead to good results.
But before moving forward I recommend answering these two questions:
Is there is any correlation between the behavior of the user today and yesterday ... Etc " temporal correlation/ temporal dependency"
Is there is any kind of dependency between the users?
Answering these questions can measure the quality of your feature ?
Sometimes there is no need even to do calculations to answer these questions , if you tell us what kind of actions or activities you are looking at we may help in better way .
For example : if user 1 used his car today , there still a high prob to use it tomorrow, but if user 1 went to barbershop today there is a high probability that will not go tomorrow


Why sparse features should have bigger learning rates associated? And how Adagrad achieves this?

Intuition behind learning rates
With Adagrad, parameters that model the influence of the features in our problem tend to be updated at the same rhythm, something that is accomplished by what you explained in your question $\rightarrow$ the parameters are updated with different learning rates/ steps which may depend on how frequent (or sparse) the features are.
But why the parameters that we are learning should be updated with different learning rates? $\rightarrow$ For example, if our cost function was $f(x,y,z) = 20x^2+y^2+z^2 \rightarrow$ and $x,y,z$ were our parameters, it is clear that having a point far away from the global minimum, the function will be more sensitive to changes on $x$ than on $y$ or $z$. But the same parameter step applied to $y$ and $z$ will lead to the same change on our cost function $f(x,y,z)$.
In the general case, when we don't know exactly the terms of our cost function, one conservative rule for updating would be to use smaller learning rates on the directions with "big" gradients. This would prevent us from overshooting on these directions if the decision of moving along them with a big step was wrong.
To visualize the previous reasoning we can plot how would be the paths followed by our parameters applying different learning rates to them. In this case, the hypotetical cost function that I have used is $f(x,y)=20x^2+y^2$ and the number of iterations made are the same in all of them ($20$). With this plots we can see that:

Left plot $\rightarrow$ Applying a "big" learning rate to both $x$ and $y$ directions lead to overshooting on the $x$ direction, which causes that the path doesn't reach the minimum.
Center plot $\rightarrow$ Applying a "small" learning rate to both $x$ and $y$, solves the issue of overshooting, but causes the learning to be very slow.
Right plot $\rightarrow$ Applying a "small" learning rate to $x$ than to $y$ not only solves the issue of overshooting on the $x$ direction, but also makes faster the optimization.

Note that the optimization used here is not exactly like Adagrad optimization, but with it I pretend to show the effects of different learning rates.

Adagrad
With Adagrad we are able to take this previous intuition of why the learning rates should be different for each parameter but with another point of view.
For understanding why Adagrad behaves in the way described by the question, we should take a look at the practical formula that is commonly used for applying it:
$$ \mathbf{w}_{t+1}= \mathbf{w}_t - \frac{\alpha}{\sqrt{\epsilon I + \text{diag}(G_{t+1})}}\,\,g_{t+1} $$
Where $\mathbf{w}$ is the vector of parameters that we want to update, $g$ is the gradient of the cost function w.r.t. these parameters $\mathbf{w}$, $t$ is the number of the iteration taking place and $G_t$ is given by:
$$G_{t+1}=\sum_{\tau=1}^{t+1} g_\tau\,g_\tau^T$$
So, if we have a look at one parameter $w^j$ from the vector $\mathbf{w}= [w^1,w^2,...,w^n]^T$, we have:
$$ w_{t+1}^j = w_t^j - \frac{\alpha}{\sqrt{\epsilon + G_{t+1}^{jj}}}\,\,g_{t+1}^j \,\,\,\,\,\, \leftrightarrow \,\,\,\,\,\, G_{t+1}^{jj}=\sum_{\tau=1}^{t+1}(g_\tau^j)^2$$
What does this mean? $\rightarrow$ As $t\uparrow\uparrow$ the learning rate related to the parameter $w^j$ will tend to decrease. This is because in the denominator we have the cummulative sum of the absolute value of its previous gradients.
Thereby, using Adagrad we could also solve the problem presented in the previous plots. Due to the fact that the plotted function was more sensitive to changes on the $x$ direction (hence the x-component of the gradient was bigger), Adagrad will automatically give a lower learning rate to the updates in $x$ than on $y$.
But what is the relation of the above with the more/less common features?
Common features here take the meaning of features that during a significant number of updates their influence on the problem has been poorly analysed. This can happen because during these iterations the components of the gradient vector related to these features have had significant lower magnitudes.
Let's look at an example using Stochastic Gradient Descent $\rightarrow$ we will be updating the parameters for every single sample that we have.
Let's consider that our model of parameters is linear, and have two parameters $(w^{(1)},w^{(2)})$ that have to be trained $\rightarrow$ so we can have a predictive function like this:
$$\hat{y} = w^{(1)} x^{(1)} + w^{(2)} x^{(2)}$$
where $x^{(1)}$ and $x^{(2)}$ are the features of our problem.
So if we make use of a cost function ($J$) that represents the mean squared error of the prediction ($\hat{y}$) given by our parameters, the gradient at iteration $t$ will be given by a quantity proportional to the features:
$$ g_t = \left(\frac{\partial J}{\partial w^{(1)}_t}, \frac{\partial J}{\partial w^{(2)}_t}\right)^T \propto (x^{(1)}_t, x^{(2)}_t)^T $$
Thereby, using SGD the quantity that we use to update each of our parameters $\delta w_{t+1}$ is also proportional to the associated feature:
$$ \delta w_{t}^{(j)} \propto x^{(j)}_t$$
With this already set up, lets imaging that our training is done in three samples (just for visualization purposes). During these samples the $x^{(1)}$ feature is more frequent (or less sparse) than the other feature $x^{(2)}$, so we have a dataset, $\mathbb{X}$, similar to the next:
$$ \mathbb{X} = \begin{pmatrix}
1 & 0\\
5 & 0\\
3 & 3\\
\end{pmatrix}$$
Where each row represents a sample of $x^{(1)}_t$ and $x^{(2)}_t$, with $t\in\{1,2,3\}$.
Now it is clear that the parameter $w^{(2)}$ won't be updated until the last sample ($t=3$), because during the first two samples ($t=1, 2$) its gradient component has been zero (becase its proportional to $x^{(2)}_t$).
But what would have happened at this point to $w^{(1)}$? $\rightarrow$ it would have been updated during the first two samples (non-zero gradient component), and because of it, its associated learning rate will be smaller than the one of $w^{(2)}$ in spite of having the same $x^{(1)}_3 = x^{(2)}_3$!
So now we understand why the parameters associated with sparse features can be updated with greater learning rates using Adagrad.
Conclusion
Adagrad allows us to give more importance to updates in parameters that have associated features which are sparse, or more generally, to give more importance to parameter updates that have experimented a record of relatively lower gradients (in magnitude).
Why is this useful? To answer this, we can cite the original authors (see paper: "Adaptive Subgradient Methods for
Online Learning and Stochastic Optimization "):
> "Our procedures give frequently occurring features
very low learning rates and infrequent features high learning rates, where the intuition is that each
time an infrequent feature is seen, the learner should “take notice.” Thus, the adaptation facilitates
finding and identifying very predictive but comparatively rare features"


How to ignore vectors of zeros (i.e. paddings) in Keras?

You can use Masking layer (with mask value of zero) before LSTM layer in order to ignore all timesteps with only zeros (i.e. zeros vector). You can find more information about this layer on its documentation. Here is an example from documentation which uses Masking layer:

  Consider a Numpy data array x of shape (samples, timesteps, features),
  to be fed to an LSTM layer. You want to mask sample #0 at timestep #3,
  and sample #2 at timestep #5, because you lack features for these
  sample timesteps. You can do:

  set x[0, 3, :] = 0. and x[2, 5, :] = 0.
  insert a Masking layer with mask_value=0. before the LSTM layer:

model = Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(LSTM(32))


Select two best classifier using F1-score,Recall and precision

f1-score combines precision and recall in a single figure. As both are pretty similar in A and C cases, f1-score is similar too.

Your choice depends on what it is less harmful in your categorization: false positives or false negatives.

I do recommend you to read the 3rd chapter of "DEEP LEARNING:From Basics to Practice" volume 1 by Andrew Glassner. There you have the three concepts (precision, recall and f1-score) described in a very illustrative way.


Here you do a single fit() of the model whose name tells for itself - Sequential.

Unless you are doing cross-validation or some kind of distributed learning with multiple models, there is no benefit of running several fits in parallel.

However, you can have significant speed-up on iteration level, depending on how Keras backend is configured to work with your hardware.

Assuming you are using Keras with the tensorflow backend (default):

If you are running on CPU, tensorflow should already pick up all available cores by default (if not, it might help checking How to run Keras on multiple cores?);
If you are running on GPU:

(1) make sure Keras backend can see your GPU

from keras import backend
print(backend.tensorflow_backend._get_available_gpus())

If your GPU is not showing up here, some configuration might be necessary
on your system (it deserves another topic);

(2) take a look at the CuDNNLSTM - it is the Keras LSTM implementation
optimized for GPU.


Learning to rank: construct absolute ranking using pair-wise ranking approach

Creating a total ranking from pairwise comparisons that don’t necessarily follow the axioms or rational preferences would certainly require some optimization, and you would need to compute a quadratic number of such comparisons in order to produce a ranking.

There is however one trick to do this easily: if the function that classifies whether one element is ranked above the other is linear in all the features, you can, intuitively, compare each element to a null element (i.e. just apply the decision function to the element alone), get a score assigned to them, and sort them by this score.

So, intuitively, your linear classifier would be something like this: element 1 is above element 2 if:
F(el1 – el2)>0 [function has to be symmetrical, thus there can't be a non-zero threshold] --> F(el1) – F(el2) >0 (only when F is linear!!!)
Which implies that el1 > el2 iff F(el1)>F(el2)


Using data agumentation for a frozen pre-trained model

No, you can definitely use data augmentation when the layers of the pre-trained model are frozen.

In this article, the author refers to the size of VGG16 which is quite a large CNN. Also, he trains the model on his CPU which slows down training even more. Therefore, he does not want to use data augmentation, as that would increase the training time even more:

  Running VGG16 is expensive, especially if you're working on CPU, and we want to only do it once. Note that this prevents us from using data augmentation.


First, make sure, you have the same vocabulary in your word2file and in your Keras code. If you Keras Tokenizer, you can simply call fit_on_texts with the text actually being the vocabulary.

Load your word embeddings as numpy matrix (just read all the numbers and then call np.array).

When creating the Embedding layer, call set_weights and set the loaded matrix and set trainable to False.

You can find more at Keras blog but I am not sure how up to date it is because it is from 2016.


Predicting three related scores with one joint model

Do you have a labeled set? Any algorithm can do it in a supervised manner, as long as you have enough labeled data to train it on.

A tree-based model (Random Forests Classifier for instance) could do really well on this task, especially since you have categorical data. All you need to do is encode the data as numbers (for instance 0: male, 1: female, 2: transgender) and feed it into your model (applying random splits and cross-validation and all that).

In case you don't, then the best you can do without collecting new data is to find clusters of related patients in your dataset. The same rules for encoding variables apply, although you may have to pay some attention to how you scale your variables since clustering algorithms are sensitive to relative scaling of features. This happens because most clustering algorithms cluster data using distance metrics which lose meaning when you are dealing with categories. See this for more info on clustering categorical data:


A heap works in clustering the same way it works outside of clustering.

It's purpose is to efficiently find the minimum or maximum of a set, remove it, then find the next.

Efficient implementations of heaps in scripting languages may be impossible. For good performance, you usually need low-level memory access to avoid copying.


Shifted feature distribution across different datasets

It'd be easier to understand if the plots were labeled. Nonetheless, I think I understand the question, and perhaps it's just sampling error. Why do you have two different pairs of train and test data? Are they from different sources? In that case, you probably don't want to be using the same classifier on both. Also, what do you mean 'validate' using two different train and test sets? Classifiers are trained on a train set, and validated on a test set, what do you mean you're validating on two different train/test pairs?


How can we use machine learning to distnguish between similarly looking images

From the images alone, assuming the links you posted are representative, I doubt a model will ever be able to predict better than random guessing.

There is a chance the model could effectively learn to read the label, but it seems like a long (and somewhat pointless) path to go down, unless you will only be predicting on labelled bottles in the future!

You will need to introduce different information with each sample, perhaps the price, amount of liquid and other such feature. At this point, you will likely want to ditch your CNN for images and instead use a standard classifier, such as Support Vector Machines or a simple feed-forward neural network.


How do you make an NN for image classification invariant to translation **and** rotation?

It is not possible to have general rotationally-invariant neural network architecture for a CNN*. In fact CNNs are not strongly translation invariant, except due to pooling - instead they combine a little bit of translation invariance with translation equivariance. There is no equivalent to pooling layers that would reduce the effect of rotation this way (although for very small rotations the translation invariance will still help).

You can however construct features and create a pipeline that reduces incoming rotation differences in your inputs. For example, this answer on Signal Processing Stack Exchange suggests calculating dominant gradient of an image, then rotating the image so that this is always oriented the same way before further processing. If your image has strong straight edges, you could do similar by detecting those (e.g. by using a Hough transform) and rotating the input so that these are always oriented the same way. These approaches work only within certain image tasks, but can save on processing time and potentially increase accuracy if they are possible. Effectively they are a form of input normalisation.

A more radical idea might be to perform a map to polar co-ordinates in your image before processing. This would effectively convert rotational (and radial) variance into translation variance in your image. A CNN processing this mapped image would effectively convert its translation invariance into rotational invariance on the original unmapped image. But the cost would be losing all translation invariance, so only worth considering if your inputs have high variance in rotation but low variance in translation.

* Never say never. This is caveated by special cases, for example the paper Group Equivariant Convolutional Networks explains an architecture that adds support for multiples of 90 degree rotation (taking advantage of grid structure in computer images and the weight matrices that construct neural network layers). However, if you want to support free rotation of values other than 90, 180, 270 degrees, then as far as I know, there is no way to do that architecturally within the network.


Online k-means (more commonly known as sequential k-means) and traditional k-means are very similar.  The difference is that online k-means allows you to update the model as new data is received.

Online k-means should be used when you expect the data to be received one by one (or maybe in chunks).  This allows you to update your model as you get more information about it.  The drawback of this method is that it is dependent on the order in which the data is received (ref).


The responses to both of those questions can be treated as categorical, as although they mention numbers you can’t fully encode the information in them as a number. You could obviously number them, e.g. “I’ll call 9-11 times per month response 1”, but the number 1 doesn’t reflect the value of the response.
These types of categorical “buckets” representing a range of values are often used when the value we want to measure is continuous and the range of possible values is large.
If you wanted numerical variables you could recode the responses. For example you could represent answers to question 2 as two numerical variables, min_times and max_times.


While doing clustering analysis, you would prefer to have features which discriminate data points and help you discover more homogeneous segments. However, if the value of a particular feature or variable is same for large percentage of the data, then that feature would not add any information for clustering.

Let us take an example, say you are dealing with bank customers data and there's a feature called 'country' in it. If in the given data, most of the customers have country as USA, then you would want to simply drop that feature as it adds no useful information. This is the extreme case in which there is single unique value.

In your network data logs example, port numbers and protocols would be mostly the same. The web server might be listening on port 443 for HTTPS protocol connection. So in that case, the two columns can be excluded from clustering analysis.


Multiple keras models parallel - time efficient

As was suggested by Erik van de Ven, it sounds like running each model on a different process should provide the requested parallelism.

I guess you could either run the fit function for each model in a different process
Or you could even load them on different cpu cores:

with K.device('cpu0'):
    input1 = Input(inputShapeOfModel1)
    output1 = model1(input1)

with K.device('gpu0'):
    input2 = Input(inputShapeOfModel2)
    output2 = model2(input2)

model = Model([input1, input2], [output1, output2])

I haven't tried any of these though, so i'm not sure what would provide the best result


Feature agglomeration: Is it testing interactions?

From the documentation:

  Similar to AgglomerativeClustering, but recursively merges features instead of samples.

In standard agglomerative clustering you receive a matrix $M^{n \times m}$ representing $n$ samples of dimension $m$ that you want to cluster. In feature agglomeration the algorithm clusters the transpose of the matrix, i.e. $M^T$ so it clusters $m$ samples of dimension $n$, these samples represent the features.

The default distance used to cluster the features (samples in the transpose matrix) is the euclidean distance, but you can also use l1, cosine and others.

For example suppose you have 3 samples of dimension 3 (a matrix 3x3 matrix)::

----------------------------------
| feature1 | feature2 | feature3 |
----------------------------------
|     1    |   1.05   |    10    |
----------------------------------
|     1    |   1.05   |    10    |
----------------------------------
|     2    |   2.05   |    20    |
----------------------------------

If you want to reduce the dimension of your dataset to 2 dimensions, the algorithm clusters together feature1 and feature2, and leaves feature3 unchanged, the new matrix is:

----------------------------------
| feature1 - feature2 | feature3 |
----------------------------------
|         1.025       |    10    |
----------------------------------
|         1.025       |    10    |
----------------------------------
|         2.025       |    20    |
----------------------------------

The resulting feature is determined by the pooling function in the case above it computes the arithmetic mean. See this for a more in depth usage example.


How Calculate Effect (percentage) label of the input variables on the output variable by DecisionTreeClassifier

I don't think there is any way of doing that with decision trees, because that's not how decision trees work: the predicted label is not the result of some linear combination of the features. Instead you can look at the actual decision tree that the model represents and see which features have been used to classify a particular instance.


Binary Search Tree with categorical and numeric values in python

If PlayTennis is your target classification label, you could map the numeric values to intervalls. Finding appropriate intervalls can be done by first sorting the numeric values and then using Intervalls at which there is a change in your target label. In this case we have a change between 48 and 60, and between 80 and 90. (48+60)/2 = 54 and (80+90)/2 = 85 are good candidates for label-intervalls. (T>54) and (T>85) would be the labels.


Are there any good general techniques for binning/histogramming arbitrary data?

I don't know if this what you want but here is a way to calculate the number of bins.

Count the number of data points in your dataset.
Take the square root of the number of data points and round up to determine the initial number of bins required: $InitialNumberOfBins = \sqrt{NumberOfDataPoints}$.
Divide the specification tolerance $Max-Min\ value$ by initial number of bins: $FinalNumberOfBins = (Max-Min\ value) / InitialNumberOfBins$.


Solution:

Finally I found the solution myself.  I just used another docker image with an older version of tensorflow (2.0.0), and the error disappeared.


How to convert unstructured texts to structured data?

If I get you right, you would like to extract the mentioned entities from the unstructured text. This sounds as Named Entity Recognition (NER) problem for me. In your case I would define the following entities:

START_DATE
END_DATE
MAIN_PRIZE
EXTRA_PRIZE

Your text now can be tagged or annotated with those entities. You would like to build a machine which will automatically tag the encountered entities in the unstructured text (free text).

I would suggest to take a look at Conditional Random Fields (CRF), a classic approach to NER. There are also Recurrent Neural Networks (RNN) which are state of the art for sequence modeling but I am not an expert with those methods.

Let's begin the process of model building. You need to decide what will be a single training example. This can be a sentence and this can be also a paragraph.

Since you did not provide any example, let me invent two:

The contest started on <START_DATE>july 1st</START_DATE> and ended on <END_DATE>july 25</END_DATE>.

The winner got away with <MAIN_PRIZE>1 million dollars</MAIN_PRIZE> and <EXTRA_PRIZE>bunch of flowers</EXTRA_PRIZE>.

Notice the way how the text is structured. Every word now is either label (surrounded by tags) or not labeled at all (outside of any tag). This is one way how to structure the text. This would be also the desired output on some unstructured text.

The machine does not understand the text and annotation. You need to come up with features that you can extract from each word. It can be the word itself, the previous word, the next word, left bigram, right bigram, suffix of length 1,2,3, prefixes of length 1,2,3 and so on. I suggest you to look at Stanford NER features.

Once you got the features, you need to create a training set. The training set is your annotated text (see my example) converted to label + feature information. It may look like this

O   feature1 feature2 ...
O   feature1 feature2 ...
B-START_DATE   feature1 feature2 ...
I-START_DATE   feature1 feature3 ...
...

This basically says that the first two words are not labeled, but the last two are part of an entity START_DATE.

Each line is essentially a word followed by features for it. The label "O" is the way to say that the word is not labeled, the labels B- and I- for DATE mean the first and second words of a named entity (e.g. "july 1st").

The sentences can be separated by empty lines.

Once you've got the training set file ready, you can do the model training. With a model in hand, you can apply it on any unstructured data (extracted features) to get the desired entities extracted.

As a tool I suggest you CRFSuite (also available with Python bindings). I can detail more each step if you would like.


First of all, precision and recall are not specific to image classification; they are relevant wherever there are two distinct "positive" and "negative" classes (for example, when you test an e-mail for "spam/not-spam", or a blood sample for "has virus/does not have virus").
You can read more on this question on Cross Validated, but to sum it up - precision is the probability that a sample is positive if a test said it is, and recall is the probability that a positive sample will be reported as positive by the test.  False positives mess up your precision, and false negatives mess up your recall.
Now, your task appears to be one of multi-class classification - with at least 17 classes, from your example. I wouldn't go with precision/recall for this - you can only do it pair-wise for pairs of classes. You can, however, plot a CxC confusion matrix (where C is the number of classes), and investigate where your models tend to miss. There's an implementation in SKLearn (link).
If you need a single-number metric, I'd start with just accuracy (and develop from there). Following Nuclear Wang's comment, I'd also suggest looking at Cohen's Kappa (see explanation on Cross Validated) to better account for class imbalance.
To read more on multi-class classification, see this question. I'd also recommend this blog post on Towards Data Science


find the values of theta in a cost function from andrew ng course

Elaborating on the previous answer, now hat you have $\theta_0 = 0.5$, take any value of $x$ and the corresponding $h_\theta$ value to find $\theta_1$. For instance, using $x=1$ will yield:

$h_\theta(1) = \theta_0 + \theta_1 \times 1$

From the graphd, $h_\theta(1) = 1.5$ so that:

$\theta_0 + \theta_1 = 1.5$

$\theta_1 = 1.5 - \theta_0$

$\theta_1 = 1.5 - 0.5 = 1$

The final equation is then $h_\theta = 0.5 + x$

As said in the comment, you can also get the slope visually, noticing that if you increase $x$ by 1, $y$ also increases by 1 when you follow the slope.

Does this help?


A doubt about the GridSearchCV function in Sklearn?

The probability that there's a mistake in SK-Learn is very low.

Reasons:

It's a mature library
Among the contributors is Sebastian Raschka. Those guys are very meticulous.

https://sebastianraschka.com/faq/docs/evaluate-a-model.html


Is the role of the validation set in a deep learning network is only for Early Stopping?

The validation set's purpose is to track your level of overfitting while you are experimenting with your network. When we try something we want to know how well it works on data it hasn't seen before.

By looking on the score on the training set we can't tell what is good from what is overfitting. If you instead use your test set to evaluate the results of all your experiments, then your test set stops being a reliable final evaluation since you have included it in your decisions. The test set is only for final evaluations.

To fill this need we set aside a part of our data as a validation set. A dataset that we do not directly train on, but instead the use to check results of training different layer/unit configurations, tuning hyperparameters, feature engineering and all the other things we possibly want to try. Early stopping is just one of the possible uses.

Regarding your comments about hyperparameters:
You don't use the validation set directly when you are tuning hyperparameters (with grid search or any other method). You still use the training set, but then you use the validation set to test what gave the best result.


How to read Feature Based Grammar from a string

You need to take care of NP and VP expansion production.

# ###################
# Grammar Productions
# ###################
# S expansion productions
S -> NP[NUM=?n] VP[NUM=?n]
# NP expansion productions
NP[NUM=?n] -> N[NUM=?n]
NP[NUM=?n] -> PropN[NUM=?n]
NP[NUM=?n] -> Det[NUM=?n] N[NUM=?n]
NP[NUM=pl] -> N[NUM=pl]
# VP expansion productions
VP[TENSE=?t, NUM=?n] -> IV[TENSE=?t, NUM=?n]
VP[TENSE=?t, NUM=?n] -> TV[TENSE=?t, NUM=?n] NP


Click Through Rate calculation (CTR) calculation problem

I have no knowledge at all about this kind of problem, but logically I would say:

Should I remove the operations where there were no products shown (even though some of them there were clicks on products)?Example (should discard this columns?):

Yes, because otherwise you could in theory end up with a rate higher than 1, this wouldn't make sense. However this raises the question of what exactly is represented: since some clicks can be counted when there is no product, it looks like the number of clicks is over-estimated in the data since the clicks are supposed to be on products.
Also note that it's the rows which would be removed in this case, not the columns.

My other doubt is wether should I sum the clicks and the products shown and then divide one on other OR should I do the CTR for each operation:

This issue means that either the data doesn't have the proper format to apply the definition (strictly speaking the data should have one row for every product shown, and the number of clicks should be either 0 or 1), or that the instructions for applying the CTR formula with this specific data are under-specified. I would say that the first option is closer to the original formula since it doesn't require to do a mean over individual rates.


Classification technique for unsupervised data?

You have many options of algorithms to use for classification of unsupervised data.

This is a very broad topic, but if you need a specific algo recommendation, try to see if self-organizing maps (SOM) may help with your specific problem. In R, try the kohonen package.

K-means is another popular clustering algorithm.

No matter which method you use, consider converting your categorical data to numerical data for clustering, as it may alleviate some of your mixed data-type woes.


value error of incompatible wrong dimension during training a model

As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("xxx"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["xxx"]).

so the code below will work.

## Important: i have passed the columns a string to CV and list of columns to OHE

transformer=ct(transformers=[('review_counts',cv(),'verified_reviews'),
                             ('variation_dummies', ohe(),['variation'])
                            ],remainder='passthrough')

Credit goes to Another man who helped me on this.


Why does English ELMo model give embeddings for non-English words?

While ELMo was trained on English data, it does not know whether the data you give it as input is English or not.
The input of ELMo is received at character-level. It may happen that the 1B Word data had hindi characters intermixed, case in which your characters would be encoded as they are or, most probably, your characters are encoded as unknown characters (just like the unknown token <unk> for word-level NLP but for characters).
ELMo is just a bunch of mathematical operations, so it takes whatever it receives and computes its operations on it, first, taking the character embedding with the characters you pass to it, then with a char-CNN followed by two highway layers and finally a bidirectional LSTM.


Model should predict the same value every time for the same input

Yes, the output is the same for the same input.

The "randomness" just concerns the approach how an ensemble is built: bagging (considering random subset of objects) and random selection of features for splitting (when we create each decision tree and choose which feature would be used as a splitting criteria).


How do companies like Amazon track what products are most frequently bought together?

This is called Association Learning. Quoting Wikpedia:

  Association rule learning is a method for discovering interesting
  relations between variables in large databases. It is intended to
  identify strong rules discovered in databases using some measures of
  interestingness

So, your question asks about how companies like Amazon and Walmart know what products are frequently bought together. So, that is done by the Apriori algorithm, which falls under the class of Association Learning algorithms.

Again, quoting Wikipedia:

  Apriori is an algorithm for frequent item set mining and association
  rule learning over transactional databases. It proceeds by identifying
  the frequent individual items in the database and extending them to
  larger and larger item sets as long as those item sets appear
  sufficiently often in the database.

This algorithm generates rules which show what items are bought along with which items, depending on the support and confidence thresholds. (As you didn't ask about the particulars, I would skip the details. Pl check wikipedia for the finer details and the algorithm itself).

The arules package of R implements the apriori algorithm. Unfortunately, there is no fast implementation of the same in Python.


Final Model fitting - subset vs entire training data

The general machine learning process is this:
Split your data into two parts, training and test. So in your example I would take 100k for test and 900k for training (don't know why you say only take 200k in your question but I digress). With the 900k training we perform hyper-parameter tuning. This can be done by splitting training into training and validation say 800k/100k or better yet we could do this using k-fold cross validation.
Once you have chosen the optimal hyper-parameters in this manner you evaluate their performance on the test set. The whole point of this process is simply to evaluate the algorithms performance, and from that select and algorithm. That is the only reason for a train/validation/test split. (As a note, this process can be even further improved by using something called nested cross-validation but I will not go into the details).
After you have selected your algorithm and determined its performance (error rate), you take your whole data set (1 million records) and perform hyper-parameter selection on that, either using a single split or by k-fold cross-validation. You no longer need the test set because you have already determined the models error rate in the previous step.
Once you have selected the best hyper-parameters in the previous step you apply them to the entire data set (the 1 million records) and build the model.


Machine Learning: Classify Array of Numbers based on Patterns

Actually you want to classify time series. Going from here you have basically two options:

Build features from this time series, i.e. RMS, Peak Values, etc.. and classify them with "classical" predictors like SVM or Random Forests.
Use neural networks, or more specifically use LSTM in order to feed directly your "array of numbers", which is a time series.


Class weighting during validation in Keras

As described in Keras.models.py[1], validation_data can be a tuple (inputs, targets, sample_weights).  So, you can use this parameter for class weighting during the validation loss calculation.

[1] https://github.com/fchollet/keras/blob/master/keras/models.py


Good performance metrics for multiclass classification problem besides accuracy?

If you can use python I suggest PyCM module. A vast variety of performance evaluation parameters is in access by this module and also you can use its documentation if you want to implement it by yourself.

There is an example of it:

>>> from pycm import *
>>> y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2] # or y_actu = numpy.array([2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2])
>>> y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2] # or y_pred = numpy.array([0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2])
>>> cm = ConfusionMatrix(actual_vector=y_actu, predict_vector=y_pred) # Create CM From Data
>>> cm.classes
[0, 1, 2]
>>> cm.table
{0: {0: 3, 1: 0, 2: 0}, 1: {0: 0, 1: 1, 2: 2}, 2: {0: 2, 1: 1, 2: 3}}
>>> print(cm)
Predict          0        1        2
Actual
0                3        0        0
1                0        1        2
2                2        1        3

Overall Statistics :

95% CI                                                           (0.30439,0.86228)
Bennett_S                                                        0.375
Chi-Squared                                                      6.6
Chi-Squared DF                                                   4
Conditional Entropy                                              0.95915
Cramer_V                                                         0.5244
Cross Entropy                                                    1.59352
Gwet_AC1                                                         0.38931
Joint Entropy                                                    2.45915
KL Divergence                                                    0.09352
Kappa                                                            0.35484
Kappa 95% CI                                                     (-0.07708,0.78675)
Kappa No Prevalence                                              0.16667
Kappa Standard Error                                             0.22036
Kappa Unbiased                                                   0.34426
Lambda A                                                         0.16667
Lambda B                                                         0.42857
Mutual Information                                               0.52421
Overall_ACC                                                      0.58333
Overall_RACC                                                     0.35417
Overall_RACCU                                                    0.36458
PPV_Macro                                                        0.56667
PPV_Micro                                                        0.58333
Phi-Squared                                                      0.55
Reference Entropy                                                1.5
Response Entropy                                                 1.48336
Scott_PI                                                         0.34426
Standard Error                                                   0.14232
Strength_Of_Agreement(Altman)                                    Fair
Strength_Of_Agreement(Cicchetti)                                 Poor
Strength_Of_Agreement(Fleiss)                                    Poor
Strength_Of_Agreement(Landis and Koch)                           Fair
TPR_Macro                                                        0.61111
TPR_Micro                                                        0.58333

Class Statistics :

Classes                                                          0                       1                       2
ACC(Accuracy)                                                    0.83333                 0.75                    0.58333
BM(Informedness or bookmaker informedness)                       0.77778                 0.22222                 0.16667
DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0
ERR(Error rate)                                                  0.16667                 0.25                    0.41667
F0.5(F0.5 score)                                                 0.65217                 0.45455                 0.57692
F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545
F2(F2 score)                                                     0.88235                 0.35714                 0.51724
FDR(False discovery rate)                                        0.4                     0.5                     0.4
FN(False negative/miss/type 2 error)                             0                       2                       3
FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5
FOR(False omission rate)                                         0.0                     0.2                     0.42857
FP(False positive/type 1 error/false alarm)                      2                       1                       2
FPR(Fall-out or false positive rate)                             0.22222                 0.11111                 0.33333
G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772
LR+(Positive likelihood ratio)                                   4.5                     3.0                     1.5
LR-(Negative likelihood ratio)                                   0.0                     0.75                    0.75
MCC(Matthews correlation coefficient)                            0.68313                 0.2582                  0.16903
MK(Markedness)                                                   0.6                     0.3                     0.17143
N(Condition negative)                                            9                       9                       6
NPV(Negative predictive value)                                   1.0                     0.8                     0.57143
P(Condition positive)                                            3                       3                       6
POP(Population)                                                  12                      12                      12
PPV(Precision or positive predictive value)                      0.6                     0.5                     0.6
PRE(Prevalence)                                                  0.25                    0.25                    0.5
RACC(Random accuracy)                                            0.10417                 0.04167                 0.20833
RACCU(Random accuracy unbiased)                                  0.11111                 0.0434                  0.21007
TN(True negative/correct rejection)                              7                       8                       4
TNR(Specificity or true negative rate)                           0.77778                 0.88889                 0.66667
TON(Test outcome negative)                                       7                       10                      7
TOP(Test outcome positive)                                       5                       2                       5
TP(True positive/hit)                                            3                       1                       3
TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.33333                 0.5

>>> cm.matrix()
Predict          0        1        2
Actual
0                3        0        0
1                0        1        2
2                2        1        3

>>> cm.normalized_matrix()
Predict          0              1              2
Actual
0                1.0            0.0            0.0
1                0.0            0.33333        0.66667
2                0.33333        0.16667        0.5


Saving a different model for every epoch Keras

Saving many models definitely possible.

Please find below code snippet:

# serialize model to JSON
model_json = model.to_json()
with open("model.json", "w") as json_file:
    json_file.write(model_json)
# serialize weights to HDF5
model.save_weights("model.h5")
print("Saved model to disk")

# later...

# load json and create model
json_file = open('model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)
# load weights into new model
loaded_model.load_weights("model.h5")
print("Loaded model from disk")

For detailed reference:Save&Load Keras model

Note: Please append epoch value with the file name, so that you have all different trained model.s


Considering you extracted the news using a TF-IDF approach what you have as a result is just one feature (frequency of terms). I think you would need to add more features to your corpus to be able to match two news as the same (or similar).

One new feature would be the temporal, where you would add a timestamp for the news. It will allow you check if two news (from different publishers) were published on the same period. (a week, 2 weeks, etc).

The second one could be a spatial, if you have geolocation of the news for example, you can add it to your train dataset. Something similar was done by Chi-Chun Pan. It would allow you be more confident if the news happened on the same place.


Topic Similarity Measure | Multi-class Text Classification Model

LDA is amazing for topic modelling, but naming the topics (regardless of what algorithm you use) is inevitably a disappointing process, because the labels you’re thinking of are very subjective and the algorithm has little respect for what words we want to go in each topic :p

The way I would approach your problem is:

Train an LDA topic model, using grid search rather than intuition to select the right number of topics
Transform your training examples into k-dimensional vectors using your trained topic model
Add your training vectors to a k-d tree and classify new tweets using a k-nearest neighbours approach


How to select 'cutoff' of classifier probability

What you're looking for is something along the line of an ROC curve:
Using the threshold as a decision parameter, you can observe the trade-off between FPR (False Positive Rate: how many of the articles not belonging to the author will be correctly classified) and TPR (True Positive Rate, aka recall: how many of the articles which are really by the author will be classified as such).
When the parameter is at one end, you'll classify all documents as belonging to the author (100% recall, but pretty bad precision), and at the other hand, you'll have 100% precision but pretty bad recall.
The plot will allow you to decide on a value that satisfies your requirements (i.e. how much will your precision suffer when you want 95% recall). You can select it based on your desired value in one metric (e.g. 95% recall), but really I'd just plot it and have a look. You can do it in SKLearn with plot_roc_curve.


What does the co-ordinate output in the yolo algorithm represent?

Simply the left high most vertex of the bounding box.

With X,Y and it's height and width you can find the bounding box in the image. (With respect to the size of entire image)


How to cluster text-based software requirements

I recommend using word2vec as feature vector of words and LSTM autoencoder to encode a sentence (or text) . After you get a vector for each sentence (or text), you can cluster your sentences (or texts) using a variety of clustering techniques like k-means or dbscan and represent them using t-sne or u-map. Start from here:
https://blog.myyellowroad.com/unsupervised-sentence-representation-with-deep-learning-104b90079a93


LSTM: Converting to Bayesian Deep Neural Network

Firstly, I suggest looking at the following paper:

  Gal and Ghahramani, A Theoretically Grounded Application of Dropout in
  Recurrent Neural Networks, 2016

which discusses how to appropriately apply dropout as an approximate variational Bayesian model. The key is to use the same dropout mask at each timestep, rather than IID Bernoulli noise.

As for measuring model uncertainty, note that while dropout gives us an approximate variational Bayesian neural network, it does not give access to the variational posterior density, and so we cannot compute e.g. the entropy of the posterior distribution. But, we can trivially compute the predictive variance, which is closely related to model uncertainty. Simply run $n$ forward passes on an input to get outputs $Y_F = (y_1,\ldots,y_n)$. Then compute your measure (e.g., the covariance matrix in the regression case) on $Y_F$. For classification as your case seemingly is, you can use predictive entropy, for instance. For more measures see

  Gal et al, Deep Bayesian Active Learning with Image Data, 2017


Instead of dropping rare words or incorporating them risking their scarcity in the training data leads to poor predictions, you can opt for a third alternative: using a subword vocabulary.
You can use approaches like byte-pair encoding (BPE) to extract a subword vocabulary, that removes the out-of-vocabulary word problem and reduces data sparsity in general. There is the canonical python implementation as well as the popular implementation by Google called sentencepiece.


Definition of a "lexicon" in Named Entity Recognition

As far as I know the word "lexicon" is mostly used for a simple list of words (or terms), I would say it's quite rare to use it for describing a list of patterns/rules. "gazetteer" and "dictionary" would be a bit more general in my opinion, for instance one can have a "dictionary of rules" which associates specific patterns with actions. But overall I agree with your impression that these terms are often used interchangeably, I'm not aware of any standard definition.


What is the difference between spiral, flame, aggregation data

Generally, without knowing the source of the data, we can't tell you much about the columns. But I assume the first two columns correspond to $x$ and $y$. The third is probably some meta-data? Maybe a cluster number?
For an illustration I found this figure coming from this publication. Maybe that helps you imagine the data's shape.
https://www.researchgate.net/publication/338167806_A_Fast_Method_for_Estimating_the_Number_of_Clusters_Based_on_Score_and_the_Minimum_Distance_of_the_Center_Point


Why don't get the expected result using a SVM training model?

Without seeing the data and your code it's hard to examine and tell what's the problem. Below are what I would think as a first step.

98% accuracy is a high level, so I would initially check for over-fitting.
The ideal solution to over-fitting is using more data. If you can't, then increasing the portion of the training dataset (maybe to 50%) and then going through cv & testing might help.

Another common reason is over/under-represented class. What's the ratio between the classes?


Perform fuzzy lookup to partially match the address column

This problem is called record linkage. This process usually involves comparing records using an approximate string similarity measure.
Here are a few related questions on the site that I'm aware of:

About string similarity measures, you might find some info in these questions.
About record linkage, I'm aware of these questions.

(Note that there are certainly many other questions/resources available, I just mentioned the ones I remember)


Open source data science projects to contribute

The Julia project is one which I actively contribute to, including the advanced computing and XGBoost libraries. So, I can definitely vouch for it's maintenence and the quality of the community.

Some really good open source data science projects where even the beginners can contribute are:

Sklearn: Always developing at a rapid pace, the sklearn community is always open to new developers and contributors.
H2O: H2O is another fast growing data science projects, working on scalable machine learning and Deep Learning solutions.
Go: Open source data science road map and resources. Not really a technical project, but is very helpful for absolute beginners and aspiring analysts.
Pylearn2: Another fast growing Machine Learning and Deep Learning project.
Vowpal Wabbit: The Vowpal Wabbit (VW) project is a fast out-of-core learning system sponsored by Microsoft Research and (previously) Yahoo! Research.

Here is a Quora discussion on such projects and some more which are not mentioned in this answer.

Here is a another nice discussion about open source Data Science and ML projects in Python.


There are methods that approximate Jaccard similarity using hash functions, one of them MinHash. They scale well with the number of dimensions and  evaluate similarity of two items. If you want to do nearest neighbor queries (finding similar items given one item) you should look at locality sensitive hashing (LSH), which essentially maps similar items to the same hash value. There is a nice implementation in scikit-learn called LSHForest. It approximates cosine distance and scales well to large number of items and dimensions.


Should we have only binary labels for LSTM?

Welcome to the site. Generally speaking, you can have any labels you need/want and I don't think that the "essence" of LSTM will be affected. Recall that LSTM is special because it has the ability to "forget" and throw away data (mostly data that leads to factor fading). So, your choice of labels will not prevent you from leveraging the wonderful benefits of LSTM.

However, it's possible that you are confusing your question with one-hot-encoding. You should definitely take your labels and convert them to one-hot-encoding during both your training and prediction cycles. You're not changing your labels per-se, you're merely creating them in a way that allows you to better use them in neural networks (all neural networks; one-hot-encoding is not unique to LSTM).


Sounds like a job for Dynamic Time Warping, there are implementations in Python and R.


How each layer of a neural net is responsible for one feature

... each layer of a neural network is responsible for recognizing one feature of the input data. For example, if we build a neural network that classifies cars, buses, vans and bicycles, a layer will be responsible to recognize the tires, another one will responsible for recognizing the size of the vehicle.

There are numerous kinds of neural networks and there are different differentiable components that are used inside them to achieve end-to-end learning. Convolutional layers are responsible for finding the features that are essential for reducing the error. These features are shared among different nodes and are not necessarily meaningful to us. They have to be used simultaneously due to being shared among different weights. What you are trying to say is that in convolutional networks, the first layers attempt to find low-level features like vertical and horizontal edges whilst deeper convolutional layers try to find high-level features which are more abstract. But the point here is that they attempt to recognize and the way they do that is by using weights and activations which are shared in different activation maps. Consequently, one neuron is not necessarily responsible for faces or for cars. There may be different neurons and filters which are responsible for them. Take a look at the link which has illustrated that.

The other point is about fully connected layers which are responsible for classification. What they do is finding decision boundaries to classify inputs or estimating a function for regression tasks. They are not feature extractors in a way convolutional layers do. As it is illustrated in the cited link, what they try to do is to separate the input space to make it possible for generalization in the current feature space.

  The question is, why is this true? i.e. each layer appears to perform similar to the others and there is no special design for each one. Is there any way to assign each layer a specific feature or it is done implicitly?

It was not true as mentioned. Although each layer seems similar to other layers, because of the cost function, their corresponding weights are set to decrease the cost function. Consequently, for fully connected layers all the featrues are feed to all the neurons in each layer and the training process decides how to use them. Moreover, the convolutional layers try to find similar patterns, kernels, in the input pattern.


kNN - what happens if more than K observation have the same distance to the centroid of the cluster

K-means does not make an assumption regarding how many observations should be assigned to each cluster. K is simply the number of clusters one chooses to generate. During each iteration, each observation is assigned to the cluster having the nearest mean. There is no assumption that all clusters should have a comparable number of observations assigned (i.e., for N observations, there is no expectation that each cluster should have ~ N/K observations assigned).

It is quite possible that the numbers of observations in the various clusters are highly imbalanced. This can be due to the distribution of the data, the number of clusters chosen (K), or even how the cluster means are initialized.


Explaining feature_importances_ in Scikit Learn RandomForestRegressor

scikit-learn's RandomForestRegressor feature importance is computed in each tree composing the forest. You can find the source code here (starting at line 1053).

What it does is, for each node in the tree where the split is made on the feature, it substracts each child node's (left and right) impurity values from the parent node impurity value. If impurity decreases a lot (meaning the feature performs an efficient split), it basically gives a high score. Of course, all that is weighted depending on how useful the test is for the result: a split between two individuals gives a high impurity decrease, but is trivial, so much easier than between two large populations.

Once the feature importance has been determined for each tree, it is summed up and normalized so that the feature_importances_ vector sums up to 1.

It might introduce some biaises, I guess that it should be the case if variables are not scaled, for instance. However it is quite easy to compute (you just have to read impurity values from the tree), so I guess this is why it is provided by default. But the method is not unequivocal, there are other methods out there that you can implement more or less manually:

shuffling a feature's values in the dataset
reverting the result of each test based on the feature

... and probably a few other approaches. All those will give you other scores that may be of help.


Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the norms, so you can isolate the terms:

dotproduct(d_1,d_2) = tfidf(w1,d1) * tfidf(w1,d2) + tfidf(w2,d1) * tfidf(w2,d2) + ... + tfidf(wN,dN)

Ranking the words w_i by descending order of the term tfidf(w_i,d1) * tfidf(w_i,d2) gives the top most contributing words for the similarity score.

Mind that if the documents have big differences in size, this will have an effect since cosine is normalized with their norms.


Guidelines for selecting an optimizer for training neural networks

AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several problems often the most critical information is present in the data that is not as frequent but sparse. So if the problem you are working on deals with sparse data such as tf-idf, etc. Adagrad can be useful.

AdaDelta, RMSProp almost works on similar lines with the only difference in Adadelta you don't require an initial learning rate constant to start with.

Adam combines the good properties of Adadelta and RMSprop and hence tend to do better for most of the problems.

Stochastic gradient descent is very basic and is seldom used now. One problem is with the global learning rate associated with the same. Hence it doesn't work well when the parameters are in different scales since a low learning rate will make the learning slow while a large learning rate might lead to oscillations. Also Stochastic gradient descent generally has a hard time escaping the saddle points. Adagrad, Adadelta, RMSprop, and ADAM generally handle saddle points better. SGD with momentum renders some speed to the optimization and also helps escape local minima better.


Dealing with a dataset having target values on different scales?

TL:DR The most common and straight-forward approach would be to scale all the numerical data to be within a given range. This makes the currency differences irrelevant, as all fluctation remains constant, relative to the original scale.

Rescaling

Here is an example function to scale the numerical columns of your dataset:

def rescale(data, new_min=-1, new_max=1):
    """Rescale the columns of Dataframe to be in the range [new_min, new_max].

    Parameters
    ----------
    data    : a Pandas DataFrame
    new_min : the target minimum for each of the columns, optional
    new_max : the target maximum for each of the columns, optional

    Returns
    -------
    out : the rescaled input data, with each column now in the range [new_min, new_max]

    """
    return (data - data.min()) / (data.max() - data.min()) * (new_max - new_min) + new_min

You can then do the following to see nice descriptions of each column in your dataframe, ensuring the min/max values are as desired:

your_dataframe.describe()

One thing to perhaps try out, would be whether or not to scale **all* data to be within a fixed range, say [0, 1], or whether to scale each individual currency to the range [0, 1]. That equates to applying the above function on either single columns of your dataframe, or the entire dataframe.

Considerations

Things to keep in mind/check for would be:

if some currencies are nominal huge compared to others (e.g. 1 euro = 17,000 Indonesian Rupiah), the smaller currencies will have time-series values all very close to zero - this makes computation/optimisation more difficult - especially if you use methods such as gradient descent.
Scaling the currencies individually to a range would lose the relativity of their nominal values. To see this, plot the currencies before and after scaling.

Unify the Currencies

If you have (or could get) exchange rate the data, another alternative would be to convert all of the currencies to the one one your choice.

As the exchange rate changes over time, this would also inherently introduce new information into the dataset, namely the relative economic health of each of the countries. Depending on your use case, this could be something that really helps, or you might really want to avoid. If you have to time, to do it, it could definitely make for some interestinf research!

You can get currency data from Quandl - they have a Python API too, which is simple enough to use.


Trying to use different image size to test my trained model

Generally image classification models require input images to be reshaped, because they use fully connected layers to classify features learned by the previous convolutional layers, and shapes must match to make classification. On the other hand, fully connected CNNs only use convolutional layers, so they can work images with any shape.
If your model has fully connected layers, then you should reshape your input images.  If you send your images via network to get predictions, then reshaping images before sending can decrease network load, and response time for big images. If it is not the case I think you simply can reshape them right before you input them to model.


Why could my SVR prediction be offset, but still follow the actual pattern?

Two things that come to my mind:

A) - have you compared the distributions with a Kolgomorov Smirnov Test or similar - (aka: are they really the same. Sure there is the big spike but besides its not that similar (by eye))

B) - Are training and test in the same magnitudes for the input data? If they are not this might confuse your SVR.


How to store subset of columns from a csv file?

Hive's RegexSerDe which is often used to process logs could be used in your use case. You could use the regex to extract the first 3 columns from a line

CREATE EXTERNAL TABLE file (A STRING, B STRING, C STRING)
AS
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
    WITH SERDEPROPERTIES
      ('input.regex'='^(\\w+)\\t(\\w+)\\t(\\w+)(.*)')
LOCATION 'input/loading/'

The regex included is using pattern groups (\w+) to extract the first 3 columns and the last group (.*) is anything else which may be in the line

See more details here Community Article and Official Documentation


Pytorch: Starting with a high loss value, but the loss converged at the end. I dont know if the model could start with a loss > 100. Help!

The value of the loss in the first epochs is irrelevant. The network weights are initialized at random, so it is perfectly expected that the behavior of the network differs from the desired one.
However, this does not mean that you should only look at the last value of the training and validation loss. Instead, you should plot the whole curves, to understand the training dynamics. From the curves, you can see whether your network overfitted or underfitted, if there's still room for improvement if you train the model for longer, etc.
There are many questions in this site asking about how to interpret training/validation curves (e.g. this and this). You may take a look at them to understand how to interpret yours. From the values you posted, it seems there's still room for improvement if you train for more epochs.


Supervised learning: classifying/learning a single discrete value

I don't really see this as a learning problem really, if it's the same malicious value you have to watch out for..

Only one "positive" sample (corresponding to the malicious value) would be enough to find the malicious integer.

Have I misunderstood something in your question?


Index for efficient argmax(w.x) query ~ 20d

The query is called a 'top-k' query, and you can answer it quickly using the ranking cube approach. http://www1.se.cuhk.edu.hk/~hcheng/paper/vldb06_rankcube.pdf

The paper does not derive the time complexity.


what does one Shot learning mean? do they only need one image to train for some new class detection?

One-Shot Learning refers to the problem when you only have very few or a single sample for some classes in your training dataset. A common application is, for example, face recognition. Here you may have only a single image per person in your dataset. Nevertheless, you'd like your neural net to be able to recognize that person from new images. A good intro is provided by Andrew Ng here.
A popular example are Siamese Nets introduced by Koch et al. The basic idea is to learn a latent representation of the images in your training set. When a new image is presented during inference the net calculates the latent representation of that new image and searches for the image in your training set whose latent representation is most similar to the one of the new image and predicts the corresponding class.
When you do not have any sample of a specific class available this is considered Zero-Shot Learning. However, in this case the neural net requires the dataset to include some auxiliary information for all images. The most popular being the Animals with Attributes (AWA) dataset:

Initial approaches to solve the Zero-Shot Learning problem did so in two steps:

For a given image learn to predict the attributes
Learn to predict the class based on the attributes

For details you may refer to the paper Attribute-Based Classification for Zero-Shot Visual Object Categorization. More recent approaches take another route though.
However, both approaches require, as always in Deep Learning, large datasets to learn.


Cost sensitive classification with individual cost

There are two types of classification costs: per class, and per instance.

In keras, for instance cost, we assign a cost to each training sample by feeding sample_weight to .fit. For example, if we have four training samples in rows 1, 2, 3, and 4, with misclassification costs 2.5, 1.5, 1.0, and 1.0, we feed sample_weight=[2.5, 1.5, 1.0, 1.0].

For class cost, if there are three classes 0, 1, and 2, with misclassification costs 1.0, 3.0, 1.0, we feed class_weight=[1.0, 3.0, 1.0].

Here is a step-by-step classification example in Keras.

You should feed your weights in this line:

model.fit(train_images, train_labels, epochs=5)


Typically, you won’t use plots to show anything larger than 3D. Yes, gradient descent will always go down the curve of convex loss function, I.e, linear regression.

The gradient value is simply the derivative “slope” of the loss with respect to the inputs. It capture how “off” the output is on average for each feature. For example, if the error of an record is high and the feature is lower than optimal, the gradient will point in the direction that increase the feature’s value.


Sequentially Training Certain Layers/Sub-Networks in Keras Functional API

Figured it out. To train certain sub-networks first and re-use the trained weights to initialize other layers:

from keras.layers import Input, Dense
from keras.models import Model

main_input = Input(shape=(5, ))

## Model A: main_input -> A_output
layer_A1 = Dense(10, name='A1')(main_input)
layer_A2 = Dense(10, name='A2')(layer_A1)
layer_A3 = Dense(10, name='A3')(layer_A2)
A_output = Dense(1, name='A_output')(layer_A3)

# Train Model A first
model_A = Model(inputs=main_input, outputs=A_output)
model_A.compile(# args)
model_A.fit(# args)

## Model B: main_input -> layer_A2 -> B_output
layer_B1 = Dense(10, name='B1')(main_input)
B_output = Dense(1, name='B_output')(layer_B1)

model_B = Model(inputs=main_input, outputs=B_output)

# Set weights of layer_B1 and freeze; essentially a copy of pre-trained layer_A2
trained_A2_weights = model_A.layers[2].get_weights()
model_B.layers[1].set_weights(trained_A2_weights)
model_B.layers[1].trainable = False

model_B.compile(# args)
model_B.fit(# args)
```


Should we convert independent continous variables (features) to categorical variable before using decision tree like classifier?

There is no need to split continuous variables because the tree already does that automatically. The only way you can test for overfitting is by either using a holdout set or by doing cross validation. If you are overfitting, changing a continuous variable to a categorical variable likely won't make a difference. If you get the sense that you're overfitting, you should reduce the depth of your tree.


Delete the label column.

Assuming that you want to compare the clusters to the labels later, then the labels must not be part of the data passed to k-means.

And k-means only works well on continuous variables anyway.


List of NLP competitions on Kaggle by popularity [number of teams]:

Two Sigma Connect: Rental Listing Inquiries [2709 teams]
https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries
Home Depot Product Search Relevance [2125 teams]
https://www.kaggle.com/c/home-depot-product-search-relevance
Quora Question Pairs [2123 teams]
https://www.kaggle.com/c/quora-question-pairs
What's Cooking? [1388 teams] https://www.kaggle.com/c/whats-cooking
Crowdflower Search Results Relevance [1326 teams]
https://www.kaggle.com/c/crowdflower-search-relevance
Bag of Words Meets Bags of Popcorn [578 teams]
https://www.kaggle.com/c/word2vec-nlp-tutorial
Transfer Learning on Stack Exchange Tags [380 teams]
https://www.kaggle.com/c/transfer-learning-on-stack-exchange-tags/data
Facebook Recruiting III - Keyword Extraction [367 teams] https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction

New entrants are prohibited, you won't be able to download data for this competition

Dato: Truly Native? [274 teams] https://www.kaggle.com/c/dato-native
The Allen AI Science Challenge [170 teams]
https://www.kaggle.com/c/the-allen-ai-science-challenge


How to handle multi-label feature for binary classification problem?

I recommend using scikit-learn's MultiLabelBinarizer to encode your category feature.  This is the multi-label analog of one-hot encoding, but it has a sparse_output mode, which should solve your problem of limited memory.


You must be getting a type error because the elements of text_features and/or author_signatures are not able to be subtracted in the part i - j.

Depending on what they are, you could try converting them. For example, you could try making one or both of them numbers or strings like this:

Diff = [float(i) - float(j) for i,j in zip(text_features, author_signature)]   # both made into numbers

Diff = [str(i) - str(j) for i,j in zip(text_features, author_signature)]   # both made into strings (words)

To give better advice, you need to give an example of what your two lists are that are in the zip()

EDIT

Based on the update in the question, I see that it works as expected for me:

In [1]: text_features = [1, 2, 3]

In [2]: author_signature = [3, 2, 1]

In [3]: Diff = [i - j for i,j in zip(text_features, author_signature)]

In [4]: Diff
Out[4]: [-2, 0, 2]


How to interpret the sample_weight parameter in MiniBatchKMeans?

Yes, the parameter is available in the vanilla K-Means too.

The algorithm supports sample weights, which can be given by a parameter sample_weight. This allows assigning more weight to some samples when computing cluster centers and values of inertia. For example, assigning a weight of 2 to a sample is equivalent to adding a duplicate of that sample to the dataset X.

scikit-learn.org


How to handle associated features in machine learning

The RandomForest algorithm may capture such interaction by splitting
an upper node by some condition on p1 and a lower node by another condition on p2, the link that you are expecting is captured if this happens under the same branch (p2 is on the path of p1). However, the ability to capture such information depends on the structure of the whole model (depth and number of trees) and the number of labels that you use.
It is a common practice to extract a third feature from coupled
features to explicitly express the linkage between them, for example, the ratio of change or the product. For not losing important information as you suggested (since a diff of 0.1 may be derived by subtracting 0.9-0.8 and 0.2-0.1) you may add to the model the third feature in addition to one of the original features.
I'm not sure that I understand the meaning of out-of-fold, but anyway note that using your labels twice (once for features preprocessing and once for training) can lead you to overfit the model since the labels leaked to the features in the preprocessing step. It is ok to explore your features in advance and in separate to the labels for understanding the links, subcategories, missing values, measuring the correlations and so.

2 last note:

there might be a confusion here between the meaning of correlation and interaction.
you can consider using linear statistical modeling for your needs and number of data points. these models enabling to capture interactions and with their output you can learn about the significance of the correlation and to interpret them with their features importance.
-- If you do so, you could also consider trying GLMM to see if there
are job effects, i.e there is a difference in the interaction between
different jobs.


Some of the "tasks" (there are many more, for example anomaly detection; and people just love to invent their own tasks marginally different from before hoping to be the first to spawn a new subdomain...) can both be supervised or unsupervised.

Example:

PCA is unsupervised dimensionality reduction
Fishers' LDA is supervised dimensionality reduction

... and both predate any notion of machine learning, data science or data mining. These things are just rebranded into ML now, squeezed into a very narrow view that never was able to capture the essence of even clustering (beyond k-means, that seems to be the only method machine learners still understand and get taught, as the L2 nature of the method fits their world view well enough). I'd rather avoid making all this "machine learning" - it was not, and it doesn't fit well.


What is the difference between a Categorical Column and a Dense Column?

Sparse vs Dense

The 'categorical column' is a sparse column. Sparse and dense columns (or matrices) are in a way opposites of each other.

Sparse columns usually contain a lot of zeros. Whereas, dense columns have more non-zero entries. This matters, because the way they are stored and processed can differ.

Sparse

If we take your sparse example:

ID   Represented using one-hot encoding
0    [1, 0, 0, 0]
1    [0, 1, 0, 0]
2    [0, 0, 1, 0]
3    [0, 0, 0, 1]

What information needs to be stored to reconstruct this matrix? All you need is:

The number of columns and rows
Value for every non-zero entry

So, by having:

n_columns = 4
n_rows = 4
non_zero_entries = {
    (0, 0): 1,
    (1, 1): 1,
    (2, 2): 1,
    (3, 3): 1
}

you can reconstruct it. This means you don't need to store all the 16 integers to reconstruct the matrix.

In Tensorflow, the 'categorical column' matrix, has the special property that it is a one-hot encoding, meaning that one and only one entry in the columns is non-zero. However, the sparse principle doesn't need to be one-hot.

For example:

matrix = [
    [0, 0,   0, 0],
    [0, 2,   0, 1],
    [0, 0, 0.5, 0]
]

Can be reconstructed with:

n_columns = 4
n_rows = 3
non_zero_entries = {
    (1, 1): 2,
    (1, 3): 1,
    (2, 2): 0.5
}

Imagine a matrix of size $100 \times 100$. There are $10,000$ values in it. Now, suppose only $1$ entry is non-zero (for whatever reason). Then, this large $10,000$ values matrix, can be represented by:

n_columns = 100
n_rows = 100
non_zero_entries = {
    (x, y): 1
}

Dense

One the other hand, dense columns are not stored by defining only the non-zero entries. The 0s are also being stored.

While more memory intensive, they are less computationally expensive because they don't need to be reconstructed. Also, often the 0s are likely to be changed over time, and if there is already an int being stored, no new memory locations need to be assigned.

Usability

Not all algorithms can process a sparse matrix. Often, matrix operations require the whole vector be present. Depending on the implementation, the assumptions that non-present elements are zero is not there, therefore an explicit zero needs be stored and passed along.

Like you mentioned in your questions:

  But you also have a dense column called indicator column, which
  'wraps'(?) a categorical column to produce something that looks almost
  identical

This is true. What they represent can be exactly the same. However, how they are stored and the efficiency of the implementations differ.


Reinforcement (Q) learning: does it learn while in production?

Now, if I deploy this model (i.e., use this model to make predictions), does it keep learning (i.e., updating the Q values)?

If you want it to (and understand how to code it) then yes a reinforcement agent - including a DQN-based one - can do this. This is online learning, and is possible also with many supervised learning techniques.

Because there is risk of the agent learning incorrectly, you may choose to limit online learning, or disable it in production.  It could also happen by mistake if you are not sure how to stop it, and simply deploy your existing agent code from the training scripts into production. So make sure that you, or whoever you get to implement production version of the agent, understands how to control whether learning is still occurring.

You may choose to enable online learning if your initial training was in simulation, and you would like the agent to learn more from real-world interactions. Or you might choose to do so if the problem was non-stationary because the environment description changes over time. Many problems involving interacting with a population of people change over time with demographic changes.

If the environment is stochastic, then in theory you could also switch off exploration - typically by setting $\epsilon = 0$ in an $\epsilon$-greedy behaviour policy. That would allow the agent a (limited) ability to refine its Q value estimates with a relatively low risk. The agent would continue to attempt to act optimally, but may learn enough to decide that different actions were optimal. Note this is still not without risk, because the learning process could fail in some way, leading the agent to predict a wildly incorrect optimal action.

Allowing Q learning to explore non-optimal actions - typically by setting $\epsilon > 0$ in an $\epsilon$-greedy behaviour policy - is more risky in production, because the agent will occasionally pick a non-optimal action in order to refine its estimates of that action. It may result in improved learning in the longer term though, so you might decide to do that if the consequences of non-optimal actions were mild.


Unknown program 'spark-itemsimilarity' chosen

Spark support for Mahout came from Mahout 0.10 release while you are using 0.9 release. So this should explain why you get the unknown program error. I would suggest using a higher version of Mahout.


The purpose of max pooling operation is to decrease the spatial dimensions of the input while also being robust by only considering the maximum values.
Generally, as you might have noticed, most CNNs aim at decreasing the spatial dimension of the input while increasing its depth. Very broadly speaking, you can think of this as trying to encode the information enclosed in the spatial image into channels representing different aspects of the input.
To answer your question, It would not have been an error to max pool the input right away, however, in that case, the input would not have been through enough layers to extract its enclosing information. Having said that, max pooling that early might result in losing spatial information while not gaining anything valuable depthwise.


Designing a pretrained DNN for image similarity

That is not currently possible.

There is no such thing as a database that supports arbitrary content-based image indexing or image queries. Image Databases (IDBs) are indexed by R-tree family or members of the quadtree family. Those only support limited SQL (text) queries.

You would have search image-by-image for each query image which would defeat the purpose of having a database.


This is well answered in this earlier question:

https://stackoverflow.com/q/31447141/1060350

Beware that Spark k-means is slow.

If your data fits into main memory (i.e. a few gigabyte, which means billions of vectors!) then other tools such as ELKI that don't have the cluster overhead will be much faster. Use spark only for preprocessing the data, if you e.g. have several TB of jsons, and you need to first extract the numbers out of the JSONs, then here is where Spark shines.

Once your data is then vectors, use ELKI instead, it's much faster.


Multiple testing of unbalanced data by R

It sounds like you are using Welch's t-test to compare groups of different sizes. That should work in your situation. It's more reliable that the standard Student's t-test when you have unequal variances and unequal sample sizes.

Note that sample sizes do not have to be identical for a Student's t-test. The important thing is that the variances are the same. If the variances are not equal, this is where the extra robustness of Welch's becomes helpful.

Whether or not you can conclude significance depends on what P-value you are you are looking for. It sounds like you are targeting p<.01, so if your result is less than .01, you can report that you found a significant difference.


Features for blink detection in real-time single channel EEG

I would try a combination of the following:

A rolling-window variance to detect unusually large deflections
Bollinger bands to detect the spikes
Autocorrelation to detect the 'up down up' pattern


is Batch Norm a little bit stochastic by default?

While I am no expert in batch normalization, I have noticed that in the paper they include a constant, epsilon (ε) to the minibatch variance for numerical stability (in the normalize step from this paper, page 3). So while your math might be right and verified by others, the actual computation could still be wrong.

This issue appears here. The solution on the bottom is to add variance clipping.

And with a stab at why you might be seeing it more with smaller learning rates. Well that might be just because you're taking a lot more steps to get to the same accuracy so that's just many more steps of numerical instability.
EDIT: I now realize when you say full-batch gradient descent you mean you are doing the entire dataset before updating the weights (just standard gradient descent). This means that with a smaller learning rate you will have to be doing more steps... so I'm sure that explains the learning rate difference.


I would suggest Recurrent Neural Nets. They are good for time series, however they need a huge dataset to get good performance. Here you can find an implementation in torch.


As you can read from here

plt.subplots() is a function that returns a tuple containing a figure
and axes object(s). Thus when using fig, ax = plt.subplots() you
unpack this tuple into the variables fig and ax. Having fig is useful
if you want to change figure-level attributes or save the figure as an
image file later (e.g. with fig.savefig('yourfilename.png'). You
certainly don't have to use the returned figure object but many people
do use it later so it's common to see. Also, all axes objects (the
objects that have plotting methods), have a parent figure object.

The parameter bins is used to set the number of ranges to be used to accumulate the data in those ranges. I don't know where else you have question.


Voting classifier using grid search for Time Series

In a time series problem, your input (x) and output (y) are, in the most basic case, the same variable, recorded at different points in time.

If you have T time points: 1, 2, 3, ... T then we think of x as an array of data points, with an index t to access each time point.

Typically y will simply be your x array shifted forward in time (in the below example, by 1 time unit, so $y_t = x_{t-1}$ in vector notation or y[t] == x[t-1] in array notation):

  x  |   y
-----|-----
 0.1 |  NaN
 0.2 |  0.1
 0.3 |  0.2
 0.3 |  0.3
 0.4 |  0.3
 0.3 |  0.4
 0.5 |  0.3
 1.0 |  0.5

Pandas has a shift() method for a time series, which allows you to shift your x column at different time intervals, and create a new y column using that shifted series. See https://stackoverflow.com/questions/10982089/how-to-shift-a-column-in-pandas-dataframe

You can add levels of complexity to this by including multiple time lags, as well as other variables, but this explains the basic principle of how to convert time series forecasting into a supervised learning problem.


How to represent the number of neurons in an LSTM for architecture schematic?

Within the recurrent cell, you can add the hidden units- often, we se this notated as:

Unit 1
  ...
Unit n

Where n is the dimensionality of the recurrent representation.


How does testing a CNN work after training?

The last layer of your neural network is a sigmoid function. During training, all the parameters are being learned. Hence, given a new picture, we can compute a value that is between $0$ and $1$ from the sigmoid function (which is used as a model of probability) to let us know the desired probability.

For multi-class classification problem. A softmax layer is used instead.


Spacy v2.0.1 custom NER: How to improve training of existing model

The model depends entirely on the training data: if you train with some data which has only PrdName as label, the model knows only this label and can predict only this label. You need to provide as much training data as possible, containing all the possible labels.

For the record, NER are usually trained with thousands of sentences in order to account for the diversity of the cases where a NE can appear.


If you're prepared to use a neural net, it might be worthwhile looking into spatial pyramid pooling.

A spatial pyramid pooling layer can take inputs of varying sizes (eg from  a convolutional layer) and convert them to an output of a fixed size (to pass on to a fully connected layer for instance). This is probably most useful where features are spatially related in some way.

Alternatively, you could continue simply implementing pca on your inputs, selecting the N dimensions with the highest variance for each example, but train the model continuously through reinforcement learning.

As there is more to how data is distributed along an axis than the variance, it is unlikely that any one pretrained model will perform well when your preprocessing step is constantly swapping out features, which is why a reinforcement model would be needed.  Combined with the pca, hopefully that won't take up too much overhead.


How to add non-image features along side images as the input of CNNs

My solution is like your first recommendation, but with slight changes.

Construct your convolutional layers and stack them till the flatten-layer. This network should be fed with the image data.
Flat your activation maps
Construct a fully connected network with the desired number of neurons and layers.
Concatenate the outputs of the flattened layer of the convolutional net and the fully connected net.
add some dense layers and connect them to the last layer which represents your classes.

You can use customary cost functions for this architecture.


What can functional programming be used for in data science?

One reason why functional programming could be useful for data science is that it lends itself more easily to parallel and distributed programming, e.g. the popular frameworks Apache Spark for cluster computing and Apache Kafka for stream-processing are both written in Scala (and Java).

Other than that "functional programming" as a skill is not directly related to data science. It's a tool that may facilitate some practicalities of data science and therefore more relevant for the "data engineering" aspect of data science.

It's useful but probably not necessary. It depends on your interests.


Dimensionality reduction with prior knowledge of colinearity between features

One option is to use Multidimensional scaling (MDS) for dimensionality reduction. MDS can create a visualization of the relative positions for data based on the distance between the data points.

In your example, the data points that have no distance between them in several dimensions will be projected close to each other.


I would phrase it like so:

"Of all records that were labelled 1 by the model, 10% were actually 1 (90% incorrect predictions). Of all records that were truly labelled 1 we predicted 83% correctly."

While this is out of context of your question, if support refers to the number of records then it would be beneficial to get a more balanced dataset. The reason your precision is so poor for label 1 is because there are many more "negatives" (0's) than "positives" (1) increasing the chance for false positives to occur, affecting your precision.

EDIT:
This on Cross Validated will help provide more explanation.


Why convolutions always use odd-numbers as filter_size

The convolution operation, simply put, is combination of element-wise product of two matrices. So long as these two matrices agree in dimensions, there shouldn't be a problem, and so I can understand the motivation behind your query.

A.1. However, the intent of convolution is to encode source data matrix (entire image) in terms of a filter or kernel. More specifically, we are trying to encode the pixels in the neighborhood of anchor/source pixels. Have a look at the figure below:

Typically, we consider every pixel of the source image as anchor/source pixel, but we are not constrained to do this. In fact, it is not uncommon to include a stride, where in we anchor/source pixels are separated by a specific number of pixels.

Okay, so what is the source pixel? It is the anchor point at which the kernel is centered and we are encoding all the neighboring pixels, including the anchor/source pixel. Since, the kernel is symmetrically shaped (not symmetric in kernel values), there are equal number (n) of pixel on all sides (4- connectivity) of the anchor pixel. Therefore, whatever this number of pixels maybe, the length of each side of our symmetrically shaped kernel is 2*n+1 (each side of the anchor + the anchor pixel), and therefore filter/kernels are always odd sized.

What if we decided to break with 'tradition' and used asymmetric kernels? You'd suffer aliasing errors, and so we don't do it. We consider the pixel to be the smallest entity, i.e. there is no sub-pixel concept here.

A.2
The boundary problem is dealt with using different approaches: some ignore it, some zero pad it, some mirror reflect it. If you are not going to compute an inverse operation, i.e. deconvolution, and are not interested in perfect reconstruction of original image, then you don't care about either loss of information or injection of noise due to the boundary problem. Typically, the pooling operation (average pooling or max pooling) will remove your boundary artifacts anyway. So, feel free to ignore part of your 'input field', your pooling operation will do so for you.

--

Zen of convolution:

In the old-school signal processing domain, when an input signal was convolved or passed through a filter, there was no way of judging a-prior which components of the convolved/filtered response were relevant/informative and which were not. Consequently, the aim was to preserve signal components (all of it) in these transformations.

These signal components are information. Some components are more informative than others. The only reason for this is that we are interested in extracting higher-level information; Information pertinent towards some semantic classes. Accordingly, those signal components that do not provide the information we are specifically interested in can be pruned out. Therefore, unlike old-school dogmas about convolution/filtering, we are free to pool/prune the convolution response as we feel like. The way we feel like doing so is to rigorously remove all data components that are not contributing towards improving our statistical model.