Combining Latitude/Longitude position into single feature

A note: for those who've ended here looking for a hashing technique, geohash is likely your best choice.

Representing latitude and longitude in a single linear scale is not possible due to the fact that their domain is inherently a 3D space. Reducing that as per your needs would require a spatial flattening technique that's unheard of to me.

Reasoning

As far as lat/long merging goes, the best of best practices would be to resort to the Haversine formula, which calculates the distance between two points over a spherical surface, and receives those points' coordinates as input.

One way to incorporate that in your use case - where each point should probably have an independent lat/long combination - would be to assume the distance's origin point coordinates to be $(\varphi_1, \lambda_1) = (0, 0)$, which would render

$$d =2r \arcsin\left(\sqrt{\sin^2\left(\frac{\varphi_2 - 0}{2}\right) + \cos(0) \cos(\varphi_2)\sin^2\left(\frac{\lambda_2 - 0}{2}\right)}\right)$$

$$= 2r \arcsin\left(\sqrt{\sin^2\left(\frac{\varphi_2}{2}\right) + \cos(\varphi_2)\sin^2\left(\frac{\lambda_2}{2}\right)}\right)$$

With $r$ being Earth's radius (~6371km) and $(\varphi_2, \lambda_2)$ your point's latitude and longitude, respectively.

However, as stated before, that couldn't possibly give you a linear relation, as you can see by 3d plotting the function:

Implementation

The circumstances imply you're likely to be using pandas, or at least should be. Here's an example implementation of this relativized Haversine formula:

from math import radians, cos, sin, asin, sqrt

def single_pt_haversine(lat, lng, degrees=True):
    """
    'Single-point' Haversine: Calculates the great circle distance
    between a point on Earth and the (0, 0) lat-long coordinate
    """
    r = 6371 # Earth's radius (km). Have r = 3956 if you want miles

    # Convert decimal degrees to radians
    if degrees:
        lat, lng = map(radians, [lat, lng])

    # 'Single-point' Haversine formula
    a = sin(lat/2)**2 + cos(lat) * sin(lng/2)**2
    d = 2 * r * asin(sqrt(a))

    return d

Which could be used as in the below minimal example:

>>> import pandas as pd

>>> df = pd.DataFrame([[45.0, 120.0], [60.0, 30.0]], columns=['x', 'y'])
>>> df
      x      y
0  45.0  120.0
1  60.0   30.0

>>> df['harvesine_distance'] = [single_pt_haversine(x, y) for x, y in zip(df.x, df.y)]
>>> df
      x      y  harvesine_distance
0  45.0  120.0        12309.813344
1  60.0   30.0         7154.403197


Implemented early stopping but came across the error SGDClassifier: Not fitted error in sklearn

It is because clone will only copy the estimator with the same parameters, but not with the attached data. So it results a new estimator that has not been fit on the data. Hence, you couldn't use it to make prediction.

Instead of clone, you can use either pickle or joblib.

1. pickle

import pickle
...

for epoch in range(1000):
    ...
    if error < minimum_val_error:
        best_model = pickle.dumps(sgd_clf)

Later if you want to use the stored model:

sgd_clf2 = pickle.loads(best_model)
y_test_predictions = sgd_clf2.predict(X_test)

2. joblib

You can also use joblib, and store the model to the disk.

from sklearn.externals import joblib
...

joblib.dump(sgd_clf, 'filename.joblib')

To use the stored model

clf = joblib.load('filename.joblib')


That's a particular peculiarity of TSNE in sklearn:  https://github.com/scikit-learn/scikit-learn/issues/5361


Unable to understand Kernel Ridge regression

Equation (4) simply gives us an identity (without proof):

$$(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} = PB^T (BPB^T + R)^{-1}.$$

Let's check that it does indeed hold. The left hand side equals

\begin{align*}
(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} &= \left(\left((P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1}\right)^{-1}\right)^{-1}\\
&= \left(RB^{-T}(P^{-1} + B^TR^{-1}B)\right)^{-1}\\
&= \left(RB^{-T}P^{-1} + B\right)^{-1}.
\end{align*}
The right hand side equals
\begin{align*}
PB^T (BPB^T + R)^{-1} &= \left(\left(PB^T (BPB^T + R)^{-1}\right)^{-1}\right)^{-1}\\
&= \left((BPB^T + R)B^{-T}P^{-1} \right)^{-1}\\
&= \left(B + RB^{-T}P^{-1} \right)^{-1}.
\end{align*}
So indeed, the equality holds.

If you take the right hand side of equation (3)
$$\left(\lambda\mathbf{I} + \sum_i \mathbf{x}_i \mathbf{x}_i^T\right)^{-1}\left(\sum_j y_j \mathbf{x}_j\right),$$
replace $\mathbf{x}_i$ with $\phi_i$  and rewrite it as
$$\left(\lambda\mathbf{I} + \Phi\Phi^T\right)^{-1}\left(\Phi\mathbf{y}\right),$$
then it matches the left hand side of the identity in equation (4) (right-multiplied by $\mathbf{y}$), where
$$P^{-1} = \lambda^{-1}\mathbf{I}$$
$$B^T = \Phi$$
$$R = \mathbf{I}.$$

Therefore, we can rewrite the right hand side of equation (4) as

$$\lambda^{-1}\mathbf{I}\Phi(\Phi\lambda^{-1}\Phi + \mathbf{I})^{-1}\mathbf{y}.$$ As $\lambda$ is a scalar, we can rewrite this as
$$\Phi(\Phi\Phi + \lambda\mathbf{I})^{-1}\mathbf{y}.$$


Are there any annotators or Named Entity Recognition for license plate numbers?

There are a lot of pretty decent tools out there for text annotation in general, and given the broad nature of the task you're approaching (license plates are about as general as words), the annotation tools you are looking at should probably come from the more classical tools for annotation.

There was actually a pretty good discussion about annotation tools on this question, which should actually apply to this problem. The most relied-upon thing in annotation right now is probably brat. You can learn more about brat here.

Hope that helps! Let me know if you've got any more questions.


What is the best way to use word2vec for bilingual text similarity?

This paper from Amazon explains how you can use aligned bilingual word embeddings to generate a similarity score between two sentences of different languages. Used movie subtitles in four language pairs (English to German, French, Portuguese and Spanish) to show the efficiency of their system.

"Unsupervised Quality Estimation Without Reference Corpus for Subtitle Machine Translation Using Word Embeddings"


How to correctly perform data sampling for train/test split in multi-label dataset?

I'm facing two different strategies. The first one consists in preprocessing the corpus all together and then make the train/test split just before training. The second one starts with a pre-made train/test split, so the preprocess is made separately.

Strategy 2 is "correct" from a "don't reuse your data" perspective. Data pre-processing is part of training. Your train/test pipeline should, in principle, account for this. For example, if you have a feature $A$ with 5 categories, but only 4 of those categories appear in the data, the correct thing to do is to only use 4 categories in your one-hot encoding, and treat the 5th category as an "unknown" value.

In practice, however, this isn't always practical. In some cases the pre-processing pipeline is compute-intensive (e.g. training word vectors) or requires processing a massive amount of data (e.g. ingesting 5 GB of images off of AWS). In those cases, you have to accept the fact that some knowledge of your test set will "leak" into your training set by way of your preprocessing steps, but it's usually not so bad.

Other times you don't have enough data after train/test splitting to conduct preprocessing correctly. Or you didn't stratify your splits, and so you ended up with incomplete class representation, or and otherwise imbalanced training set. This is where you find yourself. Just look at the headings to see why:

Train:

|ID   |TEXT |A|B|C|

Test:

|ID   |TEXT |B|C|D|

Class "A" is missing from your test set, and class "D" is missing from your training set. This will ruin just about any model.

There has been some research done on the stratification of multi-label data. Notably, there is "On the Stratification of Multi-label Data" (Sechidis, Tsoumakas, and Vlahavas; ECML PKDD 2011), which introduces "iterative stratification". There is an implementation of this technique in the scikit-multilearn Python library.


Exporting Correlation Matrix (from function)

Don't use the matrix obtained from your function rather,

It looks like correlation is a DataFrame too, so you can simply use to_csv:

correlation_df(within your function itself).to_csv("C:\destinationfolder\file.csv")


Using Gini index, how to calculate the probability of correctly classifying a new randomly selected case to the highest probability class?

While you can calculate the underlying class probabilities from the Gini index (for binary classification), it'll be more straightforward to calculate it from the "value" line in each box. This line simply represents the number of samples that belong in each node, split by the target variable, so you can use this count to calculate a probability.
In the bottom-most node, for example, you have value = [4, 55], indicating that among 59 samples which belong in that node (samples that have ca > 0.5 and cp_4.0 > 0.5), 4 are No HD and 55 are Yes HD. Therefore, the probability of Yes HD in that node is 55/59.
The Gini index is a measure of how "pure" a node is - as this number gets closer to 0, probability values will become more extreme (closer to 0 or 1), indicating that the decision tree is doing a better job of discriminating the target variable.


What does the collate function in pytorch (geometric)?

What collate does and why:

  Because saving a huge python list is really slow, we collate the list into one huge torch_geometric.data.Data object via torch_geometric.data.InMemoryDataset.collate() before saving . The collated data object has concatenated all examples into one big data object and, in addition, returns a slices dictionary to reconstruct single examples from this object. Finally, we need to load these two objects in the constructor into the properties self.data and self.slices.

Source


The paper you refer to actually states the following intuition:

  Algorithm 1 estimates $E[f(X)|do(X_S = x_S )]$ by recursively following the decision path for $x$ if the split feature is in $S$, and taking the weighted average of both branches if the split feature is not in $S$.

It seems to be a slight modification of the original description in arxiv 1706.06060:

  For a tree model $E[f(x) | x_S]$ can be estimated  recursively using
  Algorithm 1, where $v$ is a vector of node values, which takes the value
  $internal$ for internal nodes. The vectors $a$ and $b$ represent the left
  and right node indexes for each internal node. The vector $t$ contains
  the thresholds for each internal node, and $d$ is a vector of indexes of
  the features used for splitting in internal nodes. The vector $r$
  represents the cover of each node (how many data samples fall in that
  subtree).

I presume $S$ is an item in the set of all subsets of N features.


Interpret results from a lightFM factorization machines

The prediction scores are only used for ranking. The scores themselves do not provide more insight than that.


As usual in these cases, there is no magic wand to determine which splitting method to use. It all depends on your specific data. Is your collected data redundant enough so that k-fold cross validation is not necessary? Does your data capture most of the input space?

Now, taking a look at your numbers, I'd say that the number of observations you have (145) is not likely to be large enough to capture all the potential variability in the input space, giving the fact that you have a high number of features (22). This conclusion, of course, depends on the type of the features (are they binary? categorical? numerical?), and whether all these features are actually necessary to make a prediction (are there redundant/correlated features? features that do not give any information about the output variables?).

In your case, and not knowing more about the data, I'd go for a cross-validation splitting scheme.


pairwise_distances with Cosine and weighting

Instead of using pairwise_distances you can use the pdist method to compute the distances. This will use the distance.cosine which supports weights for the values.
import numpy as np
from scipy.spatial.distance import pdist, squareform

X = np.array([[5, 4, 3], [4, 2, 1], [5, 6, 2]])
w = [1, 2, 3]

distances = pdist(X, metric='cosine', w=w)

# change the result to a square matrix
distances = squareform(distances)

Result:
array([[0.        , 0.05508882, 0.04898252],
       [0.05508882, 0.        , 0.07833123],
       [0.04898252, 0.07833123, 0.        ]])


Random forest minimum number of observations

The important is not the number of observation but the quality of this observations. If you have a look at toy datasets of sklearn they are way smaller than that.

Random forest is a good algorithm when there is small data since it is a bagging of decision trees with bootstrap. Each decision tree is feed with a sample of data with replacement, in this way even if the data is small there are bigger chances of making a good model.

In a high level, yes it seems a good way to go, but with out knowing more at the data is hard to tell.

I would suggest to give it a try with a Generalized Linear Model, a support vector machine and a gradient boosting.  Since your data is small you will not need much computation time for it.


Which order is correct Feature Selection then Outlier Detection or vice versa?

In majority of the cases feature selection should be done after outlier detection. Outlier detection should be done at the initial stage of data pre-processing while feature extraction / selection can be don in the last part of  data pre-processing.

Outlier must be detected beforehand so that actual behavior o that particular predictor is known. Then depending upon the behavior after removal of outlier it can be decided whether to keep that as feature or not.

Many feature selection algorithms like PCA, regression, etc are sensitive to outliers and so if such algorithms are used to extract the features then it would be better to remove the outliers beforehand.


How to make sure that the features learned by a neural network are not correlated?

Dropout (2014 paper) is my first thought.
By effectively removing N% of your neurons on each pass through the data, you make it harder for any two neurons to work together. When its buddy disappears it is forced to find another way to learn the patterns in the data. On the next epoch, when its old buddy comes back, it has a new perspective on life, its horizons have been broadened, and it is less likely to re-enter that co-dependent relationship.
The second thought was L2 regularization: it forces the network to not rely on just one neuron in a layer, but to get them all involved.
If you are not convinced, just try it. Try to create a network where, say, only two of the neurons in the final layer carry all the signal. Then re-train it using a dropout of 50%, or a higher L2 value. It may be a weaker model overall, but it is going to be really hard to not have all the neurons share the load.  (Also try it with and without the bias, as mentioned below.)

different and independent activation functions for each neuron within the layer.
Are there such networks?

I'd be interested to hear if you find anything. I suppose you could argue that the bias is creating a unique activation function on each neuron, though I think you are suggesting something like ReLU in some neurons and tanh in others? (A bit tricky to implement efficiently?)
A little bit removed from your question, but the different heads in multi-head attention (in the Transformer model) are another example of getting multiple different "activation functions" into a layer.


My corr() function in Python keeps resulting in an "ValueError: The truth value of a Series is ambiguous..."

You should look at the documentation.  You do not pass the column names as an argument.
subset_df = df[['col1', 'col2']]
subset_df.corr()

That should solve this for you.


What is the advantage of using log softmax instead of softmax

There are a number of advantages of using log softmax over softmax including practical reasons like improved numerical performance and gradient optimization. These advantages can be extremely important for implementation especially when training a model can be computationally challenging and expensive. At the heart of using log-softmax over softmax is the use of log probabilities over probabilities, which has nice information theoretic interpretations.

When used for classifiers the log-softmax has the effect of heavily penalizing the model when it fails to predict a correct class. Whether or not that penalization works well for solving your problem is open to your testing, so both log-softmax and softmax are worth using.


Events prediction with time series of continuous variables as features

First of all, you will not be able to prove anything with a model, you will have false positives/negatives. With a good model you may be able show what variables may be an indicator of component failure.

In problems like this feature generation can have the most important influence on the accuracy of the model. The time stamps can be used for aggregation. For example, you may aggregate metrics per device per hour. The metrics/features that you might create for input into the model might be average/max temperature or fan speed, rate of change in temperature or fan speed, number of seconds device was above some threshold temperature or fan speed, boolean indicator of voltage spike, etc. There could be any number of features you may create. You can then find which features are not strong predictors and remove these columns to reduce noise, if need be.


This problem is popularly called the "Credit Card Fraud Detection"

There are several classification algorithms, which aim to tackle this problem.

With the knowledge of the dataset you possess, the Decision Trees algorithm can be employed for detecting malicious transactions from the non-malicious ones. This paper is a nice resource to learn and develop the intuition about fraud detection and the usage of basic classification algorithms like the Decision Trees and the SVMs for solving the problem.

There are several other papers which solve this problems employing algorithms like Neural Networks, Logistic Regression, Genetic Algorithms, etc. However, the paper which uses the decision trees algorithm is a nice place to start learning.

what would be a good place to host the 600 or so GB of data for cheaps?

Aws S3 would be a nice, cheap way to do that. It also integrates nicely with Redshift, in case you want to do complex analytics on the data.


Proper way of fighting negative outputs of a regression algorithms where output must be positive all the way

A standard trick is to estimate the logarithm of the desired quantity, then take its exponential, which is always positive. The drawback is that the error is optimized for the log, which treats differences in order of magnitude as equal. Another option is to do your regression as usual then project onto the feasible set (use the positive part of the output; $max(0, \cdot)$)


Dropout dividing by compensation term = overshoots the result?

Your example is cherry-picked: You mask out small numbers and keep a large one.
But dropout is applied randomly. Each of the following six masks, and of the corresponding values for the vector length, is equally likely to appear:
$$
\begin{align*}
&(1, 1, 0, 0): &\sqrt{0.1^2 + 0.1^2} &\approx 0.1414,\\
&(1, 0, 1, 0): &\sqrt{0.1^2+ 0.2^2} &\approx 0.2236,\\
&(1, 0, 0, 1): &\sqrt{0.1^2+ 5^2} &\approx 5.0010,\\
&(0, 1, 1, 0): &\sqrt{0.1^2 + 0.2^2} &\approx 0.2236,\\
&(0, 1, 0, 1): &\sqrt{0.1^2 + 5^2} &\approx 5.0010,\\
&(0, 0, 1, 1): &\sqrt{0.2^2 + 5^2} &\approx 5.0040.\\
\end{align*}
$$
The average vector length is
$$
\frac16 (0.1414+0.2236+5.0010+0.2236+5.0010+5.0040) = 2.5991,
$$
which is roughly half of the original vector length $5.006$. So it makes sense to divide it by the dropout rate of $50\%$.


In that book the author encourages the reader to write a neural network library from scratch, to gain an understanding of the principles of deep learning. Tensorflow is a mature deep learning library that can be used in production, and can make use of high performance computing resources like GPUs and TPUs. The code in the book is better for learning and Tensorflow is better for training models for real world applications.


Neural Net Backprop Weight updating Pseudo code help please

It looks like you're adding the delta to the weights instead of subtracting it. Gradient descent is given by the following calculation run for some iterations:

$$
x^{current} = x^{previous} - \alpha \frac{dy}{dx}
$$

where $\alpha$ is the learning rate. We subtract the value because derivatives go in the direction of steepest ascent. So the following lines:

HIDDEN_OUTPUT_WEIGHTS[ i ] += hi_weight_delta;
INPUT_HIDDEN_WEIGHTS[ i ][ j ] += ij_weight_delta;

should be

HIDDEN_OUTPUT_WEIGHTS[ i ] -= hi_weight_delta;
INPUT_HIDDEN_WEIGHTS[ i ][ j ] -= ij_weight_delta;

Additionally, I'm not sure what the architecture of your network is, but it doesn't look like you have bias units- consider adding them so that your neurons can have non-zero centered activations. Not having them may impact your network's ability to learn.


What is the meaning of the parameter "metrics" in the method model.compile in Keras?

The argument metrics is meant to define your criterion for training evaluation. Let me make an example: if you are training a classifier, you want to evaluate your model based on how accurate (in percentage) it is. Therefore, your metric is accuracy (experessed as a float in the [0, 1] range). The higher the accuracy, the lower the loss, the better the model.

metrics must not be confused with loss. The loss function is something you need in order to "punish" your model when it makes mistakes. The loss function is at the basics of backpropagation and of weight update, it's the loss object what Neural Networks use to learn. Metrics instead is something that humans watch to understand how good a model is and communicate it.

The definition of some metrics is optional, you can evaluate a model based on the loss value only if you want. Sometimes you don't need to specify it. For example in regression tasks, i.e. when you have to predict continuous outputs, you speecify a loss (usually MSE or RMSE) and also evaluate your model based on that.


The point in using batch training is that you can't take a step using all data due to the size of the training data which is really big. Consequently, you may want to use batch optimisation techniques which take steps which are near to the best step. The reason is that in each batch, the distribution of data points is similar to the whole training data or at least it is not very different. If you have a small size for your batch, your steps may oscillate and may not be very perfect but for bigger batches they are better.

Batch techniques also facilitate the number of steps that you can take. They also can be helpful for not being stuck in local minimum points.


Model Selection using Bias Variance Trade Off

Yes, assuming you haven't overfitted on the test set (which may happen after extensive hyperparameter optimization), you can assume that your model has a MAPE of 15%.
However, if you limit the overfitting, the test performance would probably go down!


You have a few problems here.  The first is cleaning your data.  That's a whole separate issue form anonymization and belongs in another question if you're still having problems with it.

The second is your anonymization.  After you have some sort of identifier you're satisfied with (sounds like you're using people's real names), try hashing their names to generate a new id.  This id is useful because you'll always be able to take the original name and figure out what id it is but won't be able to derive the real names from just the hashed id (providing your hashing algorithm is good).

Further reading:

https://security.stackexchange.com/a/61878
https://stackoverflow.com/a/21563966/4435034


How to find the right regression model for my project

This answer is based on additional information regarding the data provided in a comment to the question:

  [...] So it's about delivery The
  first and second are Id corresponding to cities, the 3rd is a date,
  and the last is an id corresponding to the truck that is changing (i
  sorted it that's why its the same in the first 3 columns). And the
  target Y is delay.

If you just add the time column in a unformated way to the regression, the method(s) will not be able to digest the information in a good way. Think about your problem. Delivery may be delayed because of trafic conditions (time, or the day of the week). I suggest that you encode time e.g. in hour of day, day of week, week of year (or so) and add all these (possibly one-hot encoded) variables/features to your model (together with the remaining features). You can use regularization (e.g. lasso) to "shrink" unnecessary features. This should give you an okay fit.

Essentially you need to really make sure you get as much information as possible out of the "date" column. In Python you can encode dates by predefined functions, e.g.

import datetime
Todays date datetime.datetime.today()
Day of week datetime.datetime.today().weekday()
Week of year datetime.datetime.today().isocalendar()[1]

Find an example of how to apply lasso here.


How to gather training data for simple voice commands?

What would librosa generate? If the sound clips it gives you aren't also words then I would worry about the effectiveness of the training. It'd be better to record samples of yourself saying the positive labelled words and saying a bunch of other words to use as negative labels. I recognize this is likely to be an irritating process; one approach I have read of in the past to maximise the use of small amounts of training data is to use transformations and distortions on training data to increase the effective number of samples. In this case, the training data were pictures of animals. Effectively what the researchers did was tweak the images slightly; they rotated the images various degrees, squeezed them (by lowering the height or width and not retaining aspect ratio) or stretched them and so on. You might be able to do the same;  record a smallish number of samples and then use distortions to raise the effective number of samples to a useful number.


Metrics show badly performing model for multiclass

I think you need to consider these metrics also i.e., Precision and Recall when you decide on some model.

As you can in the above outcome that there are Precision and Recall values, you din't explain anything about the problem's business perspective.

For example: Consider a data set with 100,000 observations. A data set consist of candidates who applied for Internship in Harvard. Apparently, harvard is well-known for its extremely low acceptance rate. The dependent variable represents if a candidate has been shortlisted (1) or not shortlisted (0). It was found ~ 98% did not get shortlisted and only ~ 2% got lucky.
In this scenario you need to understand that precision and recall plays a vital role at the time of selecting model.

In the same way if your business problem is inline with the above scenario then you need to select a model with high Precision and Recall.

In the above screenshot which you have shared, it think you have good performance scores too(Assuming that your model is not over fitting). So, you can select which ever model suits your business problem.

Hope my answer is helpful!


This may be an unpopular opinion to some, but in my experience Bayesian statistics is not particularly useful in data science in industry, for a couple of reasons:

A Bayesian approach is very useful when our questions are about statistical inference. However, in data science, more often than not, we are dealing with prediction. There may be some situations where a Bayesian approach works better than a frequentist approach, but I can't think of any off hand, apart from where a conjugate prior is available, in which case we are probably dealing with a very simple model.

Bayesian statistics usually requires a sampling, such as Markov Chain or Hamiltonian Monte Carlo, and this can be extremely computationally intensive, even for relatively small datasets. In industry we are often dealing with "big data" and a Bayesian model that requires MCMC or HMC just isn't practical.

Edit: To address a comment to this answer:

I have a question about this, how statistical inference is different from prediction. As for my understanding, prediction is usually about getting P(Y=y | X), where X is our data, which is similar with statistical inference

Prediction and inference are completely different. With prediction, all we really care about is the accuracy of the predictions, and that is a relatively easy thing to determine - because the test/validation procedure is minimising some measure of prediction error. On the other hand, with inference, we care about the coefficient estimates and often their standard errors. Often a researcher forms a hypothesis based on a theory of causes - eg, does drinking coffee cause cancer, and they will use a model to determine whether or not the data supports their theory, and to what extent.  This is much, much more difficult than prediction. For one thing, in causal inference the basic requrement is that we do not want the estimates for $\mathbf{X}$ to be biased - whereas in prediction we don't care if they are biased, provided that we get "good" predictions for $\mathbf{y}$, and there are many, many sources of bias in a regression model - bias can and does arise from confounding, mediation, differential selection and colliders. Usually, the crux of the issue is to decide which variables to include in the model in order to eliminate or reduce these biases. With prediction we can use automated variable selection procedure in order to choose which variables to include (feature selection). With inference this is pretty much impossible because automated procedures cannot generally handle the above mentioned biases.
There is a detailed discussion of these issues in my answer to this question:
How Do DAGs Help To Reduce Bias In Causal Inference ?


Time Decay Formula for Conversion Attribution

Here's one solution:

using the following data:

USER-LEVEL:

fss first session start (timestamp) - time of first pageview of
first session for that user
css converting session start (timestamp) - time of the first pageview of session where user converted
ns number of sessions (int) - total # of sessions
before conversion for user

SESSION-LEVEL:

ss session start (timestamp) - timestamp of first pageview in session

FORMULA: calculated for each session

FOR FIRST SESSION: Attribution Credit is just linear: (or really whatever you want to make it)
ALL OTHER SESSIONS:

First calculate the SUM of all session's "time before conversion":
Then Calculate the credit for each session:

it isn't perfect because it distributes credit based on the relative position of a session in the User's lifetime.  Ideally you have a preset halflife (like 7 days) so that comparison is more consistent.


Poor Precision-Recall curve for binary classifier trained on balanced data, with imbalanced test data

The precision-recall curve and the ROC curve respond very differently to class imbalance. The precision-recall curve is very sensitive to imbalance while the ROC curve can mask the effect of imbalance.

For the TPR (=recall) and FPR you divide by the true number of samples in each class, while for precision you divide by the number of positive predictions.

If you have few positive samples the ratio between false and true positives becomes more sensitive (compared to false positives [few] and all true negatives [many]).

Depending on your use case, this might be a problem (or not) and should be part of your evaluation.


How to get rid of some parts in a string variable?

One possibility is to use gsub function to find the first ' - ' and remove everything before and also remove the dash with the space before and after that.

> tmp
[1] "965 - Vehicle parking area"
> gsub("^.*? - ","", tmp)
[1] "Vehicle parking area"

If there are two spaces:

> tmp <- "965  -  Vehicle parking area"
> sub("^.*?  -  ","", tmp)
[1] "Vehicle parking area"


Typing error handling n-gram character index and vector space model

Trigram models can be more powerful for document retrieval than unigram models, but if you want to handle spelling errors, they will not be of much help.  You need some form of fuzzy matching for that.

For example the string, "I like dosg too" would fool a unigram model because "dosg" is likely "dogs" misspelled, and it will encode it as "dosg" : 1.  But you have the same problem in a trigram model.  It will encode "I like dosg" : 1, "like dosg too" : 1.  Which is not really better, as it will still not match any trigrams with the word "dogs" in it.


Perform several different torchvision.transforms on ImageFolder object

I've just found out about the class torch.utils.data.ConcatDataset([datasets]) . Its easy to use and documentation is here : http://pytorch.org/docs/master/data.html


This is an example of Python's builtin __call__ method, as described here. In short: BCEWithLogitsLoss is a class. The first set of parentheses (empty, in your case) provides any needed arguments to the class initializer. Then the second set of parentheses are passed to the call method. So, this is convenient syntax that allows you to instantiate the class and evaluate one of its methods in one line.
Confirming this in the source code is actually a bit difficult. You can see the source for BCEWithLogitsLoss here, which confirms it is a class. But its only method is forward; where is __call__? For that, we notice that the BCEWithLogitsLoss class inherits from the _WeightedLoss class, which inherits from Loss, which inherits from Module. We can then see that this base class implements the mapping between __call__ and forward.


This is a very general question, however, there are many different solutions as explained below. For your case, probably, item 2 is not the case because you can not gather a large number of data points. I would recommend using solutions 1, 3, 5, and 6 (I see you used this method but try to combine it with other solutions such as cross-validation, regularization, and feature selection).

Cross-validation: Use the initial training dataset to produce multiple mini train-test splits. Use these splits to tune your model.

For example, in k-fold cross-validation, partition the data into k subsets. Then, train the model on k-1 folds iteratively while using the remaining fold as the test set. In this way, you can use the cross-validation to tune hyperparameters with only the original training set.

Train with more data: Try to use more data points if possible.

Perform feature selection: There are many algorithms that you can use to perform feature selection and prevent from overfitting

Early stopping: When you’re training a learning algorithm iteratively, you can measure how well each iteration of the model performs.

Up until a certain number of iterations, new iterations improve the model. After that point, however, the model’s ability to generalize can weaken as it begins to overfit the training data.

Use regularization. As also will be discussed in item 7, the higher the complexity of the model the higher the chance of overfitting. For example, in the case of logistic regression, when the weights are large, the model gets complicated and it probably won't work on the unseen test dataset. Regularization helps to decrease the weights and so the complexity of the model.

Use ensembling methods such as Random Forest and Gradient Boosting. One of the main issues with decision trees is that they are prone to overfitting; i.e., high variance, it means that they work very well on training data but not on the unseen test dataset. One solution to prevent overfitting in the decision tree is to use ensembling methods such as Random Forest, which uses the majority votes for a large number of decision trees trained on different random subsets of the data.

Simplifying the model: very complex models are prone to overfitting. Decrease the complexity of the model to avoid overfitting. For example, in deep neural networks, the chance of overfitting is very high when the data is not large. Therefore, decreasing the complexity of the neural networks (e.g., reducing the number of hidden layers) could help to prevent overfitting.

Drop out method. In deep neural networks, randomly dropping some of the connections between layers by multiplying noise sampled from a Bernoulli distribution could help to prevent overfitting.


SGDClassifier: Online Learning/partial_fit with a previously unknown label

It sounds like you don't want to start retraining the model every time a new label category appears. The easiest way to retain maximal information of past data would be train one classifier per category.

This way you can continue to train each classifier incrementally ("online") with something like SGDClassifier without having to retrain them.
Whenever a new category appears you add a new binary classifier for just that category. You then select the class with the highest probability/score among the set of classifiers.

This is also not much different from what you are doing today, because scikit's SDGClassifier already handles the multiclass-scenario by fitting multiple "One vs All" classifiers under the hood.

If a lot of new categories keep coming up, of course, this approach might become a little tricky to manage.


Heat maps in R with more than 2 categorical variables

Here is one way of visualizing the data you have presented. However, I have used some liberties and assumptions to create this plot.

First, create a year_quarter variable that simply concatenates the year and quarter to present the time on the X axis. The following piece of code can do this in R

year_quarter = paste(dat$year, dat$Quarter, sep="-")

Now, the dataset you have will look like:

> dat

Variable  value year Quarter Location year_quarter
1        A 48.235 2011      Q1    North      2011-Q1
2        B 65.444 2011      Q2    North      2011-Q2
3        C 77.453 2011      Q3    North      2011-Q3
4        D 44.678 2011      Q4    North      2011-Q4
5        A 88.542 2012      Q1    South      2012-Q1
6        B 66.566 2012      Q2    South      2012-Q2
7        C 55.443 2012      Q3    South      2012-Q3
8        D 78.990 2012      Q4    South      2012-Q4

Finally, using ggplto2, you can create the plot such that the colour represents the value, the shape represents the Location and the size represents the Variable.

This simple one liner can help you produce such a plot:

p = ggplot(dat, aes(x = year_quarter, y = value, colour = value)) + geom_point(aes(shape = Location,size = Variable))

This is how the output plot looks like:

Note that you can also add geom_line with interaction if you would like the lines to be connected based on Location and Variable.


If you want a good and solid start for deep learning, I would suugest to start with the appropriately named book "Deep Learning" by Ian Goodfellow et al.  After that you'll have a good base that you can expend by the many different tutorials, articles and courses available online.

However, I would also add that before doing that, you should take some basic "machine learning" class (should be available at your University). Many people these days go straight to deep learning and implementing Neural networks because it is relatively easy, but than they lack the understanding to improve it or use it to its fullest potential.


XOR problem with neural network, cost function

The cost function can be found in the delta rule, meaning the way you calculate your deltas. This delta is nothing more than the derivative of your error function after the weights: $\frac{\partial E}{\partial w_{ij}}$. So, if you are just interested in where the cost is encoded, this is the answer you are looking for.

If you, on the other hand want to know why this formula works, I can suggest you to read the derivation on wikipedia. The maths behind it is quite uncomplicated, you only compute the derivative in each layer and propagate this derivative through the layers. This is, by the way, how backpropagation got its name.


confusing regarding to kmeans clulstering for data correlation

You will not get what you seek this way, but you are on the right path. Use the correlation between two instruments as a measure of similarity, and then perform spectral clustering with this measure as the kernel.

Basically, you will start with your correlation matrix $R$, and build the corresponding Laplacian matrix $L$. The eigen vectors corresponding to the smallest eigen values of $L$ will give you a projection space in which you can perform k-means clustering.

This technique is efficient if you have a good similarity measure, but only works for reasonably sized datasets (because you need the eigen decomposition of the $n \times n$ matrix).


What are the differences between Reinforcement Learning (RL) and Supervised Learning?

What are difference between Reinforcement Learning (RL) and Supervised Learning?

The main difference is to do with how "correct" or optimal results are learned:

In Supervised Learning, the learning model is presented with an input and desired output. It learns by example.
In Reinforcement Learning, the learning agent is presented with an environment and must guess correct output. Whilst it receives feedback on how good its guess was, it is never told the correct output (and in addition the feedback may be delayed). It learns by exploration, or trial and error.

  Could we say RL has more difficulty in [finding] a stable solution?

No, because the types of problems it solves are usually different, you cannot compare like with like.

However:

You could use RL as a framework to produce a predictive model for a dataset (using RL to solve a Supervised Learning problem). This would be inefficient, but it would work and there is no reason to expect it to be unstable.
In general, RL problems present additional challenges to supervised learning problems with the same degree of complexity in the relationship between input and output. They are "harder" in that sense, as more details need to be managed, there are more hyperparameters to tune.
RL can become unstable in ways that don't apply in supervised learning. For instance Q learning with neural network approximation tends to diverge, and needs special care (often experience replay is enough)

  Could we say that [getting stuck] in [a] local minimum is seen more in supervised learning?

No. Internally, a RL agent will often use one kind of supervised learning algorithm to predict value functions. RL does not include any special features that can avoid or work around local minima. In addition, a RL agent can get stuck in ways that don't apply to supervised learning, for instance if an agent never discovers a high reward, it will create a policy that ignores entirely the possibility of getting to that reward.

  Is this figure is correct saying that Supervised Learning is part of RL?

No. The figure is at best an over-simplified view of one of the ways you could describe relationships between the Supervised Learning, Contextual Bandits and Reinforcement Learning.

The figure is broadly correct in that you could use a Contextual Bandit solver as a framework to solve a Supervised Learning problem, and a RL solver as a framework to solve the other two. However, this is not necessary, and does not capture the relationships or differences between the three types of algorithm.

Otherwise the figure contains so much over-simplification and things that I would consider as errors (although this might just be missing context that explains the diagram), that I recommend you simply forget about trying to interpret it.


Log-loss - but only if too far away from target?

Have you looked into focal loss?

The idea seems to be similar to what you are describing - If predictions (~0.8) is close to GT Label of 1, it does not add to the loss value.


Cross-validated average: metrics mean or ensembling probabilities?

A standard way to provide the performance of each model would be:

providing, for each split, the value of the chosen metric (accuracy, roc_auc, etc) on the train and test sets (on your case, your one-out sample), something like this (in this case with 2 models):

as a final model performance (for each one of the 5 models), a mean metric value together with its standard deviation for the test sets is a way to inform about the model quality and its robustness, something like (preferably for the test set):

You have more detail on how to automatically get this done via scikit-learn, and in this answer and this one.
By the way, consider using another strategy as stratified k-fold, in case you have a lot of samples, as leave-one-out would be very costly.


Instead of scraping,  you might try to get the data directly here.  It looks like they have data available via ftp for movies, actors, etc.


How to do multivariable regression in Orange3?

The regression models are multivariable. You can go to the training videos of Orange to learn how to do regression.


Normalize / Standardize in a Random Forest?

Thanks for the clarification by commenting. Tree-based models do not care about the absolute value that a feature takes. They only care about the order of the values.
Hence, normalization is used mainly in linear models/knn/neural networks because they're affected by absolute values taken by features.

You don't need to normalize/standardize.

Check this post.


How to reconsider MSE metric when doing target engineering through a logarithm?

You have a $Y_{true}$
Then you calculated $Y^{log}_{true}$
So, your predictions are $Y^{log}_{pred}$
Then you can calculate the $MSE^{log}$ but it can't be compared with 100 MSE.
So, you can do a - $base^{MSE^{log}}$
to make it comparabale to 100 MSE.
A better intuitive approach will be to convert back your $Y^{log}_{pred}$
 i.e. $base^{Y^{log}_{pred}}$
Then calculate the MSE
The inverse of log is just about using power to the base
import numpy as np
input = np.array([100, 1000])
log_out = np.log10(input)

input_re = 10**(log_out)
input_re


Is there a rule of thumb for the initial value of loss function in a CNN?

The initial loss is purely dependent on your weight initialization and your data normalization. Random weights = random initial loss. If you're interested in the variance of the loss, this will roughly relate to the number of layers you have and number of weights in each layer. For a good example, check out this paper:

Understanding the difficulty of training deep feedforward neural networks -- Glorot, Bengio


It seems weird.

Your validation has a higher score than your training. This literally means that your model performs better in unseen data than what it sees.

Typical underfitting is that you achieve the same in train than in test.

In my opinion, since you are not providing much information, you are not splitting right the data. It might be for a lot of reasons:

The test is too small or too easy to predict
There is a temporal dependency and you are not using it (data leakage).
There are groups in your dataset and you are splitting by groups...

And a thousand more. From the visualizations that  you are adding my guess your train test split is not performed correctly


Can I use Linear Regression to model a nonlinear function?

You are asking two different questions:

What is linear regression?

Linear regression means that, given a response variable $y$ and a set of predictors $x_i$, you are assuming (whether or not this is true is another matter) to model your response variable as
$$
y^{(j)}=\sum_{i=1}^N x_i^{(j)}\beta^i + \epsilon^{(j)}
$$
for each observation $y^{(j)}$, where $\epsilon^{(j)}$ is an error term with vanishing expectation value. The purpose of the algorithm is to find the set of $\beta^i$ to minimise the errors between the above formula and the actual values of the response.

May I use linear regressio to model non-linear functions?

You may use the linear regression to model anything you want, this does not necessarily mean that the results will be a good fit. The mere decision to use a model makes no assumptions on whether the underlying equation is in fact reflected by the model you choose. In case of linear regression you are essentially approximating an $N$-dimensional manifold (where all true points belong) with their projections onto a plane. Whether or not this is a good idea it depends on the data.

  I want to know if that is really the case, and is it always possible to do this "change" in the functional form?

By using this or that other model you are not changing the functional form of the underlying variables. You are just dictating that the original relation (that you do not know) can be approximated by the model you choose.

  Is it still correct to measure the performance of the model using R_square metric?

The $R^2$ is defined as the ratio between the residual sum of squares of your model over the residual sum of squares of the average. Basically it tells how much of the variance of the data is explained by your model compared to just taking a straight line (in correspondence of the average) passing through all your data points.


Keras y-labels range between 0 and 1 instead of binary?

Assuming y_train is already in range [0, 1], it should be enough to specify the correct loss function while compiling a model (model.compile).

Loss functions can be found here.

Chose one which is not for categorical variables (since you have a range [0, 1]).


First of all your dataset is very small to train a deep learning model you would need at least 10k images to train the model and produce good results. Secondly, if you still think that your cost function gets stuck in local minima then you might consider using Nesterov momentum.


Keras: ValueError: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2

Your post is a duplicate : Here someone asked the same question.
Have a look at the answers, you can also try this link that gives a detailled explanation about how to reshape inputs for LSTM layers.


How to measure Covid impact by analysing credit card transaction of customer

Do you have some sort of labeled data?

Otherwise I'd hypothesize that this task is close to impossible since any sort of unsupervised algorithm using anomaly detection would likely have an incredibly low precision since almost all data would be classified as an anomaly during this time.

However, if you were to obtain some labeled data (labeling customers as 'in financial distress' and 'not in financial distress') then you could use some supervised learning algorithm to try to predict whether the unlabeled customers fall in category A or category B.

Unless you acquire some sort of domain expertise or labeled data, this would be a very hard task.


Jupyter, Python: the kernel appears to have died while training a model on a big amount of data

I have had the same problem while training in huge data sets in Jupyter Notebooks. The only solution I found was to create a scrip .py with my training process (including model persistence) and running it from my terminal (python3 myscript.py)


Searching interactions with RandomForest and/or GBM

Several times it does happen that interactions among variables improve the bias of the model. This is especially true when the effect of one independent variable on the target depends on the values of other independent variables.
I don't think there's anything mathematically incorrect in doing this.

E.g: let's say you are trying to predict revenue as a function of advertising. In this example it's reasonable to assume that the effect on revenue of one extra unit of advertising on Television  would depend on the existing level of advertising on Facebook (say). The true data generating function (if you had access to it) might be something like: $$Revenue=Ad_{TV}^{\beta_{1}}Ad_{Print}^{\beta_{1}}...$$ where $Ad_{TV}$ is the # advertisements shown on TV etc.
If you use provide the model the opportunity to handle such interaction terms you will be closer to modeling the true data generating function.

However adding more feature increases the complexity (capacity) of the model and you might have to use regularization wisely to prevent over-fitting


logistic regression coefs to excel formulation

By running a logistic regression model, the objective is to get the chance of a binary outcome based on the different predictor variables. Since the result is a chance, what is modeled is actually:

$$
log \left(\frac{P_{+}}{1 - P_{+}}\right) = \beta_{0} + \beta_{1}x_{1} + ... + \beta_{n}x_{n}
$$

So the coefficients you get from R are the $\beta$ values.
The function you have so far in Excel computes correctly the right hand side (RHS) of the above equation, but to get the probability $P_{+}$, you need to solve the equation for it. Let's call the RHS, $\mathbf{Bx}$

$$
log \left(\frac{P_{+}}{1 - P_{+}}\right) = \mathbf{Bx} \\
\frac{P_{+}}{1 - P_{+}} = e^{\mathbf{Bx}} \\
\frac{1}{e^{\mathbf{Bx}}} = \frac{1}{P_{+}} - 1 \\
\frac{e^{\mathbf{Bx}} + 1}{e^{\mathbf{Bx}}} = \frac{1}{P_{+}} \\
P_{+} = \frac{1}{1 + e^{-\mathbf{Bx}}}
$$

To your results, apply this transformation and you should get the same results as the predict function in R.


Scikit learn train test split without mixing participants in trails

You can use one of scikit-learn's options for grouped data. In particular, GroupKFold should do the trick: something like

from sklearn.model_selection import GroupKFold
group_kfold = GroupKFold(n_splits=2)
group_kfold.get_n_splits(X, y, groups)

where groups is an array of group indices.


Magnifying or reducing the size of input groups into a neural network

The main problem I see here is that OHE is almost never a good idea with that many categories. With neural networks you will usually get better performance by using embeddings. So instead of

X1 -{OHE}-> 10,000 -> {..} -> 1,000 you could go straight to X1 -{embedding}-> 50, where the embedding dimension should probably a lot lower than 1,000.

Either way, more important is that one calculates feature importance in a coherent framework. The easiest one is probably permutation importance (another could be shap values), which basically comes down to (repeatedly) shuffling the input features X1 and X2 and record the impact on Y. This works also for OHE variables if you make sure to shuffle X1 before one-hot encoding (and not shuffling the 10,000 encoded vectors independently). In such a framework X2 doesn't need to be "magnified" for a fair comparison with X1.


That's a very good question. I am facing a similar issue with different data, but my research led me to this which should be pretty much what you are looking for.


How to fill NaN based on the previous row that contains a specific value

You can groupby by 'stock', forward fill missing values with ffill and use its result in fillna. For example:
date       stock   price
22/12/20   MSFT    87
22/12/20   AAPL    99
22/12/20   FCA     81
23/12/20   MSFT    NaN
23/12/20   AAPL    100
23/12/20   FCA     80
24/12/20   MSFT    91
24/12/20   AAPL    NaN
24/12/20   FCA     NaN

df.fillna(df.groupby('stock').ffill())

Result:
       date stock  price
0  22/12/20  MSFT   87.0
1  22/12/20  AAPL   99.0
2  22/12/20   FCA   81.0
3  23/12/20  MSFT   87.0
4  23/12/20  AAPL  100.0
5  23/12/20   FCA   80.0
6  24/12/20  MSFT   91.0
7  24/12/20  AAPL  100.0
8  24/12/20   FCA   80.0


Making predictions from keras with SciKit

This seems about right.

You can use SciKit learn quite easily, as the predictions and test results you have should all be in NumPy arrays anyway. Take a look at the regression metrics. The metrics you have named are shown in the documentation to take lists, but here is an example showing they work just as well with NumPy arrays:

In [1]: from sklearn.metrics import r2_score

In [2]: y_true = [3, -0.5, 2, 7]

In [3]: y_pred = [2.5, 0.0, 2, 8]

In [4]: r2_score(y_true, y_pred)
Out[4]: 0.94860813704496794

In [6]: import numpy as np

In [13]: y_true = np.array([3, -0.5, 2, 7])

In [14]: y_pred = np.array([2.5, 0.0, 2, 8])

In [15]: r2_score(y_true, y_pred)
Out[15]: 0.94860813704496794            # identical result

One last comment:

17 examples doesn't sound like a lot. Look at model.summary() after compilation to see how many parameters your model has (spoiler - it's 1360). I would expect that you model (with your number of epochs and batch size etc.) will overfit, just memorising the dataset, and probably score 100%.

While this is a good sanity check to make sure your model can indeed learn, it might be a good idea to split a larger dataset (if available) into train, validation and test datasets. The simply use train and val as you have above, but in the prediction line, us the test set, which the model has never seen before. Unless your data is extremely homogenous, I wouldn't expect a accuracy metric near 100%.


Depth of the first pooling layer outcome in tensorflow documentation

I kept spinning my head around this question because I seem to come along the same conclusion as you do. However, it appears to be a mistake in the documentation.

https://stackoverflow.com/questions/43453712/what-is-output-tensor-of-max-pooling-2d-layer-in-tensorflow


How to standardize my data (Univariate Time Series Forecasting using Keras LSTM)?

Welcome kakarotto! The first thing, standarize with respect to the training set only, then use the statistics to standarize the other sets. As a rule of thumb, we should keep in mind that the test set does not exits while training... (although we have access to it) When training, if the network does not see the test set we (the humans) do either…

If the test set was well selected, and so the whole dataset, the training set and the test set should come from the same probability density function and, if enough data is provided, the statistics you get from the training set would be close to the statistics of the pdf (assuming unbiased estimators for these statistics, that should be the case for the mean and standard deviation)

Based on my experience, only standarize the input data. With the output data, I usually scale it (min-max scaler) if I know the boundaries (it they even exist…). Another usefull trick that usually worked for me its to not feed the net with the input data rather with its relative differences, for example:

$X = (0, X_1 - X_0, X_2 - X_0,\dots,X_m-X_0)$


Upper bound on size of sample set for decision trees

As far as I understand this theorem (not much), it doesn't really say anything about the relation between $n$ (the size of the training set) and the error rate. The main relation that it claims is between the error on the training set and the true error, assuming a minimum size on the size of the training set and that the training set is drawn from the true distribution.

I don't think it's possible to give an absolute upper bound on the number of samples needed in the training set for at least the following reasons:

The success of the learning depends on which instances are in the training set, i.e. one needs a sample representative enough so that the model correctly captures the distribution. For instance, in theory it could happen by chance that all the $n$ instances have the same value for feature $x$, so the model would never know that feature $x$ can have another value. In general there's no way to be sure that the training set contains all the diversity required for a successful model in the instances.
There's also the question of what the learning algorithm does exactly: if one wants to prove that a particular DT learning algorithm (say C4.5) can learn a particular model with less than $n$ instances (assuming these $n$ instances are sufficient), the proof must involve the particular properties of the algorithm itself. Otherwise it's obvious that the claim is false: if one chooses an algorithm which randomly builds a tree, it's clear that this cannot lead to the right model.

I think it's more feasible and more common to provide an experimental upper bound: run many cross-validation experiments, varying the training set size and possibly other parameters. Then based on this one can estimate the probability of reaching the correct model given the size and other parameters.


The distribution of dataset train and test are the differents, how to fix this?

how does it compare to longer timespans? Can you still observe noticeable drift?
to some extent this is usual behaviour
here is a guide for KL divergence, It's just a tool you can use to compare probability distributions.
exploration is equally important as hard stats. For me those plots look alright. There's no defined point, telling you when you can or cannot do things. That's whole point of ML, you don't have a priori knowledge. Best run some models and check validation metrics.


Likelihood function is the product of probability distribution function, assuming each observation is independent. However, we usually work on a logarithmic scale, because the PDF terms are now additive. If you don't understand what I've said, just remember the higher the value it is, the more likely your model fits the model. Google for maximum likelihood estimation if you're interested.

Obviously, your input data is bad. You should give your model a proper data set. While I don't have your data set, we can take a look at the likelihood function for linear regression:

You will get infinity if the likelihood function is zero or undefined (that's because log(0) is invalid). Look at the equation, most likely your sample standard deviation is zero. If it's zero, the last term will be undefined. Have you given a data set that you copied and pasted the same data over rows?

Boosted trees should also be undefined if your sample deviation is zero. However, decision tree is estimated based on impurity and won't crash here.

Summary: please check and double check your data.

EDIT I think you just had a bug. Linear regression will always give you something here. Have you fitted the models in R with the same dataset? –


How to put multiple features into RNN input vector

A common way to input several features to an LSTM (or any RNN) is, as you did, to concatenate them in a vector. I suspect your NaN are related to a different issue in the code, and I recommend you to debug it and see when and why it happens.

A different way of combining several features is using embeddings for each feature and combine them via concatenation. For example, for the note pitch, you have an embedding for each note, for the note duration, the same, and so on. The input to the RNN would be a concatenation of that.

A third option would be to have an embedding to all possible combinations of the embeddings. That would be an embedding for each triplet of note pitch, note duration and offset. This representation can be also powerful.

My recommendation is that you start with the easiest to implement, and if the result is not satisfactory in terms of model performance, think about the next one.


Defining Input Shape for Time Series using LSTM in Keras

An LSTM will expect the data to be of the format (samples, time steps, features).  In a univariate case where we will focus on just the time series with the sales data, where you say five lags are required as input:

sales data = [100, 150, 345, 5, 800, 655, 100, 200, 300, 400]

You could format your data like this using a sliding window of step size one:

[100, 150, 345, 5, 800]
[150, 345, 5, 800, 655]
[345, 5, 800, 655, 100]
[5, 800, 655, 100, 200]
[800, 655, 100, 200, 300]
[655, 100, 200, 300, 400]

this would give you six samples each of five time steps and one feature (the feature being the sales data).

So the shape you would take the 2d array above and reshape it to a 3D array of (samples, time steps, features) which you could also think about like (rows, columns, features), giving (6, 5, 1).

Let's say now you have another feature holidays like in your example, in this case you have two features, the sales data and whether or not it's a holiday or not.  So you would need dimensions of (6, 5, 2), six samples each containing five time steps of data each with two series each (or features).


Orange 3 - Feature selection / importance

You can connect Random Forest widget into Rank widget, which I guess is meant for subset feature selection. See Orange screenshot #19.


Logistic regression vs Random Forest on imbalanced data set

To have a probability of 1 in a RF, it means that your algorithm can construct a leaf containing only positive sample. Since it doesn't, this means that your features are not explaining the variance of the output or that your algorithm is under-fitted.
I suggest that you try optimize the hyper-parameters of your RF by using cross-validation and use some oversampling to reduce the bias in your dataset.


Obtaining frequently occurring phrases using Word2Vec

You may use gensim phrase vectorizer module available in Python.
You need to give threshold value which is some sort of pmi of words. The higher this value less are the number of phrases the default is 10. You can play around with this value to get results for your data.

phrase_threshold = 1
bigram = Phrases(sentences,threshold=phrase_threshold)

This is based on this based on the skipgram paper by Tomas Mikolov.


How to predict an outcome within a specific time window?

Beeing X in the future and beeing X in specific time in the future is just a subset of the first one.

So what one really needs to do is just determine the probabilities (or parameters that give us these probabilities)

P(X|t>30)

Where you can model t, also as your feature. So just fit a model on this data, where you have mutliclassification of:

dead within 30 days
dead after 30 days
alive after 30 days


Very briefly, homomorphic encryption as used by the linked paper in the question works as follows (excuse ad-hoc notation). Define an encryption method $E(x,k)$ that takes some data $x$ and secret key $k$ that has the following properties:

There is a decryption method $D( E(x,k), k ) = x$ (optionally the two $k$ here can be different)
You can define operations that work on the encrypted data in a self-consistent manner, e.g. $\text{Add}_E(E(x,k),E(y,k)) = E(x+y,k)$ - note that the result is still encrypted, and you need to know $k$ in order to use method $D()$ and find out what $x+y$ is.
You may also need to support operations like $\text{Add}_E(E(x,k),y) = E(x+y,k)$ e.g. where one of the values is normal numeric variable - this might be useful for initialising a model - or using an existing trained model - but the catch is that when training you cannot keep these values unencrypted, once they use updated due to training results, they will also be encrypted.
Note that no operation can output a decrypted value without using the key, otherwise the whole scheme is insecure, it implies some backdoor to get decrypted data.
For ML use, you need an encryption scheme E that supports multiple basic operations, e.g. $\text{Add}_E$, $\text{Subtract}_E$, $\text{Multiply}_E$, $\text{GreaterThan}_E$ etc. You need some minimal set that is enough to build a model.

You can then build variant models that instead of performing arithmetic to sum/multiply etc numeric values, work with the abstract operations on encrypted data. The models would otherwise look just like their unencrypted counterparts - e.g. you could quickly build a linear regression using only a few types of operation.

Given that you are replacing simple fast floating point add, multiply operations that are built into the processor with a more complex custom operation, this significantly affects performance compared to the unencrypted model. How much so depends on the encryption scheme, and how it is implemented. The paper shows in section 5, timings on unencrypted vs encrypted data and the difference is several orders of magnitude slower when encrypted.

The paper was published in 2012, so it is possible that some improvements have been made here. However, on a deeper read of the subject I think that this is still at the stage of research proof-of-concept. I might be wrong, and there are nice workable implementations available that could be used in something as complex as a CNN, but I have not found anything.

You also linked numer.ai from comments. Initially that looked interesting because surely they would have solved efficiency problems. But in fact their main competitions are using data obfuscation techniques, not encryption - a homomorphic encryption paper is linked from their main site, but it does not seem to be what they are using. End users are writing very familiar-looking scripts that perform logistic regression etc using regular Python (no special operators imported)

I think you should take a second look at your wider problem and analyse your threat model. What precisely are you trying to protect against? If it is about keeping your own operations separate from client data, then you might be looking into company process and auditing solutions, rather than purely technical (you may still want to add technical solutions such as disk-level encryption to protect customer's data at rest in you data centre, in case someone gets into the centre physically and just grabs a disk containing all the cat images)

For instance, look at Cloud Security Alliance which as well as having a certification scheme, has analysis showing how their recommendations map to other schemes such as UK's ISO27001. Note that this is typically significant capital investment including 1+ year project to implement, and usually undertaken by mid-sized or larger companies, when they want to work with government or large corporate's data. However, it is probably a more reliable and maybe still cheaper route than trying to research and build a technical solution involving models that process encrypted data at this time.


What is the gradient descent rule using binary cross entropy (BCE) with tanh?

Let 'a' be the output from an activation function like sigmoid or tanh.

Therefore, the derivative of sigmoid is a*(1-a) whereas the derivative for tanh is (1+a)*(1-a).

Just follow the derivation of sigmoid except replace the derivative of sigmoid with that of tanh.


Architecture for linear regression with variable input where each input is n-sized one-hot encoded

The output for every sentence is some floating-point number, representing some value that has been calculated beforehand (...) the important part is that every input is a sentence mapped to a number.

In essense this is a regression problem, i.e. given some input predict the (most likely) numeric output. However you could also see this as a classification problem if you are not interested in real numbers but in some range (=class), or a probability for each class.

  RNNs are popular in NLP so I am leaning towards them,

RNNs are well suited for sequence processing. The way you present the problem does not seem like there is much sequence processing involved - while you can get this to work with RNNs, I propose to try other less computationally involved approaches first.

  I have looked for RNNs and regression but I can only find use cases involving time series, not NLP or one-hot encoded features.

Unless there is some specific requirement to use RNNs or in extension LSTMs, I would try random forests, e.g. the RandomForestRegression regressor from scikit-learn. This should work reasonably well using the one-hot encoded sentences (as outlined in the question) as X and the pre-calculated number as Y. Should you decide to instead make this a classificastion problem, you can easily switch to the RandomForestClassification classifier from the same package.

  an unseen given sentence can be pre-processed and one-hot encoded and given as input, and that an output (floating-point number) is predicted

I suggest to use a Pipeline for both training and prediction, then serve this Pipeline to your application.


how to export the tables into a csv file pandas

Your current code overwrites the previous sheet, which is why only the last iteration is present. Setting each sheet to the same name (Sheet1) will overwrite the sheet. This name will need to be changed for each interation.

Try this:

for row_index, row in enumerate(categorical):
    for col_index, col in enumerate(numeric):
        ptable = pd.pivot_table(df, values = col, index = row, aggfunc = ['min','max','median','mean','std',lambda x: 100*x.count()/df.shape[0]])
        print(ptable)
        writer = pd.ExcelWriter('report.xlsx')
        ptable.to_excel(writer, 'Sheet1_{}_{}'.format(row_index, col_index))
        writer.save()

This should save each iteration as an individual sheet


what is meant by minimizing and maximizing in GANs?

This basically boils down to the Vanishing Gradient Problem. It is a well known issue that vanishing gradients limit the ability of deep neural nets to learn. And playing the min-max-game for GANs can lead to this issue for G as well:
When you start to train a GAN G will not produce very good results, i.e. D can easily classify these as fake or real with a probability of them being real approaching $0$. Accordingly, the expression $\log(1-D(G(z))$ converges to $0$, too, and results in a vanishing gradient for G, i.e. the generator cannot really learn when it tries to minimize this.
In contrast, $\log D(G(z))$ does not converge to $0$ when $D(G(z))$ is very small (since $\lim \limits_{x \to 0} \log x=-\infty$). Therefore, G does have a gradient to learn from.
Also see the original paper where the authors write:

Early in learning,
when G is poor, D can reject samples with high confidence because they are clearly different from
the training data. In this case, $\log(1 - D(G(z)))$ saturates. Rather than training G to minimize
$\log(1 - D(G(z)))$ we can train G to maximize $\log D(G(z))$. This objective function results in the
same fixed point of the dynamics of G and D but provides much stronger gradients early in learning.

As you can see, this is a purely practical consideration in order ease learning for G and not about theoretical equivalence of expressions.


MLP on Iris Data not working but it does fine on MNIST - Keras

By default sklearn.preprocessing.normalize normalizes samples, not features. Replace sklearn.preprocessing.normalize with sklearn.preprocessing.scale. This will center and scale (to unit variance) every feature.

Also give it more than 10 epochs. Here are learning curves (log loss) for 5000 epochs:

This should end up with an accuracy of about 96%.


Can you explain further about how you are computing in batches? The entire p-value array itself should not be large enough to cause the memory problem. So it seems like your main problem is that you are trying to hold the cross-joined DF in memory and do the computation you need. Therefore these are my thoughts:

You might be able to solve this problem the way you are currently doing it if you have access to a cluster which has more memory.
There is another way to solve it that will use less memory and could work locally but will take a long time using nested loop (see below). You can also write results intermediately instead of doing a union at the end of each loop. Afterwards you can take the initial DFS out of memory, and then load in all of the intermediary results and create the final df.
You could use the method from prior bullet point in cluster which will be a bit better.
It would be even better if you could use a cluster and parallelize it. This problem would be well suited for Spark.
If you don't have access to a cluster, you can still parallelize it on your own computer which may help a little with speed. Again, I think if you use Spark and parallelize it on your computer it would be a bit faster.

Here I will explain the first bullet which I think will at least allow you to get your final data frame without the memory issue:
I would first try making this problem into a nested loop of the columns of the data frames where you find the pvals for each gene --> all miRNAs and then subset this after computing q-values and using a cutoff. Then you will only be storing in memory the pairs which have met your q-value threshold. I would not suggest trying to create a joined DF with all of this data as that will take up too much memory. I wrote pseudo-code for a loop below. I feel like this would help with you running out of memory, however it will still be pretty slow. If you are still running out of memory, do you have a cluster available to you? Also it would be even better to parallelize this. You could write it in python but do you have Spark? If you will be continuing to work with 'big data' I would try to get access to a cluster at your institution or pay for it using AWS or something.
I am going to call your DF with the genes: geneDF and your DF with microRNA mirnaDF. Keep in mind this is python pseudo-code, and is pretty messy. I can clean it up if you think it would be useful to you. I more-so just wrote it to make what I was writing about clearer.
top_adjusted_pairs = pd.DataFrame(columns=['gene', 'miRNA', 'qval'])
for i in range(len(geneDF)):
    # initialize array for the distances between all microRNAs with just gene X
    miRNA_geneX_pvals = []*len(microRNAsDF)
    geneCol = geneDF[i]
    for j in range(len(mirnaDF)):
         mirnaCol = mirnaDF[j]
         # compute distances and write pvalue into array
         pval = dist(geneCol, mirnaCol).pval
         miRNA_geneX_pvals[j] = [miRNA.name, gene.name, pval]
    # now that you have the array of distances between gene X and all miRNAs you can use multiple hypothesis correction.
    qvals = adjust(miRNA_geneX_pvals['pval'])
    # pick q-val cut off
    top_mirna_geneX = miRNA_geneX_pvals[qvals<0.1]

top_adjusted_pairs.union(top_mirna_geneX)

```


Mutually exclusive classes are usually integers; in your case, win=0, draw=1, loss=2 (for instance).

It is mostly a convention and is convenient for implementation.


Doubt on retrieving tweets for sentiment analysis?

First you need to setup the twitter api and get the credentials and add those to the congig.cfg file linked here-
https://filebin.net/5a6lfserhqzjg93a/config.cfg?t=dcqh998i

Afterwards you can use this link https://filebin.net/nfeju0bjvh8tocl3/tweetering.py?t=dy6bk7x5 to download the tweets which contain Uber.
Just run it on command line using - python tweetering.py Uber

Hope it works for you. You can also find many links on medium.com where you can find the complete twitter sentiment analysis steps, reading which you will understand a lot of things.


export R neuralnet package model to PMML

You already mentioned the alternatives...

convert the neuralnet object to a nnet class object
implement pmml.neuralnet()

Both pmml, nnet and neuralnet packages are easy to read. You can find the source code over cran.

I think implementing pmml.neuralnet() might be the cleanest. You can just copy-paste the current pmml.nnet()  method, call it pmml.neturalnet() and make the necessary changes. Should not take more than a day's work.

ps: It would be cool if you'd sent your function to neuralnet authors afterwards of course, and link it from here. :)


In XGBoost, how is a leaf index corresponding to the particular leaf node in actual base learner trees?

I think what you are seeing is the fact that all nodes in the tree are indexed because a priori the model doesn't know where splits will happen (i.e. any node could be a leaf). My guess is that the nodes follow an ordering similar to:

In your case all of the leaf nodes are at the max depth of the tree, so nodes 3-6 show up in your list. By contrast, if your data was all the same value I would expect all labels to be node 0 (because the split criteria was not met). And then you could have intermediate situations where after 1 split there is a node which does not meet the split criteria (in this case you could see either node 1 or 2 show up in your list).
Hope this helps!


How to use PCA in CNN for image recognition using Keras?

Standard implementations of PCA calculates statistics across the entire dataset in order to find the projection that has the greatest variance. The entire dataset needs to be loaded into memory for that calculation.
You are using ImageDataGenerator to generate synthetic variations of the data, greatly increasing the size of the training set that has to be loaded into memory.
Here a couple of options to run PCA:

Reduce the number of variations in ImageDataGenerator
Use approximate PCA
Get a larger machine or distribute across a cluster


How to match a user with other users with similar interests based on their attributes?

One possible approach would be to create N classifiers (one per each user) and then pick M random items, and run those into the N classifiers.
The outcome would be something like:

        User 1 | User 2 | ... | User N
Item 1:    1   |    0   | ... |   1    --> User 1 and N both like item 1
Item 2:    1   |    1   | ... |   1    --> All users like item 2
...      ...   |   ...  | ... |  ...
Item M:    0   |    0   | ... |   0    --> No user likes item M

where the i-th row contains the result of running i-th item in all N classifiers, and the j-th column contains the results of running all the M items in the j-th classifier.

Then you could see each user as a M-dimentional point, and use a simple classifier such as KNN with haming distance for the distance metric.

With a larger M, you'd get more accurate results, since you're using more variables to compare each user.
The only caveat here is that you'd need those N classifiers to be very accurate, in order to minimize error propagation.


Counting how many rows contain chinese in mixed value dataset using python

That should be feasible using some more python and pandas features:
- errors can be caught when you wrap the detect in your own function and return None (or what you prefer) instead of a language code
- results of the detect can be added to the df and counted, filtered, you name it.

def mydetect(txt):
    try:
        return detect(txt)
    except:
        return None

Pandas allows you to add the results to a second column of the dataframe, but I do not have the command handy at the moment, please check a tutorial.
Then you can filter the dataset or count occurences of a language.

You can also limit the except clause to certain errors if you think that would make more sense.


Consider an equation of the form

$y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \epsilon$

where $x$'s are the variables and $\beta$'s are the parameters. Here, y is a linear function of $\beta$'s (linear in parameters) and also a linear function of $x$'s (linear in variables). If you change the equation to

$y = \beta_0 + \beta_1x_1 + \beta_2x_1^2 + \epsilon$

Then, it is no longer linear in variables (because of the squared term) but it is still linear in parameters. And for (multiple) linear regression, that's all that matters because in the end, you are trying to find a set of $\beta$'s that minimizes a loss function. For that, you need to solve a system of linear equations. Given its nice properties, it has a closed form solution that makes our lives easier. Things get harder when you deal with nonlinear equations.

Assume you are not dealing with a regression model but instead you have a mathematical programming problem: You are trying to minimize an objective function of the form $c^Tx$ subject to a set of constraints: $Ax \geq b$ and $x\geq0$. This is a linear programming problem in the sense that it is linear in variables. Unlike the regression model, you are trying to find a set of $x$'s (variables) that satisfies the constraints and minimizes the objective function. This will also require you to solve systems of linear equations but here it will be linear in variables. Your parameters won't have any effect on that system of linear equations.


Best way to classify datasets with mixed types of attributes

Christopher's answers seem very reasonable. In particular tree based methods do well with this sort of data because they branch on discriminative features. It's a little hard to say without knowing your specific application, but in general if you think that some of your features might be significantly more discriminative than others, you could try some dimensionality reduction techniques to clean this up a bit.

Also if you use a dimensionality reduction technique you end up getting a slightly more robust format for your feature vector (they generally end up being straight numerical vectors instead of mixed data types), which might let you leverage different methods. You could also look into hand engineering features. With properly hand engineered features Random Forest will get you very close to state of the art on most tasks.


Finding unpredictability or uncertainty in a time series

Since you have looked at Kolmogorov-Smirnov and Shannon entropy measures, I would like to suggest some other hopefully relevant options. First of all, you could take a look at the so-called approximate entropy $ApEn$. Other potential statistics include block entropy, T-complexity (T-entropy) as well as Tsallis entropy: http://members.noa.gr/anastasi/papers/B29.pdf

In addition to the above-mentioned potential measures, I would like to suggest to have a look at available statistics in Bayesian inference-based model of stochastic volatility in time series, implemented in R package stochvol: http://cran.r-project.org/web/packages/stochvol (see detailed vignette). Such statistics of uncertainty include overall level of volatility $\mu$, persistence $\phi$ and volatility of volatility $\sigma$: http://simpsonm.public.iastate.edu/BlogPosts/btcvol/KastnerFruwhirthSchnatterASISstochvol.pdf. A comprehensive example of using stochastic volatility model approach and stochvol package can be found in the excellent blog post "Exactly how volatile is bitcoin?" by Matt Simpson.


How to get a count the number of observations for each year with a Pandas datetime column?

from collecitons import Counter
Counter([date.year for date in df_raw['filed_date'].values])


Average Precision if Target Class is Not in Evaluation

I'm not sure if there is a standard for this kind of case. I think it depends whether the system predicts any of these classes, i.e. whether there are any false positive cases:

if yes, then the precision must be zero indeed: out of all the instances predicted positive none was a true positive.
if no, I think it makes more sense to consider the precision as undefined (NaN), since no instance is predicted positive at all (that is, there is a division by zero in the calculation of precision).

This is a very borderline case, since it would be very questionable to use a test set which doesn't contain all the classes contained in the training set: such a test set cannot is not fit for the purpose of evaluation.


This should not be a problem, the convolutional kernels will have bigger and bigger receptive fields when you get deeper, and either there will be kernels at the end that will combine all the components over the whole input if the receptive field is big enough, or if you use a fully connected layer at the end it will learn that specific combinations of components combine into a specific class. The number of pieces it will be able to recognize depends on the expressiveness of your network, the amount of training samples and on the difficulty of your problem.


Unable to resolve Type error using Tokenizer.tokenize from NLTK

w_tokenizer = nltk.tokenize.WhitespaceTokenizer()

dfimpnetc[column] = dfimpnetc[column].apply(lambda x: [lemmatizer.lemmatize(w) for w in w_tokenizer.tokenize(x)])

Try the above code.

Mark as correct if this helps ;)


Learning rate gives the rate of speed where the gradient moves during gradient descent. Setting it too high would make your path instable, too low would make convergence slow. Put it to zero means your model isn't learning anything from the gradients.


Object detection stops predicting well according to how I collect the images

On my problem I have to use 4k resolution (3840x2160) to record images
and detect qrcodes at the same time. The images I end resizing it to
1920x1080 to save it on disk.

Have you tried using a HAAR Cascade classifier to crop out the region with the QR code so that you'll have to deal with only an image at lower dimensions. I guess that will help you with the frame rate.

The problem is that if I train using this resized data the model stops
predicting well compared to when I record the video on native
1920x1080. I can't have good results nor if I predict with resized to
1920x1080 neither native 1920x1080 with this train. If I train using
native recorded at 1920x1080 it predicts well both with native
1920x1080 and resized 1920x1080.

Im not entirely sure what you've meant by this. A bit more clarity would be nice !

Is there some explanation about why this is occurring?

If youre training on the resized 4k image and testing on the 1920x1080 image its natural that this happens. because the sub sampling removes features. It's better to look at a few resizing algorithms and write your own resizer. because qr codes have intricate details that can be missed while sub sampling

I always try to save the images as jps with quality=95 and the camera
I'm using compresses the image before sending to opencv (directly
affects the fps).

Instead of this why can't it be recorded in 1920x1080 and supersampled using interpolations. This would not be as good as the 4k but it will be close. And then you can sharpen the image until it's at par with 4k


Remove all characters following a certain character in a column of a dataset


How to perform link prediction in text based relationship data

You are describing building a Probabilistic Graphical Model (PGM).

The most commonly used Python library to build a PGM is pgmpy.


How to report results of RL with high variance?

One way is to calculate a moving average and moving standard deviation.
My preferable method is smoothing (like tensorboard's smoothing function). Here is a very nice script in python. Just tune the weight parameter to get the results you want. For example:


For obejct detection, should I resize my custom images first and then start the annotation or it won't matter?

It is best practice to resize the input data to the size a pretrained model expects. A pretrained model has feature detectors for relative differences.
Resizing should be done early in the process so subsequent steps use consistent data. Sometimes resizing is a non-linear process that can use interpolation.


What are some method for pre-processing data in OCR?

You can do image tresholding, normalize the data first by dividing each elements of the vector by 255, then do thresholding, for each $x_i \in \mathbb{x}$ $x_i=1$ if $threshold\le x_i$ where $\mathbb{x}$ is your data vector, and $x_i$ is the element of vector, you can choose the threshold based on experiment or personaly I choose 0.5. By doing image thresholding we reduce the noise of the image, hence its increase the accuracy


How does a random forest algorithm deal with a few irrelevant input variables

Random Forest tends to be not too sensitive to features with low predictive power. The reason is that RF looks for a "best split" given a subset of features (columns) and observations (rows). So "weak" features will likely be ignored in most cases.
However, removing the $x$ percent weakest features may increase the model's performance. In case you use sklearn, there are convenience functions to do this, e.g. SelectFromModel(). See the docs for more details.
>>> from sklearn.ensemble import ExtraTreesClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.feature_selection import SelectFromModel
>>> X, y = load_iris(return_X_y=True)
>>> X.shape
(150, 4)
>>> clf = ExtraTreesClassifier(n_estimators=50)
>>> clf = clf.fit(X, y)
>>> clf.feature_importances_
array([ 0.04...,  0.05...,  0.4...,  0.4...])
>>> model = SelectFromModel(clf, prefit=True)
>>> X_new = model.transform(X)
>>> X_new.shape
(150, 2)


Quasi-linearity in deep learning regression problems (sports betting)

There are several principles in Data Science and Statistics which you will have to know before undertaking a project of any kind.

One of them is what is called parsimony principle (you could know him with other words) but it says that if you have two models with similar results, the simpler, with less parameters model should be chosen.

As you must have read, the neural networks have A LOT of parameters which of course might allow to learn your database (not generalize your model to not-seen-before data).

My recommendation is you use linear regression if you have a linear problem.

The neural networks can represent a linear problem too: Can neural network represent a relation that is a derivative?

But, if you want to do neural networks anyway, I will have to tell you that they work by trial and error. There are rules and suggested architectures, but you will have to find the best by yourself


Predictive output with your own model built

What you are proposing is a heuristic method, because you define the rules manually in advance. From a Machine Learning (ML) point of view the "training" is the part where you observe some data and decide which rules to apply, and the "testing" is when you run a program which applies these rules to obtain a predicted label. As you correctly understood, the testing part should be applied to a test set made of unseen instances. The instances in the test set should also be manually labelled (preferably before performing the testing in order to avoid any bias), so that you can evaluate your method (i.e. calculate the performance).
Technically you're not using any ML approach here, since there is no part where you automatically train a model. However heuristics can be useful, in particular they are sometimes used as a baseline to compare ML models against.

[addition following comment]

I think most of common pre-processing approach requires to convert text into lower case, but a word, taken in different contest, can have a different weight.

This is true for a lot of tasks in NLP (Natural Language Processing) but not all of them. For example for tasks related to capturing an author's writing style (stylometry) one wouldn't usually preprocess text this way. The choice of the representation of the text as features depends on the task so the choice is part of the design, there's no universal method.

how to train a model which can 'learn' to consider important upper case words and punctuation?

In traditional ML (i.e. statistical ML, as opposed to Deep Learning), this question is related to feature engineering, i.e. finding the best way to represent an instance (with features) in relation with the task: if you think it makes sense for your task to have specific features to represent these things, you just add them: for instance you can add a boolean feature which is true if the instance contains at least one uppercase word, a numeric feature which represents the number of punctuation signs in the instance, etc.
Recent ML packages propose standard ways to represent text instances as features and it's often very convenient, but it's important to keep in mind that it's not the only way. Additionally nowadays Deep Learning methods offer ways to bypass feature engineering so there's a bit of a tendency to forget about it, but imho it's an important part of the design, if only to understand how the model works.


Time Series pattern recognition and classification problem

I do not fully understand what you mean by "average frequency". But this may help to think about it.

You can calculate the power spectrum of the signal using DFT.
From the power spectrum, you will be able to get the average power of all the frequencies.
Use this value of average power to map the frequency which it closely corresponds to.

This will give a sort of the average frequency you are looking for from the average value of power.


I solved this by using torch.where to give the irrelevent entries a really big negative value, so that they vanish after an exp. I also took advantage of the log of CEL trick.

loss = torch.zeros(1).type(dtype)

states = torch.cat([states_[_] for _ in idc[:self.batch_size]]).type(dtype)
scores = torch.tensor([scores_[_] for _ in idc[:self.batch_size]]).type(dtype).view(-1, 1)
policies = torch.cat([policies_[_] for _ in idc[:self.batch_size]]).type(dtype)

values, logits = self.network(states)
value_loss = mse(values, scores)

p = torch.nn.functional.normalize(policies, dim=[1,2,3], p=1)
logit_exp = torch.exp(torch.where(p > 0, logits, inf))
s = torch.log(torch.sum(logit_exp, dim=[1,2,3])).view(-1, 1, 1, 1)
log_q = logits - torch.mul(ones, s)
policy_loss = torch.sum(torch.mul(p, -log_q))/self.batch_size

loss = value_loss + policy_loss
loss.backward()
optimizer.step()
optimizer.zero_grad()
idc = idc[self.batch_size:]


Your model have an accuracy of 0.10 so he is correct 10% of the time, a random model would do the same. It means your model doesn't learn at all. Even a bad model learn a little. So the problem come from your dataset.

I tested your model and got 97% accuracy.

Your problem probably come from how you import your dataset.
Here is how i imported:

import idx2numpy
import numpy as np
fileImg = 'data/train-images.idx3-ubyte'
fileLabel= 'data/train-labels.idx1-ubyte'
arrImg = idx2numpy.convert_from_file(fileImg)
arrLabels = idx2numpy.convert_from_file(fileLabel)

the dataset come from: http://yann.lecun.com/exdb/mnist/

You should test if you imported an processed your dataset correctly. For example :

import matplotlib.pyplot as plt
image = np.asarray(arrImg[1000])
plt.imshow(image)
plt.show()
print(arrLabels[1000])


Creating a valid dataset for obtaining results

There are two potential biases:

With this automatic method, you might have a few erroneous labels. For example it happens regularly here on DataScienceSE that a user tags a question "python" but actually the question is not specific to python at all. Same thing for the opposite case: for example it's possible that some content contains some python code but doesn't mention "python" anywhere.
The distribution between positive and negative classes is arbitrary. Let's assume you use 50% positive / 50% negative: if later you want to apply your classifier on a new data science website where only 10% of the content is about python, it's likely to predict a lot of false positive cases so the true performance on this data will be much lower than on your test set.

It's rare to have a perfect dataset, so realistically in my opinion the first issue is probably acceptable because the noise in the labels should be very limited. The second issue could be a bit more serious but this depends on what is the end application. Keep in mind that a trained model is meant to be applied to the same kind of data as it was trained/tested on.