Forecasting multiple time series with a single model

You can treat the markets as a categorical feature in a tree based (using decision tree as a weak leaner) ensemble model such as random forest or gradient boosting. Some applications are:

Sales for retail outlets of a major European Pharmacy
retailing company
Predicting the sales of
products from different outlets
Multi-Store multi-product forecasting


Is there a straightforward way to run pandas.DataFrame.isin in parallel?

Unfortunately, parallelization is not yet implemented in pandas. You can join this github issue if you want to participate in the development of this feature.

I don't know any "magic unicorn package" for this purposes, so the best thing will be write your own solution. But if you still don't want to spend time on that and want to learn something new – you can try the two methods built into MongoDB (map reduce and agg framework).  See mongodb_agg_framework.


why do we need row sampling in random forests?

First I think your understanding of “column sampling” is incorrect.  Random forest try’s a subset of features for each split.  It does not sample without replacement within an individual tree.
Random forest samples rows with replacement (bootstrap samples) to remove correlation between the decision trees.  Think about it, if you didn’t do this even though you create each split based on only a subset of features, your trees would end up looking fairly similar (or at least more similar than if you bootstrapped).  You do have larger bias due to creating trees based only on about ~63% unique values but the decrease in variance by having more uncorrelated trees makes up for it.


Forecasting/predicting techniques for qualitative data?

You can encode the various columns of qualitative data using LabelEncoder(). For example:

label_encoder = LabelEncoder()
risk = label_encoder.fit_transform(df['RISK_DECISION'])
action = label_encoder.fit_transform(df['ACTION_TAKEN])
etc...


Your reasoning is correct: for most tasks related to information retrieval and/or document classification based on the semantics of the documents, it's recommended to take into account the importance of the terms (both inside the document and across all documents, hence TF and IDF).

However TF-IDF is not necessarily always the best choice:

There are some classification tasks which are not based on the semantics of the document. For example if the goal is to classify documents by writing style (e.g. find documents by the same author) then the topic doesn't matter and therefore IDF is not relevant.
In the case of a very small dataset and/or very short documents, using TF-IDF scores can lead to overfitting. In such cases using boolean values might perform better because it makes the job of the model easier.


What to do if one out of 2 one-hot encoding variables have a very high p-value?

You should keep all of levels as they collectively describe the feature. Removing the insignificant ones will bias your coefficients and distort your interpretation (e.i. change the reference level).

Here are some stats.exchange references:

Is it advisable to drop certain levels of a categorical variable?
Can I ignore coefficients for non-significant levels of factors in a linear model?


Find suitable locations using Machine Learning

Are you sure PCA is the correct way to go?
It's an analytical problem and being able to interpret the results are very important.

How about the correlation between the number of stores and nearby features? Find out what makes a good location. What are the most important features? Run forward or backward selection as an example, or use another model/feature selection technique.

It's not a pure machine learning case you have here. It's a typical analytical data science problem.

If you still want to do classification, just train a model. You have POI features and some others. You know if there is a store or not :) I might not fully understand the problem here. You train on a 50% a store exist location, and 50% a store does not exist in this location dataset. Train a classifier, and classify other areas.

I'd still start to visualize and understand the data as I mentioned first. It's much underrated and the way to start solving most problems.

Hope that gave you some hints,

Cheers


Latent Dirichlet allocation (LDA) is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. LDA does exactly the same thing you wanted to do.


You could use the median. That protects against the "Bill Gates walks into a bar, making everyone there a millionaire, on average" effect. It's used with wage distributions for exactly the reason that they tend to be very skewed.


Combining Classifiers with different Precision and Recall values

You want to ensemble your two algorithms. The way to do that is to not just use e.g. sklearn's precision and recall metric functions, but to actually obtain probabilities for being positive (most models can output this with a simple function call), and take the average of those probabilities between your two (or more) binary classifiers. Then, test a range of probability thresholds beyond which you might label something positive, and evaluate the impact on recall and precision. This is a pretty common approach to obtaining better results through utilizing the strengths of different prediction algorithms.

Just some pythonic pseudo-code to demonstrate:

recall = []
precision = []
for threshold in [0.1, 0.2, 0.3, ..., 0.9]:
    df_of_mean_probabilities['label'] = df_of_mean_probabilities > threshold
    recall.append(TP/(TP + FN))
    precision.append(TP/(TP + FP))


Is there any difference between 'classification' and 'categorization' based on machine learning terminology?

As noted by @Emre in the comments:

"Same thing. Just call it classification; it's the more common name for the task."

Although there is no difference with respect to machine learning, this paper may provide another perspective in detail.


YOLOv1 algorithm - how to determine predictor responsibility

Confidence score

Some resources like Andrew Ngs deep.ai here set the confidence of the corresponding label vector bounding box confidence to "1" if there is an object present at that grid cell.

This is because prof. Ng is refering to true probability of an object being in a specific cell ($P(\text{object})$ value), and not the confidence score ($P(\text{Object}) \times\text{IOU}$ value). Thereby, we have two possible situations:
$$
P(\text{object}) = \begin{cases}
1 \text{ if an object is present in the cell}\\
0 \text{ otherwise}
\end{cases}
$$
However, this is not what YOLO is proposing to have as ground-truth value of confidence. Citing the paper:

[...] Formally, we define confidence as $P(\text{Object}) \times\text{IOU}^{\text{truth}}_{\text{pred}}$. If no object exists
in that cell, the confidence scores should be zero [...]

Thereby, the value of confidence that the network is trying to predict (the confidence ground-truth label) is given by:
$$ P(\text{Object}) \times\text{IOU} = \begin{cases}
\text{IOU if an object is present in the cell}\\
0 \text{ otherwise}
\end{cases}
$$
Because of this, the other source mentioned in the question (Oracle's article) is the correct one.

Bounding boxes question

The paper states we should assing a b.b. predictor to a ground truth object, but once we have done so, that bounding box predictor is "taken" and married to that object, so it can't be assigned to any other object in that grid cell?

During training I don't see why this can be a problem. If the predicted bounding box of a certain grid cell, has the highest IOU for two or more ground-truth bounding boxes (whose center fall on this cell), we can interpret that the predicted bounding box is trying to predict both ground-truth boxes (it may happen for example if both ground truth boxes have similar sizes/ aspect ratios).
Concretely, what's said by the authors is the following:

YOLO predicts multiple bounding boxes per grid cell.
At training time we only want one bounding box predictor
to be responsible for each object. We assign one predictor
to be “responsible” for predicting an object based on which
prediction has the highest current IOU with the ground
truth. This leads to specialization between the bounding box
predictors. Each predictor gets better at predicting certain
sizes, aspect ratios, or classes of object, improving overall
recall.

The part in bold only implies that a ground-truth object is related only with one predicted bounding box $\Rightarrow$ a predicted bounding box can be related to more than one ground-truth object.

Edit:
Given what's said in the article, during training if a predicted bounding box has the highest intersection over union with two different objects, then the loss function will be related to both ground-truth bounding boxes.
For example, if we have a look at the part of the loss function related to the prediction of the center of the bounding box $(\hat{x}_i, \hat{y}_i)$:
$$ \lambda_{coord}\sum_{i=0}^{S^2}\sum_{j=0}^B1_{ij}^{obj}\left[ (x_i-\hat{x}_i)^2 + (y_i-\hat{y}_i)^2
  \right]$$
We have that $(x_i,y_i)$ are the coordinates of the center of the ground-truth bounding box, and $(\hat{x}_i,\hat{y}_i)$ are the predicted ones.
Given this, particularizing to a certain grid cell, $i$, and to our predicted bounding box of interest, $j$, we have that its contribution to this part of the loss function is:
$$ \lambda_{coord}\left[ (x_i^{\{0\}}-\hat{x}_i^{\{0\}})^2 + (y_i^{\{0\}}-\hat{y}_i^{\{0\}})^2
  + (x_i^{\{1\}}-\hat{x}_i^{\{1\}})^2 + (y_i^{\{1\}}-\hat{y}_i^{\{1\}})^2\right]$$
Where the superscripts $\{0\}$ and $\{1\}$ represent each ob the ground-truth objects being detected.
As we see, we will be computing this part of the loss using both ground-truth bounding boxes. This makes sense if the objects that are being detected have similar sizes/ aspect ratios and are close to each other (they fall on the same grid cell).
If we focus on the confidence score part of the loss function, where we evaluate the predicted $\text{IOU}$ ($\hat{C}_i$), with the ground truth $\text{IOU}$ ($C_i$):
$$\sum_{i=0}^{S^2}\sum_{j=0}^B 1_{ij}^{obj}(C_i -\hat{C}_i)^2$$
Then, particularizing to a certain grid cell, $i$, and to a certain bounding box, its double contribution will be given by:
$$(C_i^{\{0\}} -\hat{C}_i^{\{0\}})^2 + (C_i^{\{1\}} -\hat{C}_i^{\{1\}})^2$$
Where the superscripts ${0}$ and ${1}$ have been used again to address this specific situation.

Note: Delving into this question I realized that the previous arguments are just a interpretation of what the algorithm would do in this specific situation. I think that another plausible methodology to follow on this situation would be to just use the ground-truth object with the highest $\text{IOU}$.


Modeling the influence of events order on probability

If the order in which the events appears matters, consider using a recurrent neural network. The setup that you propose is invariant to event ordering, whereas in a RNN the events are fed in in sequential order.


Decision Tree Classifier to classify values based on values of other columns

Couple of things:
Cant replicate the problem exactly but if you follow these steps you are not exposing yourself

TfidfVectorizer is CountVectorizer + TfidfTransformer, you are exposing yourself to unnecessary complexity and potential errors

Use pipelines, cant stress this enough, its a compact way to pack all the sklearn transformers together AND THEN use fit, predict methods...

I would advise you to follow something like that, or find similiar problem here


How to calculate compression ratio when using autoencoder in neural network

Usually the compression ratio is measured on the files. So the correct way of computing it is to calculate the size of you data before compression, run them through the AE, store them to your drive and then see how much size they occupy after compression.

Then simply divide the two:

$$
Compression \; Ratio = \frac{Uncompressed \; Size}{Compressed \; Size}
$$

Theoretically in your case this should give a 40:1 compression ratio, but, due to overhead and the way you store your data it might not be so high.


CNN can't predict images outside the dataset

That could a lot of reasons for that. For example, images are sometimes represented with numbers in range 0-1 and sometimes in range 0-255 and it's very easy to mix these ranges for in-dataset / external as the model would fail silently without any warnings.
In general, if the same model gives you different results, then images are not exactly the same.
I suggest you take the same image from your dataset and internet and compare the raw numbers on the input of the model. If results different, then input tensors should be different and you can check your preprocessing pipeline step by step to find the point of discrepancy.
The debugging tools like ipdb are usually very useful in such situations.


How to decide what threshold to use for removing low-variance features?

You can do this a few ways, which I can list in ascending order of effort:

Pick a value that seems ok for you and your dataset by eye-balling it then simply cut variables below the theshold from the dataset
Create a function, which given a threshold, tells you how many variables would be removed, if you used that threshold. Then create a simple plot and see if there is a certain level that seems appealing (this depends on your target model once data is ready).
Use some smarter functions that d a little more for you, e.g. the NearZeroVar function in the Caret package in R

There are, however, some arguments as to whether these approaches in general are optimal. Have a look at some of the discussion on this thread over at Cross-Validated. The quote from the OP in that thread if from the book of the guy who wrote the above mentioned Caret package - Max Kuhn.

The arguments against this approach say that you may be moving variables that, although they have low variance, might actually be extremely powerful in explaining your target (dependent) variable.

A final approach I can suggest goes into the realm of covariance, that is to look at the collinearity between pairs of variables. I have done this in the past and it worked out well for me. The basic algorithm would look something like this:

Compute the covariance matrix across all your variables
Find the pair with the highest covariance

correlation if you prefer that measure
the plan is to remove one of these two variables

using your covariance matrix from step 1, compute which of these two variables in step 2 has the highest sum of covariances with the rest of the variables
remove the variable with the highest sum of covariances
Repeat steps 1-4 until you arrive at your desired number of variables, or a threshold is reached in terms of covariances or individual variable variances.

EDIT:

Here is the Scikit-Learn class which can do basic variance thresholding for you - there is also a short tutorial. They also present some ways to do recursive feature selection, similar in nature to my final approach outlined above.


Database on time to churn in telecomunication

instead of the database, search dataset

https://www.google.com/search?q=telecomunication+churn+dataset&ie=utf-8&oe=utf-8&client=firefox-b-ab


Review of Restricted Boltzmann Machines

A restricted Boltzmann machine (RBM) is a generative model, which learns a probability distribution over the input. That means, after being trained, the RBM can generate new samples from the learned probability distribution. The probability distribution over the visible units $\mathbf{v}$ is given by
$$p(\mathbf{v} \mid \mathbf{h}) = \prod_{i=0}^V p(v_i \mid \mathbf{h}),$$
where
$$p(v_i \mid \mathbf{h}) = \sigma\left( a_i + \sum_{j=0}^H w_{ji} h_j \right)$$
and $\sigma$ is the sigmoid function, $a_i$ is the bias for the visible node $i$, and $w_{ji}$ is the weight from $h_j$ to $v_i$.
From these two equations, it follows that $p(\mathbf{v} \mid \mathbf{h})$ only depends on the hidden states $\mathbf{h}$.
That means that the information on how a visible sample $\mathbf{v}$ is generated, has to be stored in the hidden units, the weights and the biases.

Using RBMs for classification

When using RBMs for classification tasks, you use the following idea: as the information on how your training or test data was generated is saved in the hidden units $\mathbf{h}$, you can extract these underlying factors by feeding a training sample into the visible units of the RBM, propagate it forward to the hidden units, and use this vector of hidden units as a feature vector. You don't do any backwards pass to the visible units anymore.

This hidden vector is just a transformed version of the input data - this can not classify anything by itself. To do a classification, you would train any classifier (linear classifier, SVM, a feedforward neural network, or anything else) with the hidden vector instead of the "raw" training data as inputs.

If you are building a deep belief network (DBN) - which was used to pre-train deep feed-forward neural networks in an unsupervised fashion - you would take this hidden vector and use it as the input to a new RBM, which you stack on top of it. That way, you can train the network layer-by-layer until reaching the desired size, without needing any labeled data. Finally, you'd add e.g. a softmax layer to the top, and train the whole network with backpropagation on your classification task.


How to find optimal number of trees in random forest using Grid search in R?

mtry is the parameter indicating how many of the features are checked in each split decision.  http://topepo.github.io/caret/train-models-by-tag.html#random-forest , see method 'rf'

In Random Forest, usually more trees give more stable results, and overfitting due to number of trees is rare.  Moreover, since the trees are built independently, you could just fit many trees then take subsets to get smaller models.  But, see e.g. hyperparameter tuning in mlr for how to perform grid, random, Bayesian, and other hyperparameter searches, or grid in caret or random in caret.


Pandas: Compare two Dataframe and Groupby categorical

Try:
Minimum reproducible example:
 A = pd.DataFrame({"Text":["01", "02","03","04"], "Category":[1,1,1,1]})
 B = pd.DataFrame({"Text":["05", "02","09","04"], "Category":[2,2,2,2]})

 C = pd.concat([A,B], ignore_index= True)

 C.groupby("Text").Category.unique().to_frame()

Result:


Format for proposing a new algorithmic project?

There you go cookiecutter template. I use this, with all the necessary modifications. Each project can be really versatile in Data Science. Nonetheless, I would agree that is a general standard.


I would start with the simple option: represent every person as a boolean vector in which every position represents the answer to a particular question (the length is the total number of questions). Then you can apply any standard clustering algorithm, such as K-means (hierarchical clustering would probably also work with data like this).


Why my regression model always be dominanted by one feature?

The first thing that comes to my mind is that you might have not normalized your features correctly. Generally a feature ranging between a bigger range of values, compared to the other ones, is going to be more influencial in terms of the models' output.
In order to midigate this issue, one common practise is to transform your features into having zero-mean and a variance of one. In this way, it is guaranteed that all your features have identical range of values.
Other than that, It may be just that the dominant feature is indeed more indicative for your time series prediction and thus your model has learned to rely its predictions on this specific feature.


Would a convolution network make sense for policy based tic-tac-toe approach?

For anyone interested, I'm posting my results of trying out both CNNs and MLPs -

A 3x3 kernel 4 feature maps CNN consistently performed better than a 9 neuron MLP. I trained both on completely random tic-tac-toe training data size of 10kx20 and the CNN learned to to achieve ~80% win/draw against a random opponent while the MLP achieved around ~60% for the same data. A 2x2 CNN performed much worse than 3x3.

Training data - Inputs and outputs are same as mentioned in the question with the exception that the AI is trained only if it loses or draws the game and the dataset is chosen from randomized player for drawn games.

Note: It is possible to achieve much better accuracy with less data but in this case my goal was to just use this game to learn machine learning programming so the code was not particularly optimized for anything. The comparison of MLPs and CNNs is also incidental and needs to be done with more rigor for any serious purposes.


Use single Convolution layer for the classification of two kinds of matrix?

A Convolution neural network is not justified here.
CNN is used for images because -

Flattening the data can create very high dimensional features

Exerpt from "Hands-on Machine Learning" book -
For example, a 100 × 100–pixel image has 10,000 pixels, and if the first layer has just 1,000 neurons (which already severely restricts the amount of information transmitted to the next layer), this means a total of 10 million connections.
2. Nearby pixels tell a lot about the feature. So by using convolution, Pooling, we are able to reduce the dimension without losing information. The best part is that learning(Identifying the features) is part of Backpropagation
3. Weight sharing reduces the parm count further

 In the case mentioned in the question -

Dimension is not so large i.e. 10x10 = 100
Data is sparse and binary. I doubt it can represent spatial features. CNN is not good for sparse dataset.
Only one kernel

I doubt real CNN is happening in the background. You are getting one Feature map and then it is flattened to work as a normal Neural network.
So, it would be better to use a normal neural network
Before that, you should try SVM once as it works well on the sparse dataset. Though data-count is a bit large but can try once.


Training Image classifier on tiny dataset

CNNs require a large number of images to be trained properly. The most common workaround when having a small training set is to use a pre-trained model, which has been trained on a huge dataset (e.g. ImageNet). Through this you take a highly effective model, drop its final layers (which are used for classification) and keep all the previous (which are used for feature extraction). Then you add your own final layers and only train them. This allows you to train a model with much less parameters (because most of them have already been trained), which can be done on smaller datasets.

Another thing you can do is data augmentation. This usually consists of affine transformations (rotations, translations, scaling, shifting, etc.), flipping the image, brightness/contrast adjustments, noise induction, etc. The problem with this is that strong augmentations might ruin the image. This limits you in terms of how many and how strong augmentors you use.

That being said, in terms of image classification a tiny dataset could be considered one with a few hundred samples. If you have just one image per class, I'm not sure that you can ever effectively train a CNN without overfitting. My suggestion is, if possible, obtain more data.


How to implement global contrast normalization in python?

there are multiple issues with the code:

You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1.
This will later be interpreted as image in black and the darkest form of gray (1 out of 255).
Once you have proper floats as values PIL or pillow can't handle the array (they only do images with values in [0, 255])

The first problem happened because you/numpy wants the array to be a uint8.
The normalize version will have floats.

You should have used:

X_prime = X.astype(float)

Here is a working version of the code:

import numpy
import scipy
import scipy.misc
from PIL import Image

def global_contrast_normalization(filename, s, lmda, epsilon):
    X = numpy.array(Image.open(filename))

    # replacement for the loop
    X_average = numpy.mean(X)
    print('Mean: ', X_average)
    X = X - X_average

    # `su` is here the mean, instead of the sum
    contrast = numpy.sqrt(lmda + numpy.mean(X**2))

    X = s * X / max(contrast, epsilon)

    # scipy can handle it
    scipy.misc.imsave('result.jpg', X)

global_contrast_normalization("cat.jpg", 1, 10, 0.000000001)

PS: X_prime = X will make X_prime reference X. So changing X_prime will also change X.


There is no hard-and-fast rule for this.

The number of hidden nodes you should have is based on a complex relationship between

Number of input and output nodes
Amount of training data available
Complexity of the function that is trying to be learned
The training algorithm
Too few nodes will lead to high error for your system as the predictive factors might be too complex for a small number of nodes to capture
Too many nodes will overfit to your training data and not generalize well

To conclude,

There is no hard and fast rule. You just have to keep trying with different number of layers to see which one works best...
In neural networks, model architecture is an art that you can master with some experience and domain knowledge.

Check this answer too...


I think these are often used colloquially as synonyms, but let's try to find the differences.
Each of them begins with "Time Series" (TS). So the difference lies in the three following terms. here with my interpretation:

Analysis - wanting to describe and understand characteristics the observed data coming from the generating function$^1$.
Forecasting - wanting to somehow predict the future output of the generating function.
Modeling - wanting to describe and understand the generating function itself, via an approximation.

So as you see, I think time-series analysis and time-series modelling are almost the same thing. I just feel the latter most likely involves really using a real model, with parameters. That doesn't mean specifically parametric or a non-parametric approaches.
I personally wouldn't draw a line between them in terms of "uses statistics" vs. "uses machine learning", as your question suggests.

A natural progression in approaching a time-series problem (in my opinion) would be:

understand the data (TS-analysis)
understand the underlying function (TS-modelling)
test how well I have understood the function and its output (TS-forecasting)

$^1$
With the term "generating function", I don't mean the strict mathematical definition, rather just the underlying process or system that is generating the observed data


GPU memory requirement calculation from network architecture

There is no good way of estimating GPU RAM requirement based the number of layers and the number of nodes at each layer. The operations at each layer and connections between layers matter. For example, a multilayer perceptron (MLP) will use less compute than a Convolutional Neural Network (CNN).
A better idea would be monitor GPU usage during training. If the GPU is a NVIDIA, NVIDIA System Management Interface (nvidia-smi) is built for monitoring and management.


Why should bias increase as the training set size increases?
My intuition is that as you increase K, your test sets get smaller, increasing the variance of your evaluation metric. At the same time, you're fitting your model to more and more similar training data (since the training set in each fold is approaching the full set), so you're more likely to overfit to the training data because you're fitting your candidate models on almost the entire training data set in each fold.


Does a larger action space take longer to train an RL agent?

I think there are two things in your question:

The number of parameters of your network. So if you have more actions to predict the action layer of your network will have more parameters and it will take a little bit longer theoretically to update it (the time difference thought might be negligible depending your computational resources).
The total training time. Yes, having more choices it will take longer for the algorithm to generate some good estimations of expected reward per action in order to lead to a satisfactory performance. If for Critic you are using a state value $V(s)$ instead of $Q(s,a)$ still your algorithm will take longer to explore the action space and map your input state to a probability distribution over your action space.


Bug in sentiment analysis and classification for unlabeled text

First, congratulations for thinking to do a qualitative analysis of the results :) I know it should be obvious, but so many people just assume that the system works and don't bother checking their output.
Now, strictly speaking what you're seeing is not a bug. These are errors made by a statistical system. A statistical system is not meant to get everything right, it's only meant to label the input "to the best of its knowledge", and its knowledge is limited primarily by (1) the data it was trained with and (2) the assumptions made in the design of the model itself.
I don't know the exact characteristics of the systems that you used, but I can make an educated guess about the errors that you mention:

"fewer people are dying every day" is likely to be predicted as negative because it contains the word "dying". Probably there were no (or very few) examples in the training data which contain the word "die" and are labelled positive. As a consequence the system assumes that any sentence containing "die" is likely negative. One may notice that the positive semantics of "fewer people dying" is completely lost on the system, because it focuses on simple clues (individual words), it's not able to parse more complex phrases.
"The audience here in the hall has promised to remain silent." would be a similar case: the word "silent" or perhaps the two words "remain silent" likely were found only in negative examples during training, so the system just generalizes wrongly that a sentence containing these words is negative.
The sarcastic "Oh really?!" is an even more complex concept for the system to properly identify. The task of sarcasm detection is studied on its own because it's such a difficult task for a machine. I don't follow this field closely so I could be wrong, but I don't think the task has reached any satisfying level of maturity yet, let alone been integrated with standard sentiment analysis systems.

Nonetheless these errors don't mean that the results are useless. If you annotate manually a random sample and evaluate the performance of the system on this sample, hopefully you'll see that overall the system performs decently. That's what is expected of a statistical system: it's not reliable on an individual basis, but normally it's doing a good job in average.
More generally, all these errors show that the problem of Natural Language Understanding is far from being solved yet... and it might never be. The good news is that there's still a lot of interesting problems to solve for NLP scientists ;)


What is the meaning of Face Recognition in wild and in static?

In the wild refers to uncontrolled imagery, as opposed to images taken in lab environments or posed pictures such as driver's license/passport pictures or mug shots.
static in this context refers to still imagery, as opposed to temporal data (e.g. video).
With regard to facial expression recognition (not sure if you're interested in this related area), you could read this paper which describes a popular static database for images in the wild, which has good background information: Static Facial Expression Analysis in Tough Conditions: Data, Evaluation Protocol and Benchmark


I am currently using MLflow together with Databricks at my company, and up to now, I would say it is quite ok for managing models versions, tracking models metrics and building the models workflow from a dev/experimenting/training status to production.
I am not sure if it can be considered a standalone tool for that, meaning that you should combine it with something like Azure Databricks (what I currently use).
It's nice the way you can customize the tags, metrics or artifacts that you might want to keep logged after each run or model training, as follows:

where you can also visualize several run experiments as:

(these models are for the shake of the example, so metric values are not optimized)
and storing the generated models files (e.g. a pickle file) in the corresponding location:

Once it is generated, you can register your best model via the databricks UI (in case you finally use it) or via mlflow code as follows (here I register the last one, you should take the one with the best prefered metric which mlflow also gives you directly as an option):

so as to be loaded later on and make predictions with your model.
Once you your are happy with some model, you can transition it:

Another nice feature I like is that it offers you custom modules for the most used frameworks like scikitlearn, xgboost, keras, etc
and build a container image, deploy it... but this is another longer topic.
Of course, you can also use MLflow with another third party or even locally.
About the performance point, do you mean the inference itself or some other source of latency? You can make inferences with your model via SPARK UDFs (PySPARK in my case) info, in case you need several predictions as fast as possible or simply for batch inference.


1. what are the rows before the first red row? I thought it may be the combinations of parameters but that doesn't make a lot of sense because those are not enough

Parameters, which are the candidates of the CV are printed

2. What is the meaning of the row between the red rows? Why those parameters are there? is this after one CV?

It is printed after the completion of the 3rd(as defined in your code) fold for a parameter Candidate

3. "Done one task" - what is the task? one CV?

This is from the joblib I believe. It's the completion of one fold i.e. one out of 36.

4. Why do I get a score only for some of the parameters?

The score is printed after the completion of 3(as defined in your code) fold of each parameter candidate
In a multi-thread setup, we can't rely on the sequence of the messages as it will depend on the Thread allocation.
My answer is based on this small setup.
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RandomizedSearchCV
iris = load_iris()
logistic = LogisticRegression(solver='saga', tol=1e-2, random_state=0)

distributions = dict(C=[0.1,0.5,1.0],
                      penalty=['l2', 'l1'], max_iter=[50,100])
clf = RandomizedSearchCV(logistic, distributions, random_state=0,n_iter = 5, cv = 2, verbose=10)

from joblib import Parallel, delayed, parallel_backend
with parallel_backend('threading',n_jobs=12):
    search = clf.fit(iris.data, iris.target)

Output

Fitting 2 folds for each of 5 candidates, totalling 10 fits
[CV] penalty=l2, max_iter=100, C=0.5 .................................
[CV] penalty=l2, max_iter=100, C=0.5 .................................
[CV] penalty=l1, max_iter=100, C=1.0 .................................
[CV] penalty=l1, max_iter=100, C=1.0 .................................
[CV] penalty=l2, max_iter=50, C=0.5 ..................................
[CV] penalty=l2, max_iter=50, C=0.5 ..................................
[CV] penalty=l2, max_iter=100, C=1.0 .................................
[CV] penalty=l2, max_iter=100, C=1.0 .................................
[CV] penalty=l2, max_iter=100, C=0.1 .................................
[CV] penalty=l2, max_iter=100, C=0.1 .................................
[CV] ..... penalty=l2, max_iter=100, C=1.0, score=0.960, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=0.5, score=0.973, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=0.1, score=0.947, total=   0.0s
[CV] ..... penalty=l1, max_iter=100, C=1.0, score=0.973, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=1.0, score=0.973, total=   0.0s
[CV] ...... penalty=l2, max_iter=50, C=0.5, score=0.973, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=0.1, score=0.893, total=   0.0s
[CV] ..... penalty=l2, max_iter=100, C=0.5, score=0.960, total=   0.0s
[CV] ...... penalty=l2, max_iter=50, C=0.5, score=0.960, total=   0.0s
[CV] ..... penalty=l1, max_iter=100, C=1.0, score=0.987, total=   0.0s

[Parallel(n_jobs=12)]: Using backend ThreadingBackend with 12 concurrent workers.
[Parallel(n_jobs=12)]: Done   3 out of  10 | elapsed:    0.0s remaining:    0.1s
[Parallel(n_jobs=12)]: Done   5 out of  10 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=12)]: Done   7 out of  10 | elapsed:    0.0s remaining:    0.0s
[Parallel(n_jobs=12)]: Done  10 out of  10 | elapsed:    0.0s finished


You unroll the same RNN multiple times, but not in parallel. The RNN needs both an input and the previous step, therefore, you unroll the same RNN for the first time step, then for the second, and so on.
For each step it is always the same RNN, so the weights are the same each time:


Why softmax in YouTube’s DNN recommender

"Is it because in the candidate generation stage, the relative order of items does not matter?"

Yes this is exactly what appears to be going on, although it seems that youtube is using softmax in a untraditional way. The Candidate Generation model simply selects the few hundred candidate videos that are subsequently ranked by the ranking model.
Section 3 of the paper you referenced I think does a good job explaining whats going on:

"At serving time we need to compute the most likely N
classes (videos) in order to choose the top N to present
to the user... Since calibrated
likelihoods from the softmax output layer are not needed
at serving time, the scoring problem reduces to a nearest
neighbor search in the dot product space for which general
purpose libraries can be used."

As far as I can tell, this kind of recommender architecture is only beneficial at the scale of which an organization like youtube is operating, and has more to do with the practicality of organizing computational infrastructure rather than model performance. I'm sure their model performance vs. a more "traditional" architecture is negligible as far as things like map@k are concerned.
EDIT: Found the same question already answered with much more detail than shown here.


What is the difference between horizontal and vertical ensemble?

I think the most highly-referenced source for these terms is Horizontal and Vertical Ensemble with Deep Representation for Classification (Xe, Xu, Chuang 2013). That would be the best place to get a technical answer to your first question. For purposes of searching, you could also look into "Stacking" as a synonym for Vertical ensembling. I will provide some intuition here:

Ensembling is, to over-simplify, adding new content to your data based on results from other or intermediate modeling results. In this case, "horizontal" and "vertical" are references to where in your dataset you add this content. Horizontal ensembling takes the results from one model (such as the predicted classification) and appends it as a new column to your data. In the case of a binary classifier result, this would effectively be adding a column that says "A different model classified this row as a 1 or 0 (A or B, red or green, etc)." The final model would use this information in its own process as a new feature. Vertical ensembling adds new predicted values (as rows, if you want to think about a dataframe paradigm) that are generated by a model. To be clear, Horizontal ensembling adds what some external/interediate model thinks is the right classification while vertical ensembling adds what the model predicts as a value.

Consider a problem that is trying to predict whether the image is "cat" or "dog". Based on the features in the row, a model can say "cat", which gets added to that row when the Horizontal ensemble adds the results of a different classifier. Think of that as an extra vote that the image is of a cat. Or a Vertical ensembling can add new information from that external model for the predicted probabilities of the image being "cat" or "dog", maybe as a tuple: (0.73, 0.27).

Give the article a good read to see when you might prefer one over the other, and the statistical issues that arise in both cases. And, as always, keep the No Free Lunch Theorem in mind.
Ensembling can refer to multiple things. Boosting methods can be considered ensembling, since these algorithms are doing intermediate calculations to see what learners are performing better in order to shift weights in the modeling. These methods are built into the algorithms themselves, so I would look into scikit learn and its ensembling methods to get a sense of what's going on. To your specific concern, however, you can look through Github repos for ensemble learning.


How to count number of word embeddings in Gensim Word2Vec model

Alright, after digging around in methods available to my loaded Word2Vec model I believe the answer is len(model.wv.vectors)...


To convert from string to float in pandas (assuming you want to convert Employees and you loaded the data frame with df), you can use:
df['Employees'].apply(lambda x:float(x))

You have not given enough information about your input and expected output. So let us assume that hospital name or anything for that matter which is the input for your model is nan. You would like to remove it from the dataset because extracting features from 'nan' wouldn't make sense. Apart from that, if they are just other peripheral features, then it might be alright. In that case, if you wish to convert them into blank, then use:
df.replace(np.nan,' ', regex=True)`

Else, if you wish to remove that frame, you can check for nan using this.


The output of Model from Decision Tree and Random Forests are different?

Your df row count is 14,999

Your test data is 33% ~ 4950

So, your y_predict should be (4950,1) i.e. a binary prediction for all test rows.

This is what I am getting when running your code which you have posted here.

import pandas as pd
df = pd.read_csv('https://raw.githubusercontent.com/bhaskoro-muthohar/DataScienceLearning/master/HR_comma_sep.csv')

print(df.size)
# Map salary to 0,1,2
df.salary = df.salary.map({"low": 0, "medium": 1, "high": 2})
# dropping left and sales X for the df, y for the left
X = df.drop(["left", "sales"], axis=1)
y = df["left"]

#splitting the train and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0, stratify=y)
# Train a decision tree.
from sklearn.tree import DecisionTreeClassifier
clftree = DecisionTreeClassifier(max_depth=3)
clftree.fit(X_train, y_train)
y_pred = clftree.predict(X_test)
print(y_pred.size)
from sklearn.ensemble import RandomForestClassifier
# random forest model creation
rfc = RandomForestClassifier()
rfc.fit(X_train,y_train)
# predictions
rfc_predict = rfc.predict(X_test)
print(rfc_predict.size)


Why the Silhouette Score and optimal number of Cluster changes when using 2D and 3D data?

Yes it can happen. In fact it is quite normal since there are different clusters in 2D and different in 3D, since more or less information is added to data (by having more dimensions). This is a by-product of the curse of dimensionality.
Adding as more relevant information as possible would make clusters more close to underlying groups. So 3D would be better than 2D. This is a general observation. There are of course cases where projecting data in a low-dimensionalal manifold is indeed better since it can eliminate noise and/or capture specific attributes better than clustering on all (possibly irrelevant) dimensions (another by-product of the curse of dimensionality).
If the relevant information in your data has low dimensionality but this information is correlated along many dimensions in the original data then a feature extraction method is needed in order to capture the low-dimensional relevant information from original data (eg PCA, ICA ,..).
For some references along this direction see for example:

How to cluster in High Dimensions
An investigation of K-means clustering to high and multi-dimensional biological data
How do I know my k-means clustering algorithm is suffering from the curse of dimensionality?


Is Categorical Crossentropy Loss always bounded in the [0, 1] interval?

You can actually get values from $[0, \infty[$. Consider the cross entropy loss be following formula:

$loss (y_{pred}, y_{true}) = - \log{y_{pred, class_i}}$

, whereas the index $class_i$ states that you only use the output of the $i$th class.

For example think that your true class is $dog$, which is represented by the $3$rd output neuron. On the $3$rd output neuron your network outputs a probability of $10\%$. Your loss is then $-\log{10\%}= 2.3$. Further if your probability is only $0.1\%$ your loss will equal $6.9$ and so on.

Remember that:

$\lim\limits_{x \rightarrow 0}{\log{x}} = -\infty$


Few points that i want to mention :

First of all, Here i think the input u are preparing where I see doubt. The objective is to predict end location. The input shape u prepare is a doubt for me because your sequence_length should be "4" and you have an initial hidden dimension if "1". If I am not wrong then your final share should be (batch_size,sequence_length,hiiden_dimension) = (6,4,1). Because this is what dimension shape LSTM is expecting always.
Second is change your loss function because u want to do predict from more than one class as mentioned like Cafe =0.1, Park =0.2, Shop =0.3, Home=0.4, Movie = 0.5, School=0.6. Chnage your loss function to categorical_crossentropy in fit() function. Also change activation='sigmoid' to activation='softmax'.

I think these changes u will have to do according to your use case. Please feedback my answer.


If you have many available input parameters/features (financial data like balance, depreciations, tax for several years for several companies, the business areas where those companies are working in (e.g. telco, banks, media, etc) and you already have "labelled" data (I mean, you already have the "tax" saving for you previous customers).... well, a good start could be a standard process of feature selection and designing a neural network.


Representing cyclical features as sin/cos components

as both components need to be equally weighted in order for the feature to make sense

That is not the case.

For instance if $\text{sin}(\theta)$ of the cyclical feature is weighted strongly, it means that the original feature has the strongest positive effect on output at $\theta = \frac{\pi}{2}$.

If the two features are weighted equally, then the focus is around $\theta = \frac{\pi}{4}$

In general, you should expect different weights to apply to the different components, depending on which hours of the day maximise the target variable and which minimise it.

  For a gradient boosting model, is it better to force equal weights on the two correlated features, or is it better to let the model figure it out even if it results in different weights on the two components?

Definitely better to let the model figure it out in this case.

Your main concern would be whether a one-hot-encoded representation might be better than a cyclical representation for your problem. One-hot allows for arbitrary relationships to each hour, but adds more dimensions - and so may require more examples of each time. Cyclical has less dimensions, but is more likely to include non-linear effects, if for instance min and max effect on output target variable are not exactly 12 hours apart - thus it may require a more complex model.


How does keras (TF) know how to differentiate my custom loss?

keras will differentiate automatically by looking at the graph of operations you use as part of your custom loss function. Since you are using only operations that keras "knows about" aka that exist as operations in TensorFlow keras will automatically a graph of operations to backpropagate against.

If you however have a custom loss function that uses an operation TF doesn't know, you'd have to write a custom operation so that TF knows how to backpropagate against it


Logistic Regression - Coefficients have p-value more than alpha(0.05)

The p-value summarises a statistical test for a coefficient not to be statistically different from zero. So basically, when the p-value is > 5%, the estimated coefficient can be positive or negative (the confidence intervall includes positive and negative values).

Often this is interpreted in a way that some variable does not make a reliable contribution to a model. This usually is relevant in causal modelling. But this does not mean that the variable does not contribute to a model and should be excluded. Omitted variable bias can be a massive problem!!! (see: Econometric Theory and Methods, Davidson/Mackinnon, Chapter 3.2 for the OLS case - very interesting stuff). Also in joint significance tests, a non-significant variable can still be relevant.

Especially if you are interested in predictions, the p-value is not so much of a problem. In tendency you can say that overspecification of a model is less harmful than underspecification. In doubt, keep the non-significant variable in the model (it may only be a weak predictor).

However, if you have weak predictors you may also look towards Lasso or Ridge-Regression. In this regression types, variables with little or no contribution to prediction are "shrunken". This is a really cool thing to do and might be the first next thing for you to look at: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#log

I like Lasso/Ridge, because model selection is (at least to some extent) data driven in this case.

Also have a look at "An Introduction to Statistical Learning
with Applications in R" (Chapter 6.2). You may find a copy of the book online. The R code examples in the book are really instructive.


I think there are a lot of important soft skills to consider in the Data Science domain.

Here are some of them:

Know for a fact what the goal is, spending a lot of time on data wrangling, models, visualization and reports when it was not all for the specific goal in mind is a waste. Communicating with less technical people is a skill in itself.
Iterate repeatedly with the product owner. Keep making sure you are on the right path.
If the data doesn't tell the story they thought/want tell them it is not the case, be clear in why this is happening, what biases might be playing a role etcetera. Do not apply all kinds of filters or keep changing parameters to get the desired results.

Regarding your second question:

The objective has to be either gotten from the product owner explicitly or derived from a less mathematical objective. An example could be where you need to predict train arrivals based on some features. They want the model to predict as many times as possible within a 10-minute error range. This is relatively explicit.

Sometimes it is less clear than that, they might say we need it as accurate as possible. Then you will have to decide what to optimize, in some cases, this will just be minimizing the MSE but in other cases, other things might make more sense for your case. Usually, this will be clear from the implicit objective and something that you will get better at with more experience. Both implicit and explicit objectives derive from clear communication with the product owner.


What's an efficient way to plot many features for EDA in python?

I would suggest t-SNE (just google it). It helps you to have a general overview on what is gling on with your high-dimensional data. Please note that parameters of t-SNE are pretty sensitive so you need to put some effort to find a good embedding.


Generalised data science approach to astronomical catalogue cross-match

The generalised algorithm I was looking for was a k-dimensional tree, a space-partitioning data structure which is commonly used in nearest neighbour searches. There are implementations in scipy.spatial (KDTree and the faster cKDTree) and in sklearn.neighbours (also called KDTree).

These are the underlying engines of the Astropy and AstroML catalogue cross match methods, though for my specific use case I required the functionality of the k-d tree itself. The KDTree allows one to search in any number of dimensions, though these need to be scaled (by std dev) if they are different properties.

In some preliminary testing I found this to be over 100 times faster than the C3 algorithm I linked above for my use case. It takes around 10 secs to pick, for 85K sources, the nearest neighbour from a catalogue of 1.4M sources (when searching in 4 dimensions: 2D position, flux and size)


Weighted samples in Tensorflow for convolutional neural networks

Same as in case of the weighted classes, you have to account on sample weights in your loss function.

If you use Tensorflow in combination with Keras, you can simply use the sample_weight argument of the model.fit function.

As an alternative, you can implement your own loss function. However,
depending on your setup/framework it can get tricky to determine sample indices from inside the loss function.


Binary classification with time-series features

I would plot the measurements time $t$ and the corresponding measurement 5 to 10 samples for each category. Try to detect some patterns. Possible patterns are the trend (Is the cure growing? Is it linearly? Exponentially?), frequencies of oscillations (Does one category have oscillations with higher magnitude or frequencies? You can use Fast Fourier Transform for this) self similarity of signal (autocorrelation) Then look at means, median, standard deviation, skewness and kurtosis of your signals.

After having extracted all these features I would try to calculate the correlations of your features with the target variable. Then you can eliminate variables that are not very highly correlated with your target variable. In the next step, I would look at the correlations between your features and eliminate the variables that are highly correlated by eliminating the one variable which less correlated with the target variable. Then I would use some classical binary classifiers like discriminant analysis or logistic regression.

If you see that this method will not lead to sufficient results then you should try more sophisticated methods like neural networks/decision trees for the features that you extracted.


How to train a supervised sequence classifier like CRF, if we have to extract start date and end date from a user query in python

I'm not sure that building a model using CRF is the best approach here. It's going to require quite a bit of training data and effort to get it working like you want. Dates are pretty structured in most cases so there are more straightforward approaches to extracting them. Stanford's SUTime library, for instance, does exactly what you're describing in your question. Although is is Java based there is a Python wrapper for it. For example, for the following input:

query = 'I need a desk for tomorrow from 2pm to 3pm'

Using the SUTime Python wrapper, this is the parsed result you'd get:

[
    {
        "end": 26,
        "start": 18,
        "text": "tomorrow",
        "type": "DATE",
        "value": "2016-10-14"
    },
    {
        "end": 42,
        "start": 27,
        "text": "from 2pm to 3pm",
        "type": "DURATION",
        "value": {
            "begin": "T14:00",
            "end": "T15:00"
        }
    }
]


Which Kind of Machine Learning should I use for an Optimization Problem?

You have no existing data where the parameters you mentioned have been tweaked and you then have the resulting profits?

If you don't have any pre-labeled data then you will find it very difficult because the NN will have nothing to learn from.  NNs learn from either the information you give; the inputs and corresponding outputs, or by experimenting with values and then using the result of using those outputs as labels / targets to learn from.


Beyond its use in deep learning, backpropagation has been used in many other areas, ranging from weather forecasting to analyzing numerical stability. In fact, the algorithm has been reinvented several times in different fields. The general, application independent, name is "reverse-mode differentiation" [3].

The modern version of backpropagation, also known as automatic differentiation, was first published by Seppo Linnainmaa [1] in 1970. He used it as a tool for estimating the effects of arithmetic rounding errors on the results of complex expressions [2].

Gerardi Ostrowski discovered and used it some five years earlier in the context of certain process models in chemical engineering [2].

In the sixties Hachtel et al. considered the optimization of electronic
circuits using the costate equation of initial value problems and its discretizations to compute gradients in the reverse mode for explicitly time-dependent problems [2].

Others researchers that discovered it include Bernt Speelpenning. He arrived at the reverse mode via compiler optimization when asked to automatically generate efficient codes for Jacobians of stiff ODEs [2].

More info can be found in: Deep Learning in Neural Networks: An Overview  and in Who invented backpropagation? and references therein.

[1] Who invented backpropagation?

[2] Who Invented the Reverse Mode of Differentiation?

[3] Calculus on Computational Graphs: Backpropagation


Feature Selection for K Nearest Neighbour and Decision Trees

Like others said Good visual split is a good starting point. (To me it seems the f8-f1 is a good starting point.
However you could get better results by transforming the feature set via PCA and using top eigen factors(new combination features) to train.
You've not clarified whether it is a supervised(you know the no. of classes and actual class for some data set), assuming you don't and it is unsupervised, I'd also try an algorithm like dbscan. It is generally fast.
If you want to try the neural network approach try either sompy or neural gas


What is the difference between bootstrapping and cross-validation?

Both cross validation and bootstrapping are resampling methods.

bootstrap resamples with replacement (and usually produces new "surrogate" data sets with the same number of cases as the original data set). Due to the drawing with replacement, a bootstrapped data set may contain multiple instances of the same original cases, and may completely omit other original cases.
cross validation resamples without replacement and thus produces surrogate data sets that are smaller than the original. These data sets are produced in a systematic way so that after a pre-specified number $k$ of surrogate data sets, each of the $n$ original cases has been left out exactly once. This is called k-fold cross validation or leave-x-out cross validation with $x = \frac{n}{k}$, e.g. leave-one-out cross validation omits 1 case for each surrogate set, i.e. $k = n$.
As the name cross validation suggests, its primary purpose is measuring (generalization) performance of a model. On contrast, bootstrapping is primarily used to establish empirical distribution functions for a widespread range of statistics (widespread as in ranging from, say, the variation of the mean to the variation of models in bagged ensemble models).
The leave-one-out analogue of the bootstrap procedure is called jackknifing (and is actually older than bootstrapping).
The bootstrap analogue to cross validation estimates of generalization error is called out-of-bootstrap estimate (because the test cases are those that were left out of the bootstrap resampled training set).

  [cross validation vs. out-of-bootstrap validation] However, I cannot see the main difference between them in terms of performance estimation.

That intuition is correct: in practice there's often not much of a difference between iterated $k$-fold cross validation and out-of-bootstrap. With a similar total number of evaluated surrogate models, total error [of the model prediction error measurement] has been found to be similar, although oob typically has more bias and less variance than the corresponding CV estimates.

There are a number of attempts to reduce oob bias (.632-bootstrap, .632+-bootstrap) but whether they will actually improve the situation depends on the situation at hand.

Literature:

Kohavi, R.: A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection, Mellish, C. S. (ed.) Artificial Intelligence Proceedings 14$^th$ International Joint Conference, 20 -- 25. August 1995, Montréal, Québec, Canada, Morgan Kaufmann, USA, , 1137 - 1145 (1995).
 Kim, J.-H. Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap , Computational Statistics & Data Analysis , 53, 3735 - 3745 (2009). DOI: 10.1016/j.csda.2009.04.009
Beleites, C.; Baumgartner, R.; Bowman, C.; Somorjai, R.; Steiner, G.; Salzer, R. & Sowa, M. G. Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005).

  The only thing I could figure out that in case of bootstrapping one could artificially produce virtually arbitrary number of such subsets while for CV the number of instances is a kind of limit for this.

Yes, there are fewer combinations possible for CV than for bootstrapping. But the limit for CV is probably higher than you are aware of.
For a data set with  $n$ cases and $k$-fold cross validation, you have

CV  $\binom{n}{k}$ combinations without replacement (for k < n that are far more than the $k$ possibilities that are usually evaluated) vs.
bootstrap/oob $\binom{2 n - 1}{n}$ combinations with replacement (which are again far more than the, say, 100 or 1000 surrogate models that are typically evaluated)


Why fitting/training a model can be considered as learning?

When a baby tunes the connections between neurons in his brain until he can recognize a dog and say "this is a dog!" we say that he learned.
Why wouldn't we say the same about neural networks (or other models) that happened to be inside a computer and not in a human brain?


TypeError: object of type 'int' has no len(), LogisticRegression()

RandomizedSearchCV expects you provide dictionary of parameters-values to try, like in your code above:

rand_grid = {'log__C':[1,10,100,1000],
             'log__random_state':42} # values should be in []

Since the idea is to search values which give the best performance, you are expected to provide several values, as a list of values. Though in your case you only provided only one value, it won't raise error as long as it was written as list (inside []).

However it's a bit weird if you only have one value to search. So better you fix those values when you initialize the estimator (i.e LogisticRegression())., e.g:

# you want to see which values perform best for params C and max_iter.
rand_grid = {'log__C':[1,10,100,1000]
             'log__max_iter': [100,200,300]}

# you have fix value for params random_state, penalty and fit_intercept
logReg = LogisticRegression(random_state=42, penalty='l1', fit_intercept=False)

log_rand = RandomizedSearchCV(pipe_opt('log',logReg),
                              rand_grid,n_iter=100,cv=3,
                              random_state=42,n_jobs=-1)


How to randomly split the data set into multiple different sets:(training 70%:validation 10%(optional):testing 20%) in R?

Usually you choose the ratio between training and test data by how many samples you have, for example if you do not have that many you would go to something like 90:10, but with a larger one you can even co sider something like 75:25, although, 80:20 sounds good for your case.

For the random splitting that you are talking about, you should search and learn a lit k-fold cross-validation. It's a method with which you split your data in training and test sets k times in order to validate your model on different training and test sets each time.

About the R implementation i cannot really help you, but im pretty sure it should be quite easy to find a guide on how to do that.

Good luck.


Classification - Divide the interval (0 - 1] to lets say 100 classes and use each class to make a calculation

Correct me if I am wrong but If I understand your question correctly, what you want is a classifier such that classes close to each other (say class 2 and class 3) are preferable to those far away (class 2 and class 99). If this is the case, this problem is called "Ordinal Categorical Classification".

I was working on a similar problem a while ago, and I found this loss function during my research. I ended up not using it so I don't really know how good it works but anyways, hope that helps.


Have you checked the competition's stage?
In there you will see all the active and concluded contests
https://www.kaggle.com/competitions


How to plot/visualize clusters in scikit-learn (sklearn)?

When I run the code you posted, I get three points on my plot:

The "point" at (0, 4) corresponds to X[1] and the "point" at (0, 5) is actually three points, corresponding to X[0], X[2], and X[3]. The point at (5, 5) is the last point in your X array. The data at (0, 4) and (0, 5) belong to one cluster, and the point at (5, 5) is considered noise (plotted in black).

The issue here seems to be that you're trying to run the DBSCAN algorithm on a dataset containing 5 points, with at least 4 points required per cluster (the second argument to the DBSCAN constructor). In the sklearn example, the clustering algorithm is run on a dataset containing 750 points with three distinct centers. Try creating a larger X dataset and running this code again.

You might also want to remove the plt.ylim([0,10]) and plt.xlim([0,10]) lines from the code; they're making it a bit difficult to see the points on the edge of the plot! If you omit the ylim and xlim then matplotlib will automatically determine the plot limits.


Are more classes more favorable than a single combined class?

So what you are interested in (in terms of prediction) is $n+1$ classes, where the last class is $m=1$. This would be my default model. In other words, I don‘t see why you should have $m$ additional outcomes if these outcomes are irrelevant for your task.

For the quality of a prediction, features $X$ matter the most. I‘m not aware that the classes are relevant unlike your outcome (=classes) have an order.

However, ultimately you need to test which model performs best. So by the parsimony principle, start with a simple model and gradually see if adding complexity delivers a better result (by a predefined measure such as accuracy).


How to serialize/pickle a spacy ner model?

Yes - Here is how to pickle in Python:
import pickle

pickle.dump(nlp, open( "nlp.p", "wb" ))


Classify if someone is home based on time

This is an ideal case for feature engineering!

I did this same case for myself using the google takeaway data to predict whether I am at home or at work.

Instead of just using time I extracted the following features:

Work Day --> 1 / 0
Day of the Week
Month
Year
Time

I then trained a random forest classification model to tell me whether I am at home, at work or other place based on those five features.

As a successive step I used this model to actually identify dates where I "moved" or was "on holiday" because of the difference between prediction and actual labels.


CNN for classification giving extreme result probabilities

There are a few different factors involved here. It is difficult to tell, without getting heavily involved, which could be the most important. I will put them in the order I think worth looking at first.

Your data is images from CCTV, so you likely have more than one image from each camera. From your results (reasonable training and CV scores, but bad training), it looks like you are over-fitting. But your CV approach is not spotting this. So I think that the test set is likely to be from a different set of cameras to the training set. In order to properly measure CV therefore, you have to split train/cv by camera - you cannot just use the 0.1 split, because a random split will include images which are correlated with training data and will give you too high an estimate, allowing you to overfit without noticing.

It occurs to me it might just be your data augmentation causing a problem for you here. If you augment first then randomly split to train/CV, then your CV set will contain images very similar to training set, and will get too high a score. You can more easily check and fix this than split by camera, so give it a try.

800 original images is not much to work with. You need to do something about that. Here are some ideas:

Scale down the images. You probably don't need 340x340. Depending on the target object, maybe just 78x78 will do. You can assess this easily enough - scale down and check if you can still differentiate the classes easily by eye.
You don't have enough data to get best quality filters in the lower layers, which will limit the capabilities of the CNN. You might bootstrap from a pre-trained image model. Take a publicly available pre-trained CNN, such as VGG-19, use its weights in convolutional layers as a starting point, put different classifier layers on top and fine-tune your classifier starting from this. This might change the ideal image scale too - you want something that fits the pre-trained CNN.
Augmenting the data, which you have started. You could go a lot further. Take random patches from training examples, possibly flipped horizontally (if this maintains the object class). However, don't augment data used for cross validation, unless your model used for testing also includes augmentation - e.g. if it takes 8 random augmented variants of the test image and returns an average of predictions, then you can do similar for cross-validation.

A 0.1 train/CV split run once on this much data is not going to give you an accurate assessment of the model. You need to run k-fold cross-validation. This is annoying because NNs take a long time to train, but if you want some confidence that you have really found some good parameters, you will need to do this. Remember to split by camera if you can.
The scoring appears to be strongly related to categorical cross-entropy, so you have the right loss function. You should optimise, using cross-validation, to find the model with the lowest loss, not the best accuracy.


You may use Interpolation on the the bins and heights of the HistogramList to produce a smooth PDF for use in ProbabilityDistribution.  With a smooth distribution then you need not worry about increasing or decreasing bin sizes.
For example, with data
SeedRandom[9123]
data = RandomVariate[WeibullDistribution[2, 2], 10^6];

has Histogram
hist = Histogram[data, Automatic, "PDF", PlotRange -> Full]

You have the histogram bins and heights so I'll just use HistogramList to collect them and display with ListStepPlot.
hlData = HistogramList[data, Automatic, "PDF"];
lstep = ListStepPlot[Transpose@{First@hlData, Append[Last@hlData, 0]}, Mesh -> Full]

A smooth InterpolatingFunction can be constructed from the bins and heights with Interpolation.
ifData = Interpolation[Transpose@{First@hlData, Append[Last@hlData, 0]}];
ifPlot = Plot[ifData[x], {x, Sequence @@ First@ifData["Domain"]}, PlotStyle -> Purple]

It can be seen that it matches the bins and heights.
Show[hist, ifPlot, lstep]

We want to use this function as the PDF of a ProbabilityDistribution so that NProbability can be calculated and RandomVariates can be generated.
The function needs to be non-negative over its domain.
NMinimize[ifData[x], {x} ∈ Interval @@ ifData["Domain"]]

{0., {x -> 7.4}}

Minimum value is zero at x equal to 7.4.
Must Integrate to 1 over its domain.
NIntegrate[ifData[x], {x} ∈ Interval @@ ifData["Domain"]] // N

0.996821

The integral is just shy of 1 but we can ask ProbabilityDistribution to "Normalize" the PDF.
dist = ProbabilityDistribution[ifData[x], {x, Sequence @@ First@ifData["Domain"]},
   Method -> "Normalize"];

This distribution can be used in RandomVariate, Probability and other functions of the Random Variables guide.
Probabilities can be calculated
NProbability[0.9 < x < 2.2, x \[Distributed] dist]

0.514336

Pseudo-random numbers generated
RandomVariate[dist, 5]

{0.494373, 1.16545, 2.94366, 4.06116, 1.72519}

Properties like the SurvivalFunction, CDF, and others can be calculated.
Plot[SurvivalFunction[dist, x], {x, Sequence @@ First@ifData["Domain"]},
 PlotStyle -> Purple]

Update
Widening the bins will result in a loss of information so I would not recommend it.  However, if you must then Subdivide the InterpolatingFunction "Domain" into the number of bins you required.
downsamples =
  Function[numbins,
    With[{bins = Subdivide[##, numbins] & @@ First@ifData["Domain"]},
     {
      bins,
      NProbability[#1 < x < #2, x \[Distributed] dist] & @@@
       Partition[bins, 2, 1]
      }]
    ] /@ Range[36, 12, -6];

Addressing your question in the comment; The total probability in each of the 5 cases is still one.
Total[downsamples[[All, 2]], {2}]

{1., 1., 1., 1., 1.}

However, notice the differences in the PDF histograms.
ListStepPlot[
 Transpose[{First@#, Append[Last@#, 0]}] & /@ downsamples,
 Mesh -> Full,
 PlotRange -> All,
 PlotLegends -> StringTemplate["`` bins"] /@ Range[36, 12, -6]]

The initial histogram had 37 bins. The plot above shows how the information is lost as the bin width is widened (number of bins decreased).  I would recommend working with dist instead.
Hope this helps.


Classify pdf files - image approach vs. text approach

Both methods will be beneficial for different cases.

If you think that dependencies in the text are more discriminative of the classes, then NLP apporach. Image approach in this case will need to be super complex to catch this kind of information.

On the other hand layout and position could be very informative and this it could happen is not encoded in text and only image approach can catch this information.

Conclusion. Maybe think about hard-encoding layout and position features and pass them along to NLP model.


Fastest way to relearn machine/deep learning

If you enjoy reading, that's probably the best ML/DL book ever.


"help" decision tree by tying 2 features together

One way is to create new feature by combining A and B by a function and eliminating A and B from dataset. The less features are correlated the better. Then you don't have redundant features.

But actually, it may be enough to use Random Forest. Then the model will have enough trees to figure out that the best split is first B and then A.


Difference between Blocking and Clustering?

As you mentioned their goals are different. In clustering, we try to group data such that they have the same variability. For example, clustering customers of a company into different clusters, somehow, members of each cluster have the same behavior in their buying.

On the other side, in blocking we try to reduce the variability, to record linkage, as an instance.


sklearn package with AttributeError: 'MissingValues' object has no attribute 'to_list'

The problem was with the pandas version. I've used Python3.6 as interpreter and pycharm would only let me install version 0.25 pandas. Now I am on Python3.8 and pandas 1.2.3 and the minimum code example provided by @Oxbowerce works now. I'll be more careful with checking the dependencies in future.
The sketch goes into the epochs now, there was just a minor labelling error:
Traceback (most recent call last):
  File "C:/Users/bobin/PycharmProjects/Mussel/cnn_musseltest.py", line 348, in <module>
    plt.plot(history.history['acc'], "g--", label="Accuracy of training data")
KeyError: 'acc'

Renaming 'acc' to 'accuracy' and val_acc to val_accuracy helped, as stated in the warning log.
WARNING:tensorflow:Early stopping conditioned on metric `acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy


Illustrating the dimensionality reduction done by a classification or regression model

(note: this answer is mid-edit)
There are a number of Machine Learner explainers and diagnostics.
Disclaimers:  (these should increase over time)

I'm not making it exactly reproducible because it would be 2x as long, and its working on being book-like anyway.
This is more about showing the method than going into crazy details.  If you want a deep dive into a nuance, that is a different question.

Let's set up a sample problem.
Mnist is a fair dataset, so let’s first use a random forest to describe it, and then vivisect the learner to understand what, why, and how it works.
Here is my preferred "startup" because if I don't have it, it makes sure I get it.
#list of packages
listOfPackages <- c('dplyr',      #data munging
                    'ggplot2',    #nice plotting
                    'data.table',  #fast read/write
                    'matrixStats', #column standard deviances
                    'h2o',         #decent ML toolbox
                    'keras')       #has mnist data

#if not installed, then install and import
for (i in 1:length(listOfPackages)){
  if(!listOfPackages[i] %in% installed.packages(fields = "Package")){
    install.packages(listOfPackages[i] , dependencies = TRUE)
  }
  require(package = listOfPackages[i], character.only = T, quietly = T)
}
rm(i, listOfPackages)

Here is code for reading mnist using keras:
library(keras)
mnist <- dataset_mnist()
x_train <- mnist$train$x
y_train <- mnist$train$y
x_test <- mnist$test$x
y_test <- mnist$test$y

One has to do some housekeeping:
# Redefine  dimension of train/test inputs
x_train <- array_reshape(x_train, c(nrow(x_train), img_rows*img_cols))
x_test <- array_reshape(x_test, c(nrow(x_test), img_rows*img_cols))

df_train <- data.frame(y=y_train, x_train)
df_test  <- data.frame(y=y_test , x_test)

here is how to process it with a random forest using h2o.ai, assuming its already installed:
# Input image dimensions
img_rows <- 28
img_cols <- 28

#spin up h2o
h2o.init(nthreads = -1)

#move data to h2o
train.hex <- as.h2o(df_train, "train.hex")
test.hex <- as.h2o(df_test, "test.hex")

#prep for random forest
x <- 1:ncol(x_train)
y <- 1
x <- x[-y]

#spin up random forest
myrf <- h2o.randomForest(x, y,
                         training_frame = train.hex,
                         validation_frame = test.hex,
                         ntrees = 150, model_id = "myrf.hex")

Here is how it did.
Confusion matrices:

Here are the train/valid metrics

So what?  So what now?
We have a decent model, and it is crudely compatible with (this) benchmark that says there are things that have less error than it.  Where does it go wrong?
There are about 320 misclassifications in the test dataset, and it is beyond scope to go into each and every one of them.  It looks to be worst at 9, 8, 2, and 3.  Let's look at 8 and 3.
There is a warning given by h2o.ai:
Warning message:
In .h2o.processResponseWarnings(res) :
  Dropping bad and constant columns: [X701, X702, X309, X672, X61, X673, X60, X674, X63, X62, X65, X64, X66, X393, X778, X779, X781, X782, X420, X783, X421, X700, X78, X79, X141, X142, X780, X448, X449, X725, X726, X727, X728, X729, X81, X80, X83, X82, X85, X730, X84, X698, X731, X87, X699, X732, X86, X337, X88, X559, X561, X169, X10, X12, X11, X14, X13, X281, X16, X15, X18, X17, X560, X19, X504, X505, X111, X112, X113, X476, X477, X753, X21, X754, X20, X755, X23, X22, X25, X24, X27, X26, X29, X28, X197, X616, X617, X30, X225, X588, X32, X589, X31, X34, X33, X36, X35, X38, X37, X39, X646, X767, X768, X769, X253, X770, X771, X772, X773, X41, X532, X774, X40, X533, X775, X43, X776, X42, X777, X45, X1, X44, X2, X47, X3, X46, X4, X49, X5, X48, X6, X7, X8, X9, X756, X757, X758, X759, X760, X50, X365, X761, X366, X762, X52, X367, X763, X51, X764, X54, X644, X765, X53, X645, X766, X56, X55, X58, X57, X59].

Sherlock Holmes says (approximately):
Once you eliminate the impossible, whatever remains, no matter how improbable, must [contain] the truth.
Let's remove the impossible from our pixels.
bad_cols <- which(colSds(x_train)==0)
str(bad_cols)

There are 161 of the 784 pixel (columns), or about 21%, that have no information.
Let's see where the RF thinks most of the importance lives.
#pop importance
myimp <- h2o.varimp(myrf)

#sort by relative importance
myimp <- myimp %>% arrange(-relative_importance)
myimp$index <- 1:nrow(myimp)

#plot importances
ggplot(myimp,aes(x=log10(index), y=log10(scaled_importance))) +
  geom_point()

yields this as a plot:

This tells us that, unsurprisingly enough, the majority of the importance is in just a few pixel intensity values.  I like to think of Paretto, the 80/20 rule, Prices law, or Lotka's law.  It also shows that around $10^{2.5}$ steps in, the relationship falls-down, and stops behaving consistent to a single Lotka-style rule.  There could be a "transition in 'physics'" but it is more likely that it is a "cliff" where the data stops living.
Next I want to look at the "kings".  In particular, I want to know how many elements are required to get the same.  I could brute force it, but I don't want to.  When I look at the plot, I see clusters with jumps, and I use those to determine test-sizes to consider.  I see transitions at index values of 10 and 19.
Let's subset and then start with LIME.
(note: this answer is mid-edit)


How to better represent three sets of categorical data?

You shoud put the barcharts next to each other as described in this example.
import numpy as np
import matplotlib.pyplot as plt

N = 5
men_means = (20, 35, 30, 35, 27)
men_std = (2, 3, 4, 1, 2)

ind = np.arange(N)  # the x locations for the groups
width = 0.35       # the width of the bars

fig, ax = plt.subplots()
rects1 = ax.bar(ind, men_means, width, color='r', yerr=men_std)

women_means = (25, 32, 34, 20, 25)
women_std = (3, 5, 2, 3, 3)
rects2 = ax.bar(ind + width, women_means, width, color='y', yerr=women_std)

# add some text for labels, title and axes ticks
ax.set_ylabel('Scores')
ax.set_title('Scores by group and gender')
ax.set_xticks(ind + width / 2)
ax.set_xticklabels(('G1', 'G2', 'G3', 'G4', 'G5'))

ax.legend((rects1[0], rects2[0]), ('Men', 'Women'))

def autolabel(rects):
    """
    Attach a text label above each bar displaying its height
    """
    for rect in rects:
        height = rect.get_height()
        ax.text(rect.get_x() + rect.get_width()/2., 1.05*height,
                '%d' % int(height),
                ha='center', va='bottom')

autolabel(rects1)
autolabel(rects2)

plt.show()


Can I fine-tune BERT, ELMO or XLnet for Seq2Seq neural machine translation?

You can view models like ELMo or BERT to be encoder-only. They can be easily used for classification or sequence tagging, but the tag sequence is typically monotonically aligned with the source sequence. Even though the Transformer layers in BERT or XLNet are in theory capable of arbitrary reordering (which is used in non-autoregressive machine translation models), this is not what BERT or XLNet were trained for and therefore it will be hard to finetune for that.

If at least the vocabulary is the same on both the source and target side, I would recommend pre-trained sequence-to-sequence models: MASS or BART.

If the both the grammar and vocabulary and grammar of the sign language are quite different, maybe using BERT as an encoder and training your own lightweight autoregressive decoder might be the correct way.


How to reduce the GPU consumption size while using Elmo Model?

I understand that you are doing only inference, so you shouldn't need your model to be trainable, and therefore you can set the parameter trainable to False when invoking hub.Module.


What is the reason for the speedup of transformer-xl?

For vanilla Transformer language models (Al Rfou et al), you process [1 2 3 4], predict 5, process [2 3 4 5], predict 6, and repeat.

For a Transformer-XL language model, you process [1 2 3 4], predict [5 6 7 8], predict [9 10 11 12], and repeat. Note that we don't have to reprocess [5 6 7 8] when predicting [9 10 11 12], because the representations can be reused.

https://ai.googleblog.com/2019/01/transformer-xl-unleashing-potential-of.html


How to determine semantic differences in NLP

Out of the box, something like Google's Universal Sentence Encoder (USE) may work for your use-case. Many of the common NLP embedding techniques nowadays work on individual words and so creating sentence-level embeddings means averaging multiple word-level vectors together. USE was built to operate at the sentence level, so you may find it better.
The original paper can be found here:
https://arxiv.org/abs/1803.11175
An example blog post leveraging USE:
https://medium.com/@gaurav5430/universal-sentence-encoding-7d440fd3c7c7


Convolutional neural network fast fourier transform

By transforming both your signal and kernel tensors into frequency space, a convolution becomes a single element-wise multiplication, with no shifting or repeating.

So you can convert your data and kernel into frequencies using FFT, multiply them once then convert back with an inverse FFT. There are some fiddly details about aligning your data first, and correcting for gain caused by the conversion.

If you have a good FFT library, this can be very efficient, but there is overhead cost for running the Fourier transform and its inverse, so your convolution needs to be relatively large before it is worth looking at FFT.

I have explored this a while ago in a Ruby gem called convolver. You can see some of the code for an FFT-based convolution here and the project includes unit tests that prove that direct convolution gets same numerical results as FFT-based convolution. There is also code that attempts to estimate when it would be more efficient to calculate convolutions directly by repeated multiplications or use FFT-based solution (that is rough and ready guesswork though, and implementation-dependent).


How can I explain this chart showing 5-days moving average?

Yes it makes sense, a moving average makes the curve "smoother" in the sense that it's less sensitive to short variations. This usually makes it easier to observe the general tendency.
You could also try different time periods for the average, e.g. 10 days or 15 days.
It looks to me like there's a moderate increase trend in your data, but the variations are important and the time window is short, so it's too early to be sure. You could apply linear regression to confirm the increase.


You want to implement gradient descent on a multivariate linear regression.

You can find an implementation in python here :
https://towardsdatascience.com/implementation-of-multi-variate-linear-regression-in-python-using-gradient-descent-optimization-b02f386425b9


The pyimagesearch face recognition implementation used knn classifier to recognise the faces. I have worked with both svm and knn classifiers, from my experience you can do a couple of thinks to improve the face recognition performance.
1. use svm with higher C value (1 to 10) over knn classifier.
2. Maintain relatively equal number of images in all classes


Data science is a multi discipline subject.
In addition to math you need some programming skills and ideally - understanding of the domain where you are going to solve problems.

You can start from online-courses and some introduction literature.
Look at the answers at current site:

Data Science Sertifications
Books about Data Science
Data Science vs Data Mining
Data Science is a trend or a long term concept
Ph.D. in Data Science?
Statistics + Computer Science = Data Science?
Starting career as Data Scientist, is Software Engineering experience required?
Career switch to Big Data Analytics
And more about career...

I can't resist from placing this illustration for the millionth time:


How are boosted decision stumps different from a decision tree?

The decision boundary in (4) from your example is already different from a decision tree because a decision tree would not have the orange piece in the top right corner.

After step (1), a decision tree would only operate on the bottom orange part since the top blue part is already perfectly separated. The top blue part would be left unchanged.

The boosted stumps, however,  operate (as you mentioned) on the full dataset again, which can lead to different results.


The result that all values are equal to -1 is to be expected, since you're checking if the indices from your training set occur in your validation set, which by definition is not the case since your train and validation sets should not have the same observations. If you want to add the 0/1 indicator to you original dataset you have to use X instead of X_train:
split_index = [0 if x in X_val.index else -1 for x in X.index]

This will set the index to 0 if it occurs in X_val and -1 for all other cases.


In backpropagation, scale is also important?

In the process of backpropagation,first of all we feed forward the model.Each node receives the input from its previous layer,calculate the weighted suma and the apply the activation function to the weighted sum and pass this result to next layer as input. When we got the result.

Lets say we are classifying animals images,the each output node will correspond to the different category more or less and node with the largest output will be considered as the output of the model.

Then the loss is calculated on the result,lets say we are using SGD for the loss function.for SGD the loss is calculated as "d(loss)/d(weights)".Here Backpropagation comes into play.

Since the output is calculated with the weight and input,to change the result we need to change these two values(weight and input).Since we can't change the input ,we will change the weights.These weights are updates as

                "new weight=old weight - (gradient*learning_rate)"

Here is the role of scale

For sake of simplicity,lets say for a particular connection weight is initialized to 0.833 and its optimal value is 0.437

Also,Gradient w.r.t to a particular weight in a network is the product of some components that resides later in the network(These components are gradients w.r.t. to weights)
Therefore,the earlier the weight ,more number of components will be required to calculate its gradients.

1.Vanishing Gradient:

If we set our learning rate(or considered as scale) to 0.0001.

                              "gradient*learning_rate"

This value will be very small.
It will take a lot of time to reach the optimal state and at some point will be stuck in a point where the there is almost no change in the weight.

2.Exploding Gradient:

If we set our learning rate(or considered as scale) to 0.01.

                              "gradient*learning_rate"

The scale will be larger enough to reach the optimal value for weight and therefore the optimal value will be skipped.
for simplicity lets say gradient is 1

                "new weight=old weight - (gradient*learning_rate)"

new weight=0.833-0.01=0.823

new weight=0.823-0.01=0.813

.

.

.

new weight=0.453-0.01=0.443

new weight=0.443-0.01=0.433

new weight=0.433-0.01=0.423

It can be seen that the optimal value is skipped.
That is why the scale is important and should be used carefully.


Random Forest, Duplicating Data increases Accuracy. Why?

Without further details, my guess is that your dataset is unbalanced (not the same number of samples in both classes), so that duplicating the data increases the # samples in the most represented class. Depending on how you configured the Random Forest, it may aim at predicting correctly the most represented class and have less concern about the under represented one. In this case, by duplicating the dataset, you put even more pressure on predicting the most represented class and that's what it does, thus increasing accuracy.

As a note, I believe this would not happen if you were training on a dataset and testing on a different one (samples of the test set, on which you assess performance, should be different from the ones in the train set).


Finding the perfect algorithm for realtime optimizing of content

This sounds like a classic use for a contextual bandit solver.

In essence you can run a simple online model (pretty much any regression model, or even a simple classifier like logistic regression if your reward signal is binary success/fail such as in your case) that learns to associate your demographic data with expected reward from each possible action - for you the reward can simply be 1 for a share link created or 0 for no share link.

Whilst the model is learning, you select the next action according to predicted reward from the model. There are choices between different workable strategies. For instance you could use an $\epsilon$-greedy approach: Pick the action with maximum predicted expected reward (or randomly choose between shared maximum values), but sometimes - with probability $\epsilon$ - you choose random content. There are other approaches and options that you can discover by researching contextual bandits and the simpler multi-armed bandit problems.

As an example, you could use a logistic regression model to predict expected reward from user demographics, with one such model per possible action. For a version that picks evenly to start, but prefers items that have been shared more over time, you can use a Boltzmann distribution (also called Gibbs distribution) using the predicted rewards as the inverse "energies" for the actions, and lowering the temperature as you collect more data. You can also initialise the weights of your model to predict a small but optimistic positive reward to start with to encourage early exploration. Whenever a user views your page, you pick the action to take based on the predicted rewards, and afterwards take the user response (share or not share) as feedback to update the one model associated with that action.

In the above example, the logistic regression learning rate, temperature scheme and starting reward are hyper-parameters of your model, and you use them to trade off responsiveness to individual events versus long-term accuracy for selecting the best action.


issue with early-stopping on f1 score with imbalanced data

F1 is based on hard classification; if the probability scores are hovering near the threshold, then the classifications may be flopping a lot, leading to unstable F1 scores.

A low F1 score is not too surprising in the presence of such imbalance; the default cutoff of 0.5 will often lead to high recall but low precision.


Is k-means with Mahalanobis a valid option for clustering?

It doesnt feel right cause there are convergence problems. See here

It has tendency to fail. With kmeans and euclidian distance you have some really nice mathematical properties and you can gurantee convergence.


Does the model(best fitting line/curve) changes when the training data is changed in the cross validation?

Your intuition that the model changes depending on the data input is right. What changes is namely the parameters, e.g. weights if you are performing a regression analysis.
Weights change primarily when you are training the data. You can reliably say that whenever you train the model it changes (namely its parameters) .
During cross validation, you split your datasets into an arbitrary number of subsets. You use some of them as training sets, while you use the rest as validation sets, and measure the error.
You repeat this through and through and you measure error of each. In each iteration when you choose different subsets (kth iteration uses different test-validation sets than the k+1th ones), you do change the model as you are re-training, (however this should not be important as you are doing this to have an approximation on how the model will perform with new data). Finally, you'll have an estimation on the model's error, (but you can use this process to choose features with most predictive power, if you are performing cross validation for feature selection). Edit: It is however important noting that cross validation is used primarily for getting an estimation on how the model would perform in general, not for training test selection.
This goes for numerical models, but I do think they apply to decision trees and other probabilistic models as well, because this process would change information metrics that are given to the model as an input (e.g. split criteria of the decision tree).


How to use live data to improve a existing model?

Suppose you have a model that has been trained on $N$ data over $E$ epochs. This means that the model has seen each of the $N$ examples, $E$ times.

Now say you got $M$ more training data. Normally you would want to train the new ones for $E$ epochs as well.

However if $N$ and $M$ don't come from the same underlying distribution (or don't represent it adequately), this would result in the model "forgetting" the first $N$ examples and "paying more attention" to the latter $M$ ones.

You could try training your model for $<E$ epochs, so that it learns the latter but doesn't forget the former, but that is purely empirical and very hard to achieve in practice.

You can a few things to avoid this:

Retrain your whole model using both the $N+M$ examples (which you would shuffle). This would require a new complete training of the model on regular occasions and would be increasingly difficult to train (due to the ever increasing size of the training data). This is a very inefficient solution and wouldn't work for any on-line training application
Make use of a model that supports on-line training. Some algorithms support incremental (on-line) training, without you needing to retrain the whole thing. A scikit-learn comparison is available.
Customize an algorithm so that is has the desired effect. For example, you could train a linear SVM incrementally, with a large regularization penalty and an SGD classifier. This is discussed in more detail for scikit-learn here.


Why is predicted rainfall by LSTM coming negative for some data points?

Currently your model is not restricted from guessing any number. In this case, you may want to use something like a ReLU activation function to restrict your output domain.

The ReLU activation needs to be added to the last layer of your model. If you add it to an intermediate layer but not the last layer then your model can still output negative numbers, e.g. in the case of the last layer having a negative weight.

Note this is not really a new idea. There are strong similarities between logistic regression and what we’re doing here. In both cases we want to limit the possible guesses, in your case to non-negative numbers and in logistic regression‘s case to [0, 1].