Would Deep Q Learning work for a finite horizon problem?

DQN solves optimal control problems for maximising average reward. Although it typically uses discounted reward, the discount factor is not part of the setting - instead it is part of the solution hyperparameters, and usually set quite high e.g. 0.99 - when using function approximators.

The TD target used in DQN is a problem for you:

$$G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'}Q(S_{t+1},a')$$

as it relies on a Bellman equation that no longer holds for the given value functions. In your case, there does not seem to be any way to express the TD target for time $t$ by referencing other Q values, as you would need to then subtract future rewards, which is very ungainly. Instead, you could use a simple truncated Monte Carlo return

$$G_{t:t+3} = R_{t+1} + R_{t+2} + R_{t+3}$$

There are a few different ways you could do this, but the simplest and closest to DQN would IMO be to:

Store trajectories in order in the experience replay table
For each item in the training mini-batch:

Pick a start state/action pair to assess $s_t, a_t$ randomly from replay table
Check that $a_{t+1}$ and $a_{t+2}$ are maximising actions in your current target policy for $s_{t+1}$ and $s_{t+2}$, reject the sample if not (note you don't need to check or reject $a_t$, and that is how your code learns about exploratory actions)
Calculate the TD target, $g_t = r_{t+1} + r_{t+2} + r_{t+3}$
Your training data for that example is $s_t, a_t, g_t$

The checking for maximising action parts could be quite slow, so you might prefer to simplify the approach and not use off-policy. Alternatively, if $epsilon$ is low enough you could just store the three step returns directly in the experience replay table (wait until you have the data from $t+3$ before storing data for $t$) and ignore the fact that some returns are from exploratory actions, thus noisy/biased . . . this approach is used in n-step returns in DQN "Rainbow" version and works well enough in practice on the Atari problems despite being on shaky theoretical ground.

Note I am using the convention $s_t, a_t, r_{t+1}, s_{t+1}$ to represent a step in the trajectory, whilst in the question you appear to be using $s_t, a_t, r_t, s_{t+1}$ with a different reward index. You will need to convert back if you want to stick with your convention.


XGB custom objective function - small change to default regression squared error objective function

You can find a good explanation and example on creating a custom objective function here:
https://xgboost.readthedocs.io/en/latest/tutorials/custom_metric_obj.html
https://github.com/dmlc/xgboost/blob/master/demo/guide-python/custom_objective.py
https://github.com/dmlc/xgboost/blob/master/demo/guide-python/custom_rmsle.py
Original XGB code repo is here, you may have to do some digging to find the code for your objective function:
https://github.com/dmlc/xgboost


TDIDT stands for "top-down induction of decision trees"; I haven't found evidence that it refers to a specific algorithm, rather just to the greedy top-down construction method.  Therefore (seemingly) all the other algorithms you mention are implementations of TDIDT.  The first iteration is due to Hunt, the "Concept Learning System" in 1966.

ID3 is due to Quinlan in 1979, improving upon the CLS.  (Fun fact: it was originally designed to tackle the problem of deciding winnability of King-rook vs king-knight chess endgames.)  This was further improved to C4.5, then to C5.0.  This branch only works for classification.

CART ("classification and regression trees") was developed roughly in parallel with ID3, by Breiman, Friedman, Stone and Olshen in 1984.  As the name suggests, this branch allows prediction of continuous variables.

The main differences between branches is how splits are determined, using different metrics.  Early on it seems there were differences in handling missing data and such, but these appear to have mostly been included in all the more recent iterations.

Of course, there have been numerous other improvements or variations.  Some (historical?) tree building algorithms generate nonbinary trees, some make splits on linear combinations of features instead of just one at a time, some try to reduce the greedy nature of the algorithm by looking ahead, some produce regression models in each leaf instead of constant functions, ...

https://en.wikipedia.org/wiki/Decision_tree_learning#Decision_tree_types
W-Y Loh, "A Brief History of Classification and Regression Trees" (slides)
Lecture slides (Aida Nordman?)
Quinlan's 1986 paper
similar question on Quora
similar SO question
C5.0 introduction


From logistic regression to XGBoost - selecting features to run the model with

One of the main advantages of tree-based methods is that "high correlation" is not so much of a problem (when compared to linear models). Another advantage is that there is no parametric assumtion behind the model. Thus you have a good chance that some of the variables you dropped to do the Logit (high correlation/u-shaped) can be used with boosting.
WRT to raw levels and/or low representativeness, it could be that some features are actually useful in the boosting process. I would at least give it a try. Check the feature importance to exclude features with little predictive power.
Missings carry no information, so exclude features with lot of missings. Bossting essentially has the same problems as Logit here. Same is true for features which are excluded by expert advice. However, you could still see if some of the features help. This is a matter of testing.
Note that xgboost requires a matrix in which factors are encoded as dummies. This is highly relevant when you work with R.
You can add early stopping to boost until no more progress is made.
Regarding your list, my feeling (not knowing the data in detail) would be...
Can be used with xgboost:

lack of monotonic trend - for u-shaped variables after attempts at coarse classing;
high correlation (>70%) - on raw level or after binning;

Worth a try:

low GINI / Information Value - on raw level or after binning;
low representativeness - assessed through population stability index, PSI;

Exclude:

business logic / expert judgement.
automated data audit (through internal tool) - i.e. detected excessive number of missings, or incredibly low variance, etc.;

Some ideas for the model:
There are many hyperparameters that can/must be tuned. Make sure you pick the right "Learning Task Parameters" for your problem (likely a right skewed distribution). Have a look at the docs, there are some good hints. Also try regulation (alpha, lambda). PLay with max_depth (usually between 5 and 8 work well). Too small trees do not fit well, too large trees overfit. Look at subsample to fight overfitting by making the tree booster "stochastic". Also look at the learning rate eta. Often a low(er) learning rate works find with large(er) data.
You can also add custom losses which can be helpful. I played around with that in R and you can find some code (Python is not so different) here: https://github.com/Bixi81/R-ml
As an alternative to xgboost, you could also look at LightGBM https://github.com/Bixi81/Python-ml/blob/master/boosting_regression_boston.py


What should I use as training data for base (level 1) classifiers in ensembling?

You can do that, but your model will not generalize well. You should not use base-model predictions from data, which were used to fit the base model.
Thus, you have to get the base model predictions for the training data using cross-validation. This is called "model stacking".

This page has a good explanation:

Split your training data into subsets, predict the target for each subset using all other subsets.

Fit the base model on the whole training data and predict the target for the test set.

Do this for multiple base models. Now you have train and test set predictions for each base model. In this example we have two base models:

Fit an ensemble model on the base training predictions and evaluate the performance on the base test predictions.


Reading values from a column into a variable and then correlating using Python


Is it true more CPU core is better for deep learning?

It is better to go with a good GPU than CPU. Most of the time you will train your model on a GPU which gives you around 100x speed boost compared to CPU. So i guess your first config is good enough for normal DL task.


How do I forecast sales data down to the individual item?

Do you want to forecast sales by day over a series of days? That would be more like ARIMA. Do you want to forecast "How many widgets will we sell in the next month"? That would be more like a regression problem.

As an aside, if the retailer you're working with has a large assortment of products that changes over time (e.g. seasonal clothing) then you should also look into forecasting by product category and not an individual product. It will be much more accurate.


Trying to analyze market data in R, but companies continually show very low average growth over large time intervals

This simply has to do with the fact that returns are compounding, i.e. the daily percentage changes are cumulative. If a stock gets has a daily percentage change of 1% for a single year (trading for 252 days), and therefore also an average change of 1% the total return over this year will not be equal to 1%. The actual return is equal to $(1 + 0.01) ^{252} - 1 \approx 11.274$, meaning around 1127.4%, which is much larger than the average daily change of 1%. To get the correct total return over a specific period you have to multiply the percentage change for each period, which in your case is daily.


Well, there are a few ways to do the job. Here are some I thought of:

Scatterplots with noise:
Normally, if you try to use a scatter plot to plot two categorical features, you would just  get a few points, each one containing a lot of instances from the data. So, to get a sense of how many there really are in each point, we can add some random noise to each instance:

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

# This is to encode the data into numbers that can be used in our scatterplot
from sklearn.preprocessing import OrdinalEncoder
ord_enc = OrdinalEncoder()
enc_df = pd.DataFrame(ord_enc.fit_transform(df), columns=list(df.columns))
categories = pd.DataFrame(np.array(ord_enc.categories_).transpose(), columns=list(df.columns))

# Generate the random noise
xnoise, ynoise = np.random.random(len(df))/2, np.random.random(len(df))/2 # The noise is in the range 0 to 0.5

# Plot the scatterplot
plt.scatter(enc_df["Playing_Role"]+xnoise, enc_df["Bought_By"]+ynoise, alpha=0.5)
# You can also set xticks and yticks to be your category names:
plt.xticks([0.25, 1.25, 2.25], categories["Playing_Role"]) # The reason the xticks start at 0.25
# and go up in increments of 1 is because the center of the noise will be around 0.25 and ordinal
# encoded labels go up in increments of 1.
plt.yticks([0.25, 1.25, 2.25], categories["Bought_By"]) # This has the same reason explained for xticks

# Extra unnecessary styling...
plt.grid()
sns.despine(left=True, bottom=True)

2. Scatterplots with noise and hues:
Instead of having both axis being feature we can have the $x$ axis be one feature and
the $y$ axis be random noise. Then, to incorporate the other feature, we can "colour
in" instances based on the other feature:
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

# Explained in approach 1
from sklearn.preprocessing import OrdinalEncoder
ord_enc = OrdinalEncoder()
enc_df = pd.DataFrame(ord_enc.fit_transform(df), columns=list(df.columns))
categories = pd.DataFrame(np.array(ord_enc.categories_).transpose(), columns=list(df.columns))

xnoise, ynoise = np.random.random(len(df))/2, np.random.random(len(df))/2

sns.relplot(x=enc_df["Playing_Role"]+xnoise, y=ynoise, hue=df["Bought_By"]) # Notice how for hue
# we use the original dataframe with labels instead of numbers.
# We can also set the x axis to be our categories
plt.xticks([0.25, 1.25, 2.25], categories["Playing_Role"]) # Explained in approach 1

# Extra unnecessary styling...
plt.yticks([])
sns.despine(left=True)

Catplots with hues:
Finally, we can use catplots, and colour in fractions of it based on the other feature:

import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

sns.histplot(binwidth=0.5, x="Playing_Role", hue="Bought_By", data=df, stat="count", multiple="stack")


The only difference is when you need to carry out certain operations across the channel axis such as BatchNormalisation. If you look at https://keras.io/layers/normalization/, one of the parameters for batch normalisation is as such:

  axis: Integer, the axis that should be normalized (typically the
  features axis). For instance, after a Conv2D layer with
  data_format="channels_first", set axis=1 in BatchNormalization

You need to provide the axis as the model needs to know to carry out normalisation across which axis. Other than that, it doesn't make a difference in training.


What to use pretrained models (Glove) or train my own model?

Usually, I would test my models in both scenarios: with pre-trained word embeddings (GloVe, Word2Vec, etc.) and with my own text, giving that I have a reasonably large data set.

The difference will be that the pre-trained models are more general in context. They are trained with really large data sets, thus capturing information and relationships (semantic, syntax) in a more generic way. The famous example of Mikolov's paper where linear combinations of two words such as city and Paris gives a result near to France in a high dimensional vector. In your own data set, you might capture some different relationships, as an example I could think in a situation where Paris is a character name (I am giving an extreme example for sake of clarity, hope it workds) and then it will be closer to other characters in the embeddings, etc.

To conclude, I have read some papers where word embeddings from specific events (in this case was disaster management) did not work better than the general ones for topic classification. I have tried myself with the same task in a large Portuguese data set and the results were better with my own embeddings (Actually, SVM beats all Deep Learning algorithms in this case) and worked better. Maybe someone with more expertise can give a more detailed answer, but my advice is: try with both.


Why non-differentiable regularization lead to setting coefficients to 0?

Look at the penalty terms in linear Ridge and Lasso regression:

Ridge (L2):

Lasso (L1):

Note the absolute value (L1 norm) in the Lasso penalty compared to the squared value (L2 norm) in the Ridge penalty.

In Introduction to Statistical Learning (Ch. 6.2.2) it reads:
"As with ridge regression, the lasso shrinks the coefficient estimates towards zero. However, in the case of the lasso, the L1 penalty has the effect of forcing some of the coefficient estimates to be exactly equal to zero when the tuning parameter λ is sufficiently large. Hence, much like best subset selection, the lasso performs variable selection."

http://www-bcf.usc.edu/~gareth/ISL/


Can we generate huge dataset with Generative Adversarial Networks

This is unlikely to add much beyond your direct data collection efforts.

The quality of current GAN outputs (as of 2017) will not be high enough. The images produced by a GAN are typically small and can have unusual/ambiguous details and odd distortions. In the paper you linked, the images generated by the system from a sentence have believable blocks of colour given the subject matter, but without the sentence priming you what to expect most of them are not recognisable as any specific subject.

GANs with a less ambitious purpose than generating images from sentences (which is despite my criticism above, a truly remarkable feat IMO) should produce closer to photo-realistic images. But their scope will be less and probably not include your desired image type. Also, typically the output size is small e.g. 64x64 or 128x128*, and there are still enough distortions and ambiguities that original ground truth photos would be far preferable.

The GAN is itself limited by training library available - it will not do well if you attempt to generate images outside of the scope of its training data. The results shown in the research paper of course focus on the domain supplied by the training data. But you cannot just feed any sentence into this model and expect a result that would be useful elsewhere.

If you find a GAN that has been trained on a suitable data set for your problem, then you are most likely better off trying to source the same data directly for your project.

If you are facing a problem with limited ground truth data, then maybe a better approach to using a GAN would be to use a pre-trained classifier such as VGG-19 or Inception v5, replace the last few fully-connected layers, and fine tune it on your data. Here is an example of doing that using Keras library in Python - other examples can be found with searches like "fine tune CNN image classifier".

* State-of-the art GANs have got better since I posted this answer. A research team at Nvidia has had remarkable success creating 1024x1024 photo-realistic images. However, this does not change the other points in my answer. GANs are not a reliable source of images for image classification tasks, except maybe for sub-tasks of whatever the GAN has already been trained on and is able to generate conditionally (or maybe more trivially, to provide source data for "other" categories in classifiers).


Performance gain of GPU when learning DNNs

3 years ago the rule of thumb was: About 15 times faster

Your CPU does 113 GFlops on float operations (source) and your GPU does 3 Tflops (source).

My bet: somewhere between 15 and 30 times faster


Custom thresholds on categorical classification

What you are talking about sounds a lot like meta-learning (use outputs of several models and use another model to train on that predictions). You can read through more detailed example here (link to machinelearningmastery)
Should you do it for your binary classification task - no. (this in not a strict "no", but generally you should not do it given your task can really benefit from such approach, but this has little to do with classification threshold)
About threshold search, you can check this thread with quite good answers:
https://stats.stackexchange.com/a/386433/232020
From your question, it sounds like you are dealing with the binary dataset, not sure what you mean by  Unfortunately, the method seems not to be scaled on categorical classification.


Are there free cloud services to train machine learning models?

There are no unlimited free services*, but some have starting credit or free offers on initial signup. Here are some suggested to date:

AWS: If specifically deep learning on a large data set, then probably AWS is out - their free offer does not cover machines with enough processing power to tackle deep learning projects.
Google Cloud might do, the starting credit offer is good enough to do a little deep learning (for maybe a couple of weeks), although they have signup and tax restrictions.
Azure have a free tier with limited processing and storage options.

Most free offerings appear to follow the "Freemium" model - give you limited service that you can learn to use and maybe like. However not enough to use heavily (for e.g. training an image recogniser or NLP model from scratch) unless you are willing to pay.

This best advice is to shop around for a best starting offer and best price. A review of services is not suitable here, as it will get out of date quickly and not a good use of Stack Exchange. But you can find similar questions on Quora and other sites - your best bet is to do a web search for "cloud compute services for deep learning" or similar and expect to spend some time comparing notes. A few specialist deep learning services have popped up recently such as Nimbix or FloydHub, and there are also the big players such as Azure, AWS, Google Cloud.

You won't find anything completely free and unencumbered, and if you want to do this routinely and have time to build and maintain hardware then it is cheaper to buy your own equipment in the long run - at least at a personal level.

To decide whether to pay for cloud or build your own, then consider a typical price for a cloud machine suitable for performing deep learning at around \$1 per hour (prices do vary a lot though, and it is worth shopping around, if only to find a spec that matches your problem). There may be additional fees for storage and data transfer. Compare that to pre-built deep learning machines costing from \$2000, or building your own for \$1000 - such machines might not be 100% comparable, but if you are working by yourself then the payback point is going to be after only a few months use. Although don't forget the electricity costs - a powerful machine can draw 0.5kW whilst being heavily used, so this adds up to more than you might expect.

The advantages of cloud computing are that someone else does the maintenance work and takes on the risk of hardware failure. These are valuable services, and priced accordingly.

* But see Jay Speidall's answer about Google's colab service, which appears to be free to use, but may have some T&C limitations which may affect you (for instance I doubt they will be happy for you to run content production of Deep Dream or Style Transfer on it)


Binary classification for variable outputs

Actually what you need is a linear regression model and not a binary classifier. You can trying to predict continuous variables using a classifier. So we need to build a regression model which predicts continuous variables in a range i.e from -1 to 1.

You can do this with some changes in your code.

model = Sequential()
model.add(Dense(30, input_dim=60, activation='relu'))
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])

So basically we maintain a linear activation function in the output layer for regression.

Tip:

Instead of accuracy as the metric, use mae or mean absolute error to monitor the model's progress.


Import data from google drive to Kaggle Kernel

Go to your Google Drive and right click the dataset you want to upload to Kaggle.
Generate the shareable link and the code that comes after https://drive.google.com/open?id=, it should be a long code (like 3tlxSE__5eL3q7Zb31M3CEKHlieGWZYgM)
Insert the long code at the end of https://drive.google.com/uc?export=download&id=
Copy that link. Should look something like https://drive.google.com/uc?export=download&id=3tlxSE__5eL3q7Zb31M3CEKHlieGWZYgM
When uploading a dataset to Kaggle, select to add a file from remote URL.
Insert copied URL.
Done!


approach for predicting machine failure using maintenance history

Sequence labeling seems like a good fit for your problem. Your features could be the frequency of each kind of basic-msg in the past 4 runs (but maybe it would work better normalized by unit of time).

There are a good few libraries for Conditional Random Fields for example.


How to use k-means outputs (extracted features) as SVM inputs?

'Prediction' of k-means algorithm for each observation is just the corresponding centroid. So you can take vector of predicted centroids and use it as a categorical feature (maybe one-hot encoded).

But it is just one feature. With little coding you can do better. For example, you can find for each sample its distance to each of $k$ cluster center, and so create $k$ new features. A Python example:

from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.svm import SVC
import numpy as np
iris =  load_iris()
X = iris['data']
y = iris['target']
kmeans = KMeans(n_clusters=6).fit(X)
distances = np.column_stack([np.sum((X - center)**2, axis=1)**0.5 for center in kmeans.cluster_centers_])
svm = SVC().fit(distances, y)

Another (and maybe simpler way) is to fit a gaussian mixture model (e.g. by scikit-learn). It is similar to k-means, but for each observation produces a probability distribution over clusters, instead of a single cluster label. These vectors of predicted cluster probabilities may be used as features as well.

from sklearn.mixture import GaussianMixture
gmm = GaussianMixture(n_components=6).fit(X)
proba = gmm.predict_proba(X)
svm2 = SVC().fit(proba, y)


I didn't watch the linked video but based on your explanations: yes, your understanding is correct.
A common confusion is to assume that cross-validation is similar to a regular training stage and therefore produces a model. This assumption is wrong: CV includes repeated training/testing for the purpose of evaluating the method/parameters. From this understanding it follows that:

for doing model training using k fold CV, we re-train on the entire dataset after the end of the CV loop and that is the final model.

Yes, since we want to obtain the final model as accurate as possible so we should use all the data. In this case the CV has been used to calculate a good estimate of the performance.

We do not select any model from inside the CV loop if the idea of doing CV training is to check the accuracy of the ML algorithm on the entire dataset.

Correct, otherwise there's no point using CV.

However, if we have multiple ML algorithms say random forest, neural network, SVM inside the CV loop then we select the algorithm with the highest accuracy.

Any case where multiple methods and/or parameters are being evaluated is a bit more complex than the regular case of a single method: evaluating multiple systems is by itself an additional layer of training, in the sense that we select some parameters (typically the best model) based on the data. This means that the selection itself is based on the whole data used in the CV stage, so the CV performance of the best model is akin to a performance obtained on a training set. This is why one needs another test set (or nested CV) in order to obtain the final performance of the model. An intuitive way to understand this is to imagine evaluating say millions of models with CV: the only way to know if the best performance is due to chance or not is to evaluate the corresponding model on some fresh test set.
Note: the case of combining the outputs of all the models is a different story, since this boils down to a single meta-model.


Computing weights in batch gradient descent

What you need to use is mini-batch gradient descent or stochastic gradient descent. You will need to shuffle your samples and make draws from it of the batch size you are aiming for, also you will have to make sure all of the samples in your data are included which will constitute of one epoch. Train for a few epochs depending on the data. Here is a good blog post on mini-batch blog post.


I suspect that the problem has more to do with your data than with your algorithms. My recommendation is to spend some time studying your data and ensuring that it is a robust representation of the kinds of problems you're expecting to solve. If possible, come up with a way to generate extra data. Given the fact that you already have many permutations, you could perhaps write a script to create additional permutations by modifying existing samples with rules that you know.


How do autoencoders reconstruct images/color?

Autoencoders ( AE ) are obviously unsupervised learning algorithms as they try to reconstruct the input or something similar to their input. To get a better understanding, we may use autoencoder to colourizing grayscale images. Given an input grayscale image, the AE will predict coloured images.

I have tried one such AE here.

  Autoencoders learn to represent the given grayscale image into a latent ( compressed ) form. The encoder does this job. Now when the image has been brought down to its latent form, the decoder reconstructs the coloured image using that latent representation.

Due to this ability of autoencoders, they are used in image colorization, neural implanting and even generating images ( VAEs ). See this video.

Image colourization is not pixel-level operation. We need a smarter system which can literally observe the surroundings and assign a colour to the pixel. If we are training one such model on images of various landscapes ( sky and green croplands, for instance ) the most probable colour for green ( grass ) will be blue ( sky ).

Meaning, image colourization AEs extract spatial features which may be responsible for a colour change in the target image. A model trained on face images knows that there is a dark-coloured region above the eyes ( hair ). Many such features are learnt by even more complex models which give excellent results.

  Autoencoders store the knowledge required to colourize an image by learning features from grayscale images. After enough they tend to know which region has to colourized with which colours.


How to do backward features elimination when considering interactions between them

My first question is : for example, if the highest p value is for the X1X2 feature, is it okay to eliminate this feature even when X1 and X2 can be statistically significant ?

  Of course, the interaction can have no information about the target. Per example if the problem is perfectly defined by X1 and X2. The interaction $X_1 \cdot X_2$ won't add nothing to the model.

My second question : in the case when all the interaction of some feature have a p value greater than 0.05 in the first iteration, Could I eliminate this feature and all the interactions ?

  I would try a more experimental approach of removing them only if they don't improve the model accuracy rather than having a low P-Value.

As a further reccomendation I would reccomend sklearn.


Python library that can compute the confusion matrix for multi-label classification

Also take a look at scikit-multilearn. It is a very good library that extends sklearn for multi-label learning. However, I'm not sure how the confusion matrix works for multi-label problems...

This guy claims he has solved it.


Surface Pro 6 vs Macbook Pro for Professional Data Science Practice

Although once asked I knew this was a very subjective question, and I was not looking for specific yes or no, but rather hearing the community's experience, here now I would like the share the following short conclusion for those that might have to make a similar decision.

Golden Rule is Majority Voting What I do mean by that is that it is best to be aligned with other team members and the companies's IT support. If majority or all are Windows based, then one should go for that, and vice versa for Linux, otherwise you are on your own and will suck up much of your time to get certain things up and running (printer driver etc.).
Minor Differences I have witnessed minor differences actually. I have to admit that I do miss Linux terminal. Windows has been trying so hard to integrate a proper terminal, but so far all solutions like Windows PowerShell or cmd or Ubuntu terminal for Windows or even Anaconda Prompt doesn't give that freedom and experience one have with Linux kernel. The rest is pretty much the same, and there has been no really big issue for me migrating from Mac OSx to Surface Pro 6.
Hardware/Software Stability: For long I have been biased that Mac OSx is the most stable OS one could experience, perhaps due to terrible past experience working with Windows (Win 98, XP, 2000 etc.). Although still typical issues remain, which is true to all OSs, I have to say that Windows there days is a super reliable system comparable to Mac OS if not better, you know for daily tasks (I am not bench-marking anything here) and not causing dramas. For the Hardware, honestly if we account for Hardware for Price comparison, Windows wins. It is far more powerful better CPU, RAM, Graphics, Storage for the price you pay. I would get a much lower performing Mac OSx, and nothing is worst that having worried about 250 GM storage you have, while you can get a double or tripple space easily in a Windows equivalent machines. Certainly if price is not a concern, one should not be concerned about this as well. Last but not least, since we are comparing Hardware, I have to mention I am loving my touchable Surface Pro 6 display, and one can detach the screen to turn it to a Tablet which is awesome, and the screen itself is so durable, in contrast to a super sensitive Mac screens. Please note I am not advocating for Windows, just stating facts and sharing personal experience using both.

In case over long period of time, these facts change, I will get back to this post and update again my personal high-level experience.


Author of the paper here - I missed that this is apparently not a TensorFlow function, it's equivalent to Sonnet's scale_gradient, or the following function:

def scale_gradient(tensor, scale):
  """Scales the gradient for the backward pass."""
  return tensor * scale + tf.stop_gradient(tensor) * (1 - scale)


Doing predictive modeling on predicted value

Your approach seems valid. It is not even biaised because you don't use other features than A to build F.

Performing unsupervised learning prior to supervised learning in a very common approach, and there are numerous types of workflows where supervised and unsupervised techniques are combined to solve the problem.

However, be careful with random forest feature importance: there are several ways to compute importances, and not all will give the same results. There are also other techniques to make complex models interpretable, such as LIME for local explanation or SHAP values for both local and global explanation.


It'd be good idea to extract face segment first. In OpenCV there're predefined HaarCascades for face detection. Though, you can run into some problems using them. Photos really need to be well-exposed. They're also prone to false positives as hell.

https://realpython.com/face-recognition-with-python/

Slightly better, but still vanilla, there's dlib.
Worst problem with them is you'll only get bounding box. Background may creep in and mess your color analysis.

To get seminal-like scores you gotta go DNN. Here's some example.

I would propose a bit different approach. Take your data set. Extract faces using existing tool and label skin colour manually. Then train CNN. Just be careful, when you augment data set. Some operations might be damaging, like swapping channels. Color-based analysis is tough. You need to think of covering different exposure values, contrast, white-balance, saturation etc.


How does LSTM fights vanishing gradient?

There are two factors that affect the magnitude of gradients - the weights and the activation functions (or more precisely, their derivatives) that the gradient passes through.

If either of these factors is smaller than 1, then the gradients may vanish in time; if larger than 1, then exploding might happen. For example, the tanh derivative is <1 for all inputs except 0; sigmoid is even worse and is always ≤0.25.

In the recurrency of the LSTM the activation function is the identity function with a derivative of 1.0. So, the backpropagated gradient neither vanishes or explodes when passing through, but remains constant.
The effective weight of the recurrency is equal to the forget gate activation. So, if the forget gate is on (activation close to 1.0), then the gradient does not vanish. Since the forget gate activation is never >1.0, the gradient can't explode either.
So that's why LSTM is so good at learning long range dependencies.


What do graphs of signal vs background neural network outputs represent?

Okay, your output makes sense, given what you wrote in the comments. But if you don't know where the other output comes from, all else is speculation. It could be that it was generated by a neural network with two sigmoid outputs, rather then softmax, and the classification was done based on which output was larger. In that case you are not guaranteed symmetric graphs. So the asymmetric graph does not necessarily imply that it's not generated by a neural network, only that the output does not come from a two-dimensional softmax or a pair of sigmoid/1-sigmoid.


Want to Add More Sample in Majority Class

Any oversampling technique should work just fine.

For example, SMOTE is a technique for generating samples for a specific class. It works by selecting a class, finding the closest samples and generating one at a random position between them. The algorithm doesn't explicitly require those samples be from the minority class. Theoretically, you can apply this principle to the majority class as well.

You can see more details about oversampling techniques (including SMOTE) from this jupyter notebook, which implements logic behind manually.


Error when trying .transform for OrdinalEncoder from Scikit Learn

It is really hard to figure out the logic behind what you are doing, it look odd
But assuming you are trying to apply a preprocessing step to a data frame I would go as follows:
from sklearn.compose import make_column_transformer
from sklearn.preprocessing import OrdinalEncoder

ordinal_features=['LotShape','ExterQual','ExterCond','BsmtQual','BsmtCond',
'BsmtExposure','BsmtFinType1','BsmtFinType2','HeatingQC','KitchenQual',
'FireplaceQu','GarageQual','GarageCond','GarageFinish','Fence','PoolQC']

ordinal_orders=[
    #LotShape
['Reg','IR1' ,'IR2','IR3'],
    #ExterQual
['Fa','TA','Gd','Ex'],
    #ExterCond
['Po','Fa','TA','Gd','Ex'],
    #BsmtQual
['None','Fa','TA','Gd','Ex'],
    #BsmtCond
['None','Po','Fa','TA','Gd','Ex'],
    #BsmtExposure
['None','No','Mn','Av','Gd'],
    #BsmtFinType1
['None','Unf','LwQ', 'Rec','BLQ','ALQ' , 'GLQ' ],
    #BsmtFinType2
['None','Unf','LwQ', 'Rec','BLQ','ALQ' , 'GLQ' ],
    #HeatingQC
['Po','Fa','TA','Gd','Ex'],
    #KitchenQual
['Fa','TA','Gd','Ex'],
    #FireplaceQu
['None','Po','Fa','TA','Gd','Ex'],
    #GarageQual
['None','Po','Fa','TA','Gd','Ex'],
    #GarageCond
['None','Po','Fa','TA','Gd','Ex'],
    #GarageFinish
['None','Unf','RFn','Fin'],
    #Fence
['None','MnWw','GdWo','MnPrv','GdPrv'],
    #PoolQC
['None','Fa','Gd','Ex']
]

transformer = make_column_transformer((OrdinalEncoder(categories= ordinal_orders),ordinal_features), remainder = "passthrough").fit(X_train)

Then you can apply the transformer to both X_train and X_test
transformer.transform(X_train) or transformer.transform(X_test)


If I understand correctly you don't have your target and want to create it using associacion rules.
Fp-growth algorithm already reduces itemset checks e.g. based on the fact that if a certain itemset doesn't match selected threshold any other superset of this itemset won't match it either.
For this reason it is several times faster than apriori.
If you want to create your target using associacion rules and then train a model to predict this target you need let the algorithm go over each example and let it discard each itemset not matching your threshold.
What you can do to speed up the computation is:

Use FP-Growth on a distributed system.
Use even more restrictive condititons to discard itesets faster.
Learn association rules only on a subset of items for example top 1000 bestsellers.


Machine Learning Models for Non-Imaging Medical Clinical Data

Some of the non-imaging examples are

ECG (https://www.researchgate.net/publication/324457712_A_deep_learning_approach_for_ECG-based_heartbeat_classification_for_arrhythmia_detection   ,  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5805987/)
Arrhythmia (https://archive.ics.uci.edu/ml/datasets/arrhythmia , http://ecg.mit.edu/time-series/)
Diabetes  https://www.sciencedirect.com/science/article/pii/S2405959518304624
Genotyping : https://www.mdpi.com/2073-4425/10/1/18/pdf , https://www.biorxiv.org/content/10.1101/241414v1


MR images segmentation for feature extraction

"Feature extraction" is basically a process of reducing the input data (an image of X pixels) to a lower dimension (that is still enough to perform the model's task). So you have 2 options, to use a man-made features (for this you need to be the expert that determines what is a good and relevant feature) or train a feature extractor.

For a man-made feature extractor you need to understand what is the difference between the tumor area and the rest of the image (is it just a matter of the grayscale, where you can use some threshold value? maybe texture or spatial frequency where discrete wavelets can help?). This is not an easy task (there is a reason why the tumor was segmented by some physicist?? {I would think he should be some sort of a doctor}).
I am not an expert in MRI imaging but I've found some papers that talk exactly about your question:

"Feature Extraction for MRI Segmentation - Wiley Online Library"

"Feature extraction and selection from MRI images for the brain tumor classification"

If we are talking about training a feature extractor, then you need to define a model with an input and an output and some quantifiable training goal (loss/score function).
From my understanding, your final goal is to segment just the tumor part of the image, right?

Which means that your model's output should be a binary image with the same size as the input image. The pixels of the output image will be 0 for not-tumor and 1 for tumor pixels. This is similar to tasks of "Salient Object Detection".

The only difference with segmentation is that your output image is binary. In segmentation you will get an output image where each pixel is classified to one of k segments. This might also be relevant to you if you are trying to both segment the tumor and classify it as one of several known tumor-classes. Or if you want to divide the image into different segments, with the tumor being one of them.

If we are talking about a deep-learning model, both tasks share similar architectures known as Fully Convolutional Networks (with many different variations). You can read more about it at a paper named: "Fully Convolutional Networks for Semantic Segmentation".
In these models, the output of the second-to-last (refereed to as the "embedding layer") can usually be used as a feature extractor.


How to develop small workable embeddings for debugging

The embeddings are floating point vectors, it's very unlikely that they are exactly the same. And even if they were, it will only minorly impact the performance of your model, which is not what you are after anyway, because you are debugging. That said, if you want to use it for model selection you have a few alternatives. The easiest one is to apply PCA one time to your whole embedding space and take the first n dimensions. This way you reduce your dimensionality without throwing away too much information. A better but more difficult approach would be to train your own Glove vectors, either on a public corpus or a private one, and set a low dimensionality during the training procedure. It might be difficult to use your findings on these embeddings for model selection because more expressiveness could lead to much better performance with deeper models or could lead to more overfitting.


What is the difference between nDCG and rank correlation methods?

A central problem in ranking is to design a measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the Normalized Discounted Cumulative
Gain (NDCG) which is a family of ranking measures widely used in practice. Although
there are extensive empirical studies of the NDCG family, little is known about its theoretical properties

NDCG has two advantages compared to many other measures. First, NDCG allows each retrieved document has graded relevance while most traditional ranking measures only allow binary relevance. ... Second, NDCG involves a discount function over the rank while many other measures uniformly weight all positions.

More: Info with Theoretical & Mathematical


There are several separate issues:

Survey design
Collection
Data querying
Visualization

Each issue has its own best practices for solutions.

Generally, survey design and collection are independent of coding language. Data querying from a database is done with SQL. Visualization can be done with or without a coding language.

Python and Java can help to automate the process.


How to create a parquet file from a query to a mysql table

I can think of a few ways besides using Apache Spark.

There is a python script doing just that: https://github.com/rondunn/odbc2parquet. It utilizes pyodbc though.
Depending on your use case you may find that https://github.com/blue-yonder/turbodbc would deliver a much speedier experience querying the data. turbodbc can also directly emit arrow arrays.
There is a command line tool written in Rust by yours truly. It is also called odbc2parquet. https://github.com/pacman82/odbc2parquet


What are some good models to test the speed of a data science machine?

MLPerf is a set of benchmarks designed just for that purpose. It tests a variety of common machine learning tasks across a variety of possible hardware.


deep learning for non-image non-NLP tasks?

Yes you can use deep learning techniques to process non-image data. However, other model classes are still very competitive with neural networks outside of signal-processing and related tasks.

To use deep learning approaches on non-signal/non-sequence data, typically you use a simple feed-forward multi-layer network. No need for convolutional layers or pooling layers. The best architecture other than that needs to be explored with cross-validation, and can be time-consuming to discover as deep NNs take a lot of computation to train.

In my experience attempting to use deep(-ish, typically ~ 5 layers) neural networks in Kaggle competitions:

Dropout is still highly effective for regularisation and improving accuracy
Input normalisation - usually to mean 0, standard deviaton 1, is important
Hidden layer activation functions can make a difference. Although ReLU reduces some problems with vanishing gradients, in my experience it is less robust with non-signal data and you will want some other form. If you have only a few layers, then sigmoid or tanh still work OK. Otherwise, look into leaky ReLU, PReLU, ELU and other ReLU variants that attempt to patch its problems with "dead" neurons.
Make use of optimisers designed for deep learning, such as Adam, Adagrad or RMSProp
Use a weight initialisation approach that works with deep learning, such as Glorot.
Consider using Batch Normalisation layers. Not something I have much experience with, but I have seen other people do well with this approach.

Despite all this, XGBoost can routinely and easily beat deep NNs with minimal tuning and training effort in comparison (depending of course on the problem and the data you have). If accuracy is everything to you though, it is possible - although not guaranteed - that an ensemble of deep NNs and other model such as XGBoost will perform better than either singly.


You are completely right, imbalance of labels does have an impact on ranking problems and people are using techniques to counter it.
The example in your notebook applies list-wise gradient boosting. Since pairwise ranking can be made list-wise by injecting the NDCG into the gradient, I will focus on pair-wise rank loss for the argument. I will base myself on this paper (https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/MSR-TR-2010-82.pdf).
$C = -\bar{P}_{ij}$log$P_{ij} - (1 - \bar{P}_{ij})$log$(1 - P_{ij})$
with, $P_{ij}\equiv P(U_{i}\rhd U_{j})\equiv {1\over{1 + e^{-\sigma(s_{i} - s_{j})}}}$ and $\bar{P}_{ij} = {1\over2}(1 + S_{ij})$ for $S_{ij}$ being either $0$ or $1$.
This is actually just a classification problem, with 0 being article i being less relevance than article j and 1 in the opposite case.
Imagine that now you are working with queries which have a lot of matching documents but only a couple of documents have been tagged as relevant. Often such sparse tagging does not mean that ONLY these documents were relevant, but only is caused by the limitation of the estimation of relevance (https://www.cs.cornell.edu/people/tj/publications/joachims_etal_05a.pdf).
Hence, it is not uncommon to down-sample high rated documents.
Another reason for applying imbalance methods such as reweighting labels, is for example bias due to position (see for example https://ciir-publications.cs.umass.edu/getpdf.php?id=1297). The loss is reweighted based on the observed position of the documents when their relevance was tagged.


SMOTE oversampling for class imbalanced dataset introduces bias in final distribution

Class imbalance is a frequent problem in machine learning and techniques to balance the data usualy are of two flavors: undersampling the majority, oversampling the minority or both.
One can always partition the data according to some variables and separately oversample each partition so as to maintain some measure (eg given data distribution). In the same way that separate oversampling can be achieved for only $1$ variable, in the same way separate oversampling can be achieved for $n$ variables. Of course more complex but certainly doable. For example one takes all distinct combinations of variables (or ranges of variables for continous variables) and separately oversamples each such cluster in order to maintain the given data distribution.
The above is a straightforward technique, although one should note that if minority class does not have enough samples there is no guaranty that the given data distribution reflects the (true) underlying data distribution (in other words it may not constitute a representative sample in statistical sense). So for these cases oversampling the whole data, without extra assumptions about underlying distribution, is a maximally unbiased method in the statistical sense.
There is some research lately on hybrid and intelligent methods for (oversampling) class imbalance problems without introducing bias during the process. The following references will provide the relevant background:
Cross-Validation for Imbalanced Datasets: Avoiding Overoptimistic and Overfitting Approaches, October 2018

Although cross-validation is a standard procedure for performance
evaluation, its joint application with oversampling remains an open
question for researchers farther from the imbalanced data topic. A
frequent experimental flaw is the application of oversampling
algorithms to the entire dataset, resulting in biased models and
overly-optimistic estimates. We emphasize and distinguish overoptimism
from overfitting, showing that the former is associated with the
cross-validation procedure, while the latter is influenced by the
chosen oversampling algorithm. Furthermore, we perform a thorough
empirical comparison of well-established oversampling algorithms,
supported by a data complexity analysis. The best oversampling
techniques seem to possess three key characteristics: use of cleaning
procedures, cluster-based example synthetization and adaptive
weighting of minority examples, where Synthetic Minority Oversampling
Technique coupled with Tomek Links and Majority Weighted Minority
Oversampling Technique stand out, being capable of increasing the
discriminative power of data

Learning from Imbalanced Data, 9,  SEPTEMBER 2009

With the continuous expansion of data availability in many
large-scale, complex, and networked systems, such as surveillance,
security, Internet, and finance, it becomes critical to advance the
fundamental understanding of knowledge discovery and analysis from raw
data to support decision-making processes. Although existing knowledge
discovery and data engineering techniques have shown great success in
many real-world applications, the problem of learning from imbalanced
data (the imbalanced learning problem) is a relatively new challenge
that has attracted growing attention from both academia and industry.
The imbalanced learning problem is concerned with the performance of
learning algorithms in the presence of under represented data and
severe class distribution skews. Due to the inherent complex
characteristics of imbalanced data sets, learning from such data
requires new understandings, principles, algorithms, and tools to
transform vast amounts of raw data efficiently into information and
knowledge representation. In this paper, we provide a comprehensive
review of the development of research in learning from imbalanced
data. Our focus is to provide a critical review of the nature of the
problem, the state-of-the-art technologies, and the current
assessment metrics used to evaluate learning performance under the
imbalanced learning scenario. Furthermore, in order to stimulate
future research in this field, we also highlight the major
opportunities and challenges, as well as potential important research
directions for learning from imbalanced data.

Data Sampling Methods to Deal With the Big Data Multi-Class Imbalance Problem, 14 February 2020

The class imbalance problem has been a hot topic in the machine
learning community in recent years. Nowadays, in the time of big data
and deep learning, this problem remains in force. Much work has been
performed to deal to the class imbalance problem, the random sampling
methods (over and under sampling) being the most widely employed
approaches. Moreover, sophisticated sampling methods have been
developed, including the Synthetic Minority Over-sampling Technique
(SMOTE), and also they have been combined with cleaning techniques
such as Editing Nearest Neighbor or Tomek’s Links (SMOTE+ENN and
SMOTE+TL, respectively). In the big data context, it is noticeable
that the class imbalance problem has been addressed by adaptation of
traditional techniques, relatively ignoring intelligent approaches.
Thus, the capabilities and possibilities of heuristic sampling methods
on deep learning neural networks in big data domain are analyzed in
this work, and the cleaning strategies are particularly analyzed. This
study is developed on big data, multi-class imbalanced datasets
obtained from hyper-spectral remote sensing images. The effectiveness
of a hybrid approach on these datasets is analyzed, in which the
dataset is cleaned by SMOTE followed by the training of an Artificial
Neural Network (ANN) with those data, while the neural network output
noise is processed with ENN to eliminate output noise; after that, the
ANN is trained again with the resultant dataset. Obtained results
suggest that best classification outcome is achieved when the cleaning
strategies are applied on an ANN output instead of input feature space
only. Consequently, the need to consider the classifier’s nature when
the classical class imbalance approaches are adapted in deep learning
and big data scenarios is clear.

Hope these notes help.


Constructing the Confusion matrix from given metrics

[edit thanks to comment] I'm assuming this is a binary classifier, since normally a multi-class classifier would not be evaluated with precision/recall (it would require micro/macro precision/recall).

Yes, that should be enough:

accuracy = 92.7%:

$$\frac{TP+TN}{110}=0.927 \rightarrow TP+TN=101.97$$

This means we have 102 correct predictions, so $FP+FN= 8$ incorrect predictions (since $TP+FP+TN+FN=110$).

precision = 96.9%:

$$\frac{TP}{TP+FP}=0.969 \rightarrow TP=31.258\times FP$$

recall = 95%:

$$\frac{TP}{TP+FN}=0.950 \rightarrow TP=19 \times FN$$

This gives us:

$$\frac{TP}{31.258}+\frac{TP}{19}=8 \rightarrow TP = 94.6$$

let's assume that means 95 true positive instances, so we get:

$FP = 3$
$FN = 5$
$TN = 7$


Machine learning mode retraining after getting new data

Once you trained your model, you have a measure of accuracy.

You should retrain your model when the accuracy drops below a certain threshold.

Another approach is to use a Bayesian model, in which you update your model with each new observation.

I would suggest looking into pymc3's examples


Hand-crafted decision tree inspired from learned decision tree

I did this once, in a position similar to yours.

My constraints were,

People skeptical of machine learning
Required interpretability
Required very fast inference
Explainable in text documentation
Implementable in any language

Traditionally, this would have been done by an expert system, so training a decision tree was essentially just one step further.

On my end, all of the work was being done in Python. I tried a handful of ML models, with and without hand-crafted features. A single decision tree performed nearly as well as anything I tried, and outperformed a fully handcrafted expert system. I then created an "inference model" in the form of if-statements for each split in the tree to confirm performance was equal to the decision tree itself. I even manually ignored some low entropy splits or where a single feature was underutilized, finding that for the same number of branches, a larger trained tree manually retaining fewer nodes outperformed a small tree where I replicated it exactly.

Some of the splits were unintuitive, but that's partially the point of throwing a model at it. Be sure to really dig into the failure modes of every alternative to help everyone involved have confidence in your work.

Given the constraints, it was a great solution to the problem.


The way to calculate $p(x)$ is indeed:
$$p(x) = \sum_k p(C_k) \ p(x| C_k)$$
Since in general one needs to calculate $p(C_k,x)$ (numerator) for every $k$, it's simple enough to sum all these $k$ values. It would be incorrect to use the product, indeed.

Lastly, I understand we don't actually need the denominator to find our probabilities but am asking out of curiosity.

Calculating the marginal $p(x)$ is not needed in order to find the most likely class $C_k$ because:
$$argmax_k(\{ p(C_k|x)  \}) = argmax_k(\{ p(C_k,x)  \})$$
However it's actually needed to find the posterior probability $p(C_k | x)$, that's why it's often useful to calculate the denominator $p(x)$ in order to obtain $p(C_k | x)$, especially if one wants to output the actual probabilities.


In the above case, you can collect annotated data ( which is not seen by the model during training) and validate the predictions made by the model.
And another way is if you are a domain expert or have sufficient knowledge on the data, you can tweak the input values and test the predicted output with your expected output.


Creating validation data for model comparison

Predicting and scoring are two different tasks.

And according to your answers and comments you are not solving prediction problem. You just want to set to each student a number in range [1,100] according to some rule. This is ranking (or scoring, whatever).

Therefore, the terms #prediction_model, #accuracy, #validation, #training_set are out of this scope. You don't need to validate anything. You are not making predictions.

What you want is to map ranks to students.

But a problem is that you have mostely categorical data (school name, location etc) that cannot be 'ranked'. Some of them are useless at all: how does the student name refer to his school progress? :)

If you change it somehow to numerical (e.g. 'Skill_1_level', 'Skill_2_level', 'remoteness_of_location', 'school rank' etc) than you can do some ranking:

Normalize data: each of your factors

Multiple by 100, as you want [0,100] range instead of [0,1]
Set up weights based on your experience according to factor's importance. So that the sum of weights is 1.
And finally build a rank (score):

Rank = 0.1 * skill_1_level + 0.2 * skill_2_level + 0.05 * remoteness_of_location + 0.5 * school_rank + ...


How to improve regex while fetching record dynamically

Based on your sample data, I replicated it 50000 times and the result is as follows-

>>> df = pd.DataFrame({'COL1':[12 ,57],'COL2': [':402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:22:hhhh',':42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:hhhhhh']})

>>> for _ in range(50000):
        df = df.append({'COL1':12,'COL2': ':402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:22:hhhh'}, ignore_index = True)
        df = df.append({'COL1':57,'COL2': ':42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:hhhhhh'}, ignore_index = True)

>>> df.shape
(100002, 2)

Then I defined a custom function and applied to the columns-

>>> def somefunc(x,y):
        res = []
        for i in range(len(x)):
            ix = y[i].find(x[i]) + len(x[i])
            res.append(y[i][ix+1:ix+8])
        return res

>>> df['col3'] = somefunc(df['COL1'].astype(str),df['COL2'])
>>> df.head()
    COL1                                   COL2             col3
0    12  :402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:2...  fgh,ghg
1    57  :42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:...  hghgh,g
2    12  :402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:2...  fgh,ghg
3    57  :42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:...  hghgh,g

I did not use regex and this function took nearly 5 seconds to complete on 100000 rows.


how are split decisions for observations(not features) made in decision trees

Do i also need to try all the combinations of a feature split, for above case, height < 100 & height > 100, then height < & height > 110 & height < 90 & height > 90 and so on

Suppose that you only have 3 posible values on height 80,90,100. Then the decision tree will try to split on if X >= 80,X >= 90 and X >= 100. If the instance does not satisfy the condition will fall on the other leaf.
So if the split is above the threshold it will fall into another leave, if its below it will fall into another. And then there recursively.

So the answer to this is NO. It will only try on of the logical conditions -- >= for example. I believe this depends on the implementation.
I encourage you to build a decision tree from scratch to fully understand the algorithm.
I reccomend you this video https://www.youtube.com/watch?v=y6DmpG_PtN0 but there is a lot of tutorials there. Choose the one that suits you most.


Is using cross validation on your entire dataset acceptable when dealing with a small sample size

It depends, there can be different reasons to use CV:

if you're using it to evaluate your model only, then it's fine to use the whole training set.
If you're using it to evaluate multiple models and then select the one which performs best (e.g. tuning hyper-parameters), then you shouldn't use the whole training set as you still need to evaluate the result of this tuning step on some unseen data.


Overfitting due to features correlating with training set generation rules

I would say that there are no "problems" in the sense what is happening is to be expected.

First of all, here is a key ML reminder which somehow often gets lost:

Performing well on the test set is pointless if that test set is not representative (i.e. is not similar to unlabeled instances)
Adding more training data does not help if that training data isn't covering new situations

You say you created your training set using string matching rules, I assume that these rules are similar to " "mortgage repayment" was a "housing" observation." as you pointed out in the comments.

Since your model considers bigrams as input, it is not surprising that it found a way to reverse-engineer which string matching rules you used.

To improve your model, I would look at what mistakes your model currently makes on unlabeled data, this should provide you with training instances that do not fit your string matching rules.


I work on plant disease image recognition for my Master's thesis. Until now, I did not find any good database with fruit images, but there are two quite large databases which contain leaf-images of infected plants. One is the Plantvillage-dataset which has pictures taken under laboratory conditions and the other one is the Digipathos-dataset which contains pictures that are nearer to real-life scenarios with multiple objects in focus and difficult backgrounds.

You can download the Plantvillage-dataset from this Github-Repo. For the Digipathos-dataset, I wrote a downloader which you can also find on Github.


Not sure how intense your professor is going to make either course, but assuming it's the hardest possible introductory course, it would be better to take linear algebra before statistics. A lot of statistical operations require fundamental linear algebra concepts. You need to be able to understand abstract notions like vector spaces and fields to know when your data is in the correct range, you need to be able to understand what operations are linear and what are non-linear so you don't apply a wrong operation on something, and you need to most importantly be able to understand linearity and non-linearity because that is the basis of a lot of statistical methods. Statistics involves transforming data in a way that makes sense but you learn the fundamentals of transformation in linear algebra. But if your stats course is just about p-values and superficial hypothesis testings, and your linear algebra course is just direct applications like row-reducing matrices, then it doesn't really make a difference which you take first.


Yes, you can perturb your data (and targets) in ways that you wish your model to be robust against, for example by adding small amounts of noise (possibly Gaussian) or synthetic anomalies, or by creating meaningful aggregates. Some models also improve just by running the same training data through them more than once, randomly shuffled (“epochs”).


To include this logic into a pipeline you have to create a custom transformer. You need to ask yourself:

[INIT] Are there any parameters in my logic?

The variable you want to impute and the category you want this imputation to be based on.

[FIT] What part of the logic is related to computing what the transformation will be?

When you compute the median() by groups and store the data somehow for later transform.

[TRANSFORM] What part of the logic transforms the data, given the parameters (in 1) and the setting that was made (in 2)?

When you get the parameters (access specific key in the dictionary) to retrieve what was the mean for that group, and then fill the missing value with this.

Here is an example :

from sklearn.base import BaseEstimator, TransformerMixin

class CustomImputer(BaseEstimator, TransformerMixin) :
     def __init__(self, variable, by) :
          #self.something enables you to include the passed parameters
          #as object attributes and use it in other methods of the class
          self.variable = variable
          self.by = by
          return self

     def fit(self, X, y=None) :
          self.map = X.groupby(self.by)[variable].mean()
          #self.map become an attribute that is, the map of values to
          #impute in function of index (corresponding table, like a dict)
          return self

     def transform(self, X, y=None) :
          X[variable] = X[variable].fillna(value = X[by].map(self.map))
          #Change the variable column. If the value is missing, value should
          #be replaced by the mapping of column "by" according to the map you
          #created in fit method (self.map)
          return X

Now, it can be included in any pipeline :

#Minimal example, you could include this imputer in columns transformer to
#apply it multiple time
pipeline = Pipeline(steps = [('myImputer', CustomImputer('variabletofill',
                                                         'based_on_variable'),
                              ('model', LinearRegression())])

y_pred = pipeline.fit(X_train, y_train).predict(X_test)

As you can see, the mapping is computed only based on train data. Then it is re-used to impute the missing values. It is data-leakage-proof.
Here is a good article that explains how to create a custom transformer.

Hope this helps


How to decide who to market? Clustering or Decision Tree?

Since you have label data (i.e., sale amount), you can apply supervised machine learning.
After a model has been fit, the features that contributed to predicting targets can be found.
Decisions trees would be a relatively simple option because it easily handle the different types of features and can a yield a decision path.


Since the weight of each feature (column) determines how important that column is in determining the output value, the abstraction means the most efficient and correct weights. For instance, here the features for pattern 1 are 1 and 0, and each of them are given a weight multiplying with which the output is calculated:

feature1 * weight 1 + feature2 * weight2 = output

So based upon the current pattern, the best values for weights to give the already extant outputs, are the ones given.

The test for the abstraction can be done via the compile method. First initial wights are assigned to features, then a single record or a patch of data passes through network and a cost function is calculated, then the error is backpropagated and new weights are set for the features (based on how much responsible each of them is for the error) and this continues.


Whether to replace NaN values in result column

There is no correct way as dealing with nan values it depends on the dataset.But i would suggest rather than doing single imputation you could do multiple imputation using IterativeImputer in sklearn.
As very less target variables are missing it won't make a significant impact unless the problem statement is sensitive to finding outliers like fraud detection.
Refer Iterative Imputation


Your model is indeed overfitting. There can be a lot of reasons why it could be happening - How many images are you training with?, Aren't you using Regularization, data augmentation using random crops or flips, Dropout to prevent overfitting?

  So my question is at what loss (val_loss) can we consider the model
  trained?

We can generally stop training when Validation loss doesn't decrease anymore. This is generally done by defining a variable eg: n = 5 and checking if the validation loss has decreased after running n epochs. If you're solving a well-known problem(MNIST, Fashion-MNIST, CIFAR-10, etc), you can check the best score/loss achievable by the model and tweak your model(add or remove layers, prevent overfitting, etc) to achieve the score.

  And is there a problem if the images that I feed the VGG-16 are
  RGB/255?

You should preprocess the images for faster training and also subtract  the mean RGB pixel values of images used to train pretrained network(if the pretrained net is trained using imagenet data,then subtract imagenet dataset's mean RGB pixel values from your input images) from your input images. Here you can read keras preprocessing_input
method.

  And if someone could explain how high validation loss (even if validation accuracy is high) could impact the model?

Validation loss measures how generalizable your model is to unseen data. If your training loss and validation loss don't improve during training, then you are underfitting. If you're training loss is low but your validation loss is stagnant or increasing then you're overfitting, which means that your model is learning unwanted noise or patterns which help it learn or memorize training data but are not generalizable on an unseen validation set.

Here is a link explaining data preprocessing, augmentation, transfer learning using pretrained net.


Using Pandas to_numeric() in Azure Machine Learning Studio

What’s new in Pandas v0.17.0

  DataFrame.convert_objects has been deprecated in favor of
  type-specific functions pd.to_datetime, pd.to_timestamp and
  pd.to_numeric (new in 0.17.0) (GH11133).

So for Pandas versions < 0.17.0 you can and should use: df.convert_objects(convert_numeric=True)

Demo:

In [213]: x = pd.DataFrame({'a':['11', 'aaa', '0', np.nan, '123']})

In [214]: x
Out[214]:
     a
0   11
1  aaa
2    0
3  NaN
4  123

In [215]: x.dtypes
Out[215]:
a    object
dtype: object

In [216]: x = x.convert_objects(convert_numeric=True)

In [217]: x
Out[217]:
       a
0   11.0
1    NaN
2    0.0
3    NaN
4  123.0

In [218]: x.dtypes
Out[218]:
a    float64
dtype: object


How to use multiple cross-section observations per subject for churn prediction?

If you know SQL, then my method might work. I'll break it out into subqueries to make it a little more understandable. I don't know your columns so this might not translate perfectly.
The point is to have 1 query identify all active students at a particular point in time. The point of time you choose is arbitrary, but it must have enough time forwards enough to the end of your data so that you can establish a forward-looking window to satisfy churn or not. I am calling this the snapshot_date. Don't have students who churned before the shapshot_date in this resultset at all.
Then, you join this to a subquery which uses that shapshot_date and looks forward for the churn condition. This returns only students who churned, but churned within the window that you decide. If they churned in the future beyond your window, treat them as if they have not churned.
The very first query that returns a list of active students might still return multiple rows for each student. You handle this by deciding how to aggregate those rows into a single row for each customer. You might end up creating a lot of good variables for your model this way. I added a few as an example.
Then, your SQL would look similar to this:
SELECT active_students.*,
case when student_churned is not null then 1 else 0 end as target_student_churn
FROM
    (
        SELECT Key,
        DateDiff(snapshot_date, date_of_enrollment, "Days") as time_since_enrollment,
        min(date_of_enrollment) as date_of_enrollment,
        max(felev) as max_felev,
        COUNT(DISTINCT course) as distinct_courses_taken,
        ...
        FROM table
        WHERE date_of_enrollment < snapshot_date
        AND {student has not churned as of snapshot date}
        GROUP BY Key
    ) as active_students
LEFT OUTER JOIN
    (
    SELECT Key
    FROM table
    WHERE {student churned within hard coded time period AFTER snapshot date}
    GROUP BY Key
    ) as student_churned
ON active_students.Key = student_churned.Key;

You may wonder then how this accounts for seasonality over time if you chose your snapshot date randomly. Well, just change the snapshot_date and repeat while you continue to stack the results on top of one another. Having the snapshot date as a column will help you extract the "season" from each resultset.
Now the students might be repeated again in the final dataset. So, when you build the model you want to make sure that the students' rows fall into the same validation or holdout dataset. So use GroupKFold or something like that. I hope this helps.


Un-learning a single training example from a trained model

is it even possible to "un-learn" a single training example without retraining on the new data?

To the best of my knowledge, the answer is no except in some very special cases.

The most obvious exception that comes to mind is instance-based learning, such as kNN: since the "model" itself consists only of the set of training instances, it's straightforward to remove an instance.

In general, supervised ML relies on generalizing patterns based on the instances from the training set. Any non-trivial model consists of multiple such patterns, with every pattern potentially resulting from a different subset of instances. Even if there was a way to trace which instance participated to which pattern (that would be extremely inefficient), removing any pattern would probably cause the model to fail.


It looks like a classical email classification problem: Spam or Ham.
In the following I will use some specific slang, do not worry, look it up in Google, if you will see specific terms you do not know.

Prerequisite: ideally your dataset will be "balanced", having, say, 50% cars and 50% houses, could be 40/60 as well. Problems will arise, when one class will be say < 10%.

So called "bag of words" can be seen as a start.

Steps:

Your dataset consist of two columns, say, "description" and "label"
Label has only two values car or house
Tokenize your decription column (single words)
Cleanse the desc column: remove stop words (like and, the , a - as they do not have any value), remove punctuation, possibly stem the words, remove numbers
calculate document-term-matrix, possibly use TF-IDF

By the end of this operation you will have a dataset with a label column (car or house) and long number of word (or even n-gram, e.g. "great_view" is a bi-gram) columns containing binary values:

Label; vehicle, balcony, wheel, ..., ...., great_view
Car,   1,        0,      1, ....., ...., ..., 0
House, 0,        1,      0, ..., ...,         1

Then use naive bayes or logistic regression as a start to train your model.
Pre-process every new description as above and use your trained model to assign a probability to be a "car" or a "house", check the confusion matrix, maybe adjust the threshold .

Everything I described can be done e.g. in R or Python.
In R use the text mining package "tm".


One Hot Encoding vs Word Embedding - When to choose one or another?

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each new category. However, that speed and simplicity also leads to the "curse of dimensionality" by creating a new dimension for each category.
Embedding is a method that requires large amounts, both in the total amount of data and repeated occurrences of individual exemplars, and long training time. The result is a dense vector with a fixed, arbitrary number of dimensions.
They also differ at the prediction stage: a One-Hot Encoding tells you nothing of the semantics of the items; each vectorization is an orthogonal representation in another dimension. Embeddings will group commonly co-occurring items together in the representation space.
If you have enough training data, enough training time, and the ability to apply the more complex training algorithm (e.g., word2vec or GloVe), go with Embeddings. Otherwise, fall back to One-Hot Encoding.


Feature engineering from date, mean and standard deviation

Date fields are quite interesting data since the limit of what you can "feature engineer" with them is your imagination. However, it is difficult to know a priori if one of them will improve your model before you try it.

Here some ideas:

Year
Month (1-12)
Day of Month (1-31)
Day of week (1-7)
Week of year (1-52)
The quarter of the year (1-4)
Is it weekend or weekday (0,1)
Season (Winter, Spring, Summer, Autumn)

If you have also the time on the date field, then you can try these:

Hour of day (1-24)
Part of the day based on eating habits (breakfast, lunch, dinner etc)

If you have the destination of the flight, then you can use an external source and get the holidays or specific "big events"

Is the flight during a holiday (0,1)
What is the distance in days from the closest holiday (before and after the flight)
Any big event on the destination like Olympic Games, Superbowl, Football finals etc


PyData - talks about Python Data tools
Link: http://pydata.org/events/

There is one PyData conference on the east coast and one on the west coast each year.

NIPS - Neural Information Processing Systems (NIPS)
Link: https://nips.cc/

This is one of the hardest / most prestigious academic Machine Learning conferences to get an abstract / poster accepted.

The 5th International Workshop on Parallel and Distributed Computing for Large Scale Machine Learning and Big Data Analytics (IEEE IPDPS 2016)
Link: http://parlearning.ecs.fullerton.edu/

This one is also an academic conference with paper submission.

Note:
I am not sure if you want academic or nor academic conferences (have conference proceedings / papers associated with the conference).
Some conferences are not about new data science methodologies  but the tools and libraries (e.g. PyData) that implement existing methodologies.
Also, data science is very broad and includes Stat, Machine Learning and data warehousing / mining etc.


Tweepy is one of the best libraries for analyzing and hacking around with the Twitter API.  (Being a contributor for tweepy, I can vouch for it's stability and quality)

For a Python wrapper for the Facebook graph API, you can use the Facebook-Insights library, which is well-maintained and neat documentation.

There are services out there which can mine you information, but they are limited to the complexity of the query.

For example: "How many people tweeted about banana on Monday?" can be answered from existing tools

"How many people happy people tweeted about banana on Monday" can also be done, but would require more efforts and the software or tool should be able to detect emotion.

So, if you are into research, I would advise you to go with the API's and good ol' programming!


First, you have to have an object detector for recognizing different objects. After that, you have to post-process the outcomes of your detector to count the numbers. Basically, you cannot recognize all objects due to the fact that the number of labels that a detector finds is limited. I highly recommend you using built in models in the libraries you mentioned due to this reason that training detector models are very time consuming and need appropriate hardwre. If you want to learn something, take a look at papers that already exist and try to implement them. For this case there are numerous studies that I separate them in this way.

Pixel-wise classification based techniques
YOLO-based techniques

This is a general classification that there may not be any consensus on that but is something that shows you the mainstream directions.

I guess the ImageNet data-set contains the label clock. Try to use them alongside another label after implementing the model you want. Don't train your model using all the labels which takes to much time.


Replace this for loop with a pandas vector operation

Try using list comprehension:
df_means = [np.diff(x)[0] for x in [df_common.sample(1000, replace = True).groupby(["doggo"]).retweet_count.mean().values.tolist() for _ in range(10000)]


how to use word embedding to do document classification etc?

So, one document is a matrix. If I want to use some traditional method like random forest to classify documents, how to use such data?

You can't, at least not directly because traditional methods require a fixed number of features for every instance. In the case of document classification the instance must represent the document, so unless all the documents have exactly the same length (unrealistic) it's impossible to use a set of vectors as features.

The traditional approach would consist in representing a document with a vector where each cell represents a word in the vocabulary, and the value is for instance the TFIDF weight of the word in the document.


Accuracy value constant even after different runs

I see that you have set a Random Number Generator(rng) seed in the following line in your code: rng(1).

So, this splits the data in the same way no matter whichever run. So, that is the reason why you are getting the same error values.

Try removing the line. Then, the data shall be slit randomly (as there is no seed now). You shall get different error then, depending on how the split is done (which is, randomly).

A model which generalizes well should be robust to the choice of seed


K-means clustering on big data stored on multiple nodes on HDFS

Posting details from the link in comments above.

The data can be partitioned among multiple processors (or streams or threads). Each processor can read the previous iteration's cluster centers and assign the rows on the processor to clusters. Each processor then calculates new centers for its of data. Each actual cluster center (for the data across all processors) is then the weighted average of the centers on each processor.

  In other words, the rows of data do not need to be combined globally.
  They can be combined locally, with the reduced set of results combined
  across all processors. In fact, MapReduce even contains a "combine"
  method for just this type of algorithm.

  You postulate a shared file that has the centroids as calculated for
  each processor.  This file contains:

The iteration number.
The cluster id.
The cluster coordinates.
The    number of rows assigned to the cluster.

This is the centroid file.

An iteration through the algorithm is going to add another set of
   rows to this file. This information is the only information that
   needs to be communicated globally.

  There are two ways to do this in the MapReduce framework. The first
  uses map, combine, and reduce. The second only uses map and reduce.

link with all details


Yes, you are correct in thinking that the squared error of an autoencoder prediction for a single example is the average of the squared error of the prediction for all dimensions. Similarly, the squared error for a whole batch of examples will be the average of the error of each example in the batch.


What is the relationship between "square loss" and "Mean squared error"?

$$V(f(x), y) = (1-yf(x))^2$$

In the context of classification where $y_i$ is $1$ or $-1$:

\begin{align}
MSE &= \frac1m \sum_{i=1}^m (y_i-f(x_i))^2 \\
&= \frac1m \sum_{i=1}^m y_i^2 (1-y_if(x_i))^2\\
&= \frac1m \sum_{i=1}^m (1-y_if(x_i))^2\\
&= \frac1m \sum_{i=1}^m V(f(x_i), y_i)
\end{align}


Is there a way of performing stratified cross validation using xgboost module in python?

What you are doing is a typical example of k-fold cross validation.

XGBoost is just used for boosting the performance and signifies "distributed gradient boosting".

First, run the cross-validation step:

kfld = sklearn.cross_validation.KFold(labels.size, n_folds=10)

Then, use the train and test indices in kfld for constructing the XGBoost matrix and re-scaling weights by looping over them(the indices).

A very neat implementation has been given as a Kaggle example here.

So, cross validation is not done with the xgboost package, it is done with the cross_validation module of sklearn, and then the gradient boosting is done on the indices of the k-fold validation variable results.


Generalization of Correlation Coefficient

I assume that when you speak of correlation coeficient, you have the Pearson linear correlation in mind. Indeed, there are other options. Two very popular ones are the rank correlations respectively called Spearman's $\rho$ and Kendall's $\tau$.

To give you an idea of what they are, consider $n$ observations from a $d$-dimensional random vector $X = (X_1,\dots,X_d)$. Also let $X_{ij}$ be the $i$th observation for variable $j$. These measures are called rank correlations because they can be computed using the ranks only. What I mean is that if you sort all $X_{ij}$, $i=1,\dots,n$, and replace the biggest observation by $n$, the second biggest by $n-1$, and so on (do that for all columns $j$) and call you new observations $R_{ij}$, then

the empirical Spearman's $\rho$ (matrix) is simply the Pearson linear correlation (matrix) of $(R_1,\dots,R_d)$; and
the empirical (pairwise) Kendall's $\tau$ between $X_{i_1}$ and $X_{i_2}$ is the probability of concordance minus the probability of discordance between two iid observations, say $(X_{1 i_1},X_{1 i_2})$ and $(X_{2 i_1},X_{2 i_2})$, which  can equivalently be computed from the ranks $(R_{1 i_1},R_{1 i_2})$ and $(R_{2 i_1},R_{2 i_2})$ instead.

A rank correlation between $X_{i_1}$ and $X_{i_2}$ of one indeed means perfect concordance (i.e. $X_{i_1}$ always increases with $X_{i_2}$), but that does not necessarily means they are linearly related. The ranks are linearly related.

Just to make the concept of concordance clearer, here the (bivariate) observations are all concordant

and here they are all discordant

So that when you consider a cloud of points you have some pairs that are concordant and others that are discordant.

Note that this answer provides examples, but there are many other ways to approach the question. As commented by Emre, information-theoretic measures are also an option.


How to Manipulate data for multiple visits per person?

So, here you have a couple of questions:

Find total visits per customer by date

To do this bit, you have simply do a pandas Dataframe groupby, like so:

customer_id.groupby(['user_id', 'date']).size()

Resources for working with dates in pandas

https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html
This is official documentation from pandas, so should hopefully give you all the tools you need.

Unfortunately cannot answer the question on feature tools, so I look forward to seeing answers on this.
How to model the next customer visit date given their previous visits.

One way to do this would be that if customer visit frequently, then one idea would be to model this data with a Recurrent Neural Network (RNN) or LSTM (A video on the principles behind RNNs: https://www.youtube.com/watch?v=UNmqTiOnRfg), where it takes the sequence of inputs (dates when particular customer has visited) and return an output date after observing the sequence.
Before this though, might be useful to do customer segmentation and treat observe visiting patterns over customer groups than as individuals. One way to do this any clustering algorithm like agglomerative clustering.


After taking a look at your code, it seems that you've not employed any kind of regularization. You may want to use dropout. Moreover, in convolutional autoencoders, in the decoder part, there is a well-known artifact called checkerboard. I don't know how this can be a problem for your task since you're using one-dimensional convolution in the decoder. By the way, I guess using dropout will suffice. Try to use both in decoder and encoder.


Plot dataframe with two columns on the x axis

Using matplotlib, you could create a custom tick formatter to show the right ticks. The year and month can either be fetched from the dataframe via the index (df.iloc[value]['Month']) or just be calculated.

Here is an example. You can also read the month name in the status bar when you hover over a position in the plot. The xticks (the positions where to have a tick with a label) can either be set automatically or manually.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

month_name = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']

def format_func(value, tick_number):
    value = int(value)
    year = 2001 + value // 12
    month = value % 12
    # return f"{month+1}-{year}"
    return f"{month_name[month]}-{year}"

N = 229
# create a dataframe resembling the example
df = pd.DataFrame({'Year': np.repeat(range(2001, 2021), 12)[:N],
                   'Month': np.tile(range(1, 13), N // 12 + 1)[:N],
                   'monthlycount': np.random.binomial(5000, .7, N)
                   })

plt.plot(df['monthlycount'], marker='o', c='crimson')
plt.xlim(0, N - 1)
plt.xticks(range(0, N, 12), rotation=20) # set major ticks every year
# plt.gca().set_xticks(range(0, N, 12), rotation=30)
plt.gca().set_xticks(range(0, N), minor=True)
plt.gca().xaxis.set_major_formatter(plt.FuncFormatter(format_func))
plt.tight_layout()
plt.show()


Recommendation System to integrate with an android app

I don't want to sound harsh but building a recommender system on a android isn't really Data science.

In any recommender system context, the science on data is usually done in ad-hoc manner.

As for the algorithm implementation, validation and scaling, data science can play a part in that. But this is still not related to the fact of serving the data in a mobile application nor a website. That's mainly computer engineering.

Recommender systems algorithms a quite expensive to compute on any mobile nevertheless. It's best fit to be done by a dedicated server which can also play a role in serving the information when needed. I can't get into the details of this thought as it's quite broad to fit in a single answer on the site.

I advice you to read my answer on the time/space complexity challenge in building this types of application. That may help you getting some insights about the architectural design of a recommender engine in the real world.

I hope that this answers your question.


Voting combined results from different classifiers gave bad accuracy

The approach you are considering is similar to a multi-class SVM or a one-vs-the-rest approach.

And here is how I describe the problem. The support vector machine, per example, is fundamentally a two-class classifier.

In practice, however, we often have to tackle problems involving K > 2 classes. Various methods have therefore been proposed for combining multiple two-class SVMs in order to build a multi-class classifier.

One commonly used approach (Vapnik, 1998) is to construct K separate SVMs, in which the kth model y_k(x) is trained using the data from class C_k as the positive examples and the data from the remaining K − 1 classes as the negative examples. This is known as the one-versus-the-rest approach where :

y(x) = max_k y_k(x)

Unfortunately, this heuristic approach suffers from the problem that the different classifiers were trained on different tasks, and there is no guarantee that the real-valued quantities y_k(x) for different classifiers will have appropriate scales.

Another problem with the one-versus-the-rest approach is that the training sets are imbalanced. For instance, if we have ten classes each with equal numbers of training data points, then the individual classifiers are trained on data sets comprising 90% negative examples and only 10% positive examples, and the symmetry of the original problem is lost.

Therefor, you got your bad accuracy.

PS: Accuracy, in most cases, is not a good measure for evaluating a classifier model.

References :

Vapnik, V. - Statistical Learning Theory. Wiley-Interscience, New York.
Christopher M. Bishop - Pattern Recognition and Machine Learning.


Entity recognition with context/relation

With Wolfram Language you may use TextCases or TextContents from the Text Analysis guide.  These are both experimental functions so may change a bit before they are finalised in a future version.  Both have the TargetDevice option so you can run them on one or more GPUs if you have them.

Reference the Text Content Types guide for the list of entities these functions have been trained on.

I think TextContents is an optimised call to TextCases for multiple cases so I will only use TextContents below.

Use TextContents to locate all trained entities in your sentences.

res1 = TextContents["The temperature today is 35°C.", TargetDevice -> "GPU"]

res2 = TextContents["Store risperidone tablet at 20°C.", TargetDevice -> "GPU"]

TextContents returns a Dataset object that can be Query'ed for specific cases.

For example for your "DrugTemperature"

res2[ContainsAll[{"Chemical", "Quantity"}], "Type"]

True

and the quantity units

res2[
  SelectFirst[#["Type"] == "Quantity" &],
  "String" /* SemanticInterpretation /* QuantityUnit]

"DegreesCelsius"

An Entity of type "Chemical" has a wealth of properties so additional information can be obtained if needed.  For example,

res2[
 SelectFirst[#["Type"] == "Chemical" &],
 "String" /* Interpreter["Chemical"] /* (#["MoleculePlot"] &)]

Hope this helps.


Did you split your „original“ data into Train/Dev/Test?
if not, try splitting your data into three parts. Train the model on your Train set, tune your hyperparameters on your Dev set and test your network on the Test set. The validation error doesn’t give an unbiased error, because the hyperparameters are tuned to it. That’s why you should always test your model on a given Test set. If the error is also low for the Test set I would guess you have another problem:
Without knowing too much about your approach and assuming you have no bugs, my guess is that the high error is because your Validation and Test set come from different distributions. I assume that maybe sales in 2015-2017 are quite different to those in 2018.
You should also take a look at Andrew Ng's explanation of this particular problem.


Reason: Average word vector embedding encodes word content and word order effectively

When reading the paper there actually is some points towards the background of these phenomena:

About predicting a word meaning there is only the surprise present, but about predicting the order there was a whole section about the statistical influence of natural language underlying mechanics to the performance of the CBOW. It appears, that order of sentences or even word pairs was easy for CBOW whereas a random permutation with no natural order dropped also the performance.

If we go very basics, CBOW (Continuous Basket Of Words) mission is to:

  Predicting a word given its context. [1]

Thus, with even an average (or only because of it) we can identify which word is which, although in a little bit weird way around. Just because that is how it is behaving. Secondly, basket is for training the neural network, not the full algorithm. What is the model obtained would be different of the way it gets its data.

What makes me guessing is the drop on ability to identify from larger range of words. As we all are surprised how averages make the trick so, well, maybe the true nature is revealed with larger dimensions.

[1] https://www.quora.com/What-are-the-continuous-bag-of-words-and-skip-gram-architectures