Adjusted R-squared is too high (=1) in Linear Model

I'd strongly disagree with Viktor an adjusted r² of 1 is not "luck to have a good model" it is a clear indicator of data leakage and a model that will absolutely overfit on new data!
To mediate this you need to understand your data and domain better (which means we need more infos to help).
But generally try to answer the following questions:

In any case I have not measured Y would I also be unable to measure X (the good variable)?

Is X a derivative value of Y? E.g. if you try to predict height in meters it does not make sense to include height in cm, you'd simply calculate it.

What is my use case? Do you want to explain relations / impact, do you want to predict something on new unseen data, etc.? Depending on the use case you have to deal with it in different ways.

It does not make sense however to keep this variable in your model because it invalidates the need for the model in the first place.
Either you could go ahead simply calculating Y from X and ignore all other variables (in case it is a simply unit conversion or a deterministic classifier like age > 18y --> adult) or you have to remove X to actually learn something about all the other variables (in case you want to explore impact / relation of other variables).
Also keep in mind that unless the linear regression of Y and the single good X is also r² = 1 you might have to remove or cluster two variables!


Recommendation Engine - Content based and Collaborative recommendation?

Not sure what your production requirements are for your collaborative filtering system, but from a research standpoint, I have had a good experience with the Surprise Package for Python.


Getting count of frequent itemsets in Python mlxtend

consider the master dataframe on which the apriori algorithm is applied as df1

  ls1=frequent_itemsets['support']*len(df1)
  frequent_itemsets.insert(loc=2,column='Count',value=ls1)

Now count is appended in your frequent_itemsets


When to remove outlier in preparing features for machine learning algorithm

Dealing with outliers requires knowledge about the outlier, the dataset and possibly domain knowledge. Given this, there are many options to handle outliers.

Without taking a look at your specific data, it could be that this outlier represents a total? Perhaps the data source you have included totals, which should be removed.

Generally, figuring out what to do with outliers requires investigating the outlier.

If the outlier is a data processing or entry error, it can generally be removed, or replaced with say, the mean (without the outlier).

If errors have been ruled out think about whether the outlier could be a legitimate value? There are no Right or Wrong answer.

Instead, documentation of your decisions is really important. I'd suggest running your models with the outlier, and without, comparing the results and document any decision to remove or transform the outlier.

Depending on your purpose, it should become relatively clear as to whether the outlier should be removed, eg if you're visualising the data, removing the outlier will likely have more a meaningful impact than retaining it.


Unable to reduce loss when doing sentiment analysis with Keras

There is no universal loss value result that you can or should achieve. Beside your model it will heavily depend on the data that you have.
What is good or bad result is subjective and depends on your use-case.
It's a very broad question how to improve the result of a neural network and you should be more specific but some ideas that can work generally:

Preprocess your data differently (i.e. use TF-IDF or use text as a sequence)
Collect more data (neural networks usually prefer that, also your test set seems small)
Use different optimizer (i.e. Adam)
Use different architecture (i.e. some RNN)
Use different algorithm (i.e. XGBoost)
Use BatchNormalization
Use some callback that modifies your learning rate
Tune your parameters


which Model to apply on panel data where unique id has 6-8 records and total records are 2,000,000?

Your data is a good candidate for a mixed effect model.
You have two potential random effects that look to be crossed: I see ID as one and state as another, as it seems that any ID can belong to any state.
Essentially, you can leverage the fact that some states and some IDs will have less data than others (and in general, some states and IDs will vary more than others). States and IDs who exhibit such characteristics will have their predictions bolstered by other states and IDs (called partial pooling). You also need to use time as a variable (perhaps convert quarter to 2019.25, 2019.5, etc. or use two time variables year, quarter) because you wish to predict the next period.
I think generalized linear mixed effect models can scale to 2 million observations. There are also machine learning methods that allow for mixed effects: mixed effect random forests (MixRF), mixed effect gradient boosting (GPBoost/mboost), and of course you can go full Bayesian as well (though you will likely need to use an approximation method since your data is large). The ML models may scale better.


Is there any way to read Xlsx file in pyspark?Also want to read strings of column from each columnName

Is pandas itself available on the cluster?
If so, you may try to go with the in-built read_excel().

You may also try the HadoopOffice library, it contains a Spark DataSource, also available as Spark Package, you can easily test it out without any installation:

$SPARK_HOME/bin/pyspark --packages com.github.zuinnote:spark-hadoopoffice-ds_2.11:1.0.4

Some people also recommend the Spark Excel dependency.


Why i get OOM error although my model is not that large?

1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?

Yes, the batch size is probably the reason.

Also, another the reason is that you don't use the second GPU at all (otherwise, both GPUs will split the batch - computation - and you could use larger batches).

  2) How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

By default, Tensorflow occupies all available GPUs (that's way you see it with nvidia-smi - you have a process 34589 which took both GPUs), but, unless you specify in the code to actually use multi GPUs, it will use only one by default.

Here is an official TF docs about how to use multi GPUs: https://www.tensorflow.org/guide/using_gpu#using_multiple_gpus

Here is some tutorial for multi-gpu usage with more examples: https://jhui.github.io/2017/03/07/TensorFlow-GPU/


High Recall but too low Precision result in imbalanced data

Given that both the f1-score and PR AUC are very low even for the prevalence of ~0.45%, it can not be deduced if the limitations are imposed by the nature of the data or the model (features plus the algorithm used).
In order to build a better understanding and to resolve the issue, I would suggest to break the problem into two parts:

Build a model that works for the selected features. For this purpose, you may try creating a slightly balanced dataset 80-20? both for training and testing. Once you are satisfied with the performance of your approach, move to 2 below
Use your original imbalanced dataset and see if the situation is any better. If not, it is now clearer that the issue is with the imbalanced nature of the data, and you should try all standard techniques for dealing with imbalanced classes. I hope this helps you, because otherwise only options at hand are synthesizing or collecting more data for the minority class.


Are you familiar with deep learning? If yes, I would advise you to one-hot encode these records and use a 2D convolutional neural network for prediction. Since these are mere numbers and there is no way to engineer new features, I'm afraid conventional ML methods won't work. And the records have only 4 unique values, so 2D-Convolutional NN can handle it. Treat them as a picture and let the NN capture the patterns and predict the result.

Best of Luck, Eden.


You are only defining the record function but never actually running it. Adding record() to the end of the code works for me in Google Colab and saves the audio to the audio.wav file.


Can neural network represent a relation that is a derivative?

Interesting question.

What you are asking for is if a neural network is able to represent and/or approximate functions.

The answer is: Yes, the Neural Networks can represent derivatives of functions.

In this paper, the possibility is explained.

Approximation of functions and their derivatives: A neural network implementation with applications

Other author take the same conclusion:

Multilayer feedforward networks are universal approximators

Generalization and Approximation Capabilities of Multilayer Networks

Approximation Capability of Layered Neural Networks with Sigmoid Units on Two Layers


How do you detect seasonality(multiplicative or additive) in a time series data?

Once divide the time-series by its Moving Average and once subtract the moving average from it.

If the seasonality is additive, then the result of subtract will have almost similar fluctuations in magnitude.

If it is multiplicative, then the division result has such a property.

Like the figures below from my course in Feature Engineering:

So the seasonality was multiplicative as the division has similar magnitudes of fluctuations.

Please note that more linear the dynamic of the time-series is, better this naive approach works.

Hope it helps!


Yes you are correct. Gradient descent (or various flavors of it) is the mechanism by which you find a local minima of your loss space using some learning rate. Backpropagation calculates the gradient of the error function with respect to the NN's weights and biases.


how to calculate quadratic features in computer vision Neural Network

Sorry for being so late in the response. I have just read your message.

The instructor is using $\frac{x^2}2$. So $\frac{2500\times 2500}2$, and this will get approx $3$ millions features.

BR


Time series prediction with Lstm on patients data

LSTM seeks inputs having the same shape. Hence, you may first consider defining a sequence length and then padding the sequences whose lengths are less than the defined sequence length.


How to make two parallel convolutional neural networks in Keras?

You essentially need a multi-input model. This can only be done through keras' functional api and can work with the pretrained nets in keras.applications. To create one you can do this:

from keras.layers import Input, Conv2D, Dense, concatenate
from keras.models import Model

1) Define your first model:

 in1 = Input(...)  # input of the first model
 x = Conv2D(...)(in1)
 # rest of the model
 out1 = Dense(...)(x)  # output for the first model

2) Define your second model:

 in2 = Input(...)  # input of the first model
 x = Conv2D(...)(in2)
 # rest of the model
 out2 = Dense(...)(x)  # output for the first model

3) Merge the two models and conclude the network:

x = concatenate([out1, out2])  # merge the outputs of the two models
out = Dense(...)(x)  # final layer of the network

4) Create the Model:

model = Model(inputs=[in1, in2], outputs=[out])


No.  What you want is probably not achievable in practice, because the approach you are considering obfuscates only the weights but does not obfuscate the inputs and outputs to the network.  In any reasonable ischeme I can imagine, the input $x$ to the obfuscated network will be known and under the attacker's control, and the output $y$ from the obfuscated network will be known to the attacker.  In particular, the attacker can pick any $x$ of his choice, and observe the output $y$.

If that is true, your goose is cooked.  The attacker can use the ability to invoke your neural network to learn his own neural network that is just as good as yours.  In particular, the attacker can assemble an arbitrarily large training set by picking many potential inputs $x$ and for each one computing the corresponding $y$ by running your obfuscated network on $x$, then putting the pair $(x,y)$ in the training set.  This doesn't require labelled examples; it only requires the adversary to be able to put together a large set of unlabelled instances (and then use your obfuscated network to label those instances).  Typically, it's not too hard for someone to create a large set of unlabelled instances.

Finally, once the adversary has assembled this training set, the adversary can train their own neural network using standard techniques.  It's likely that the resulting neural network will be approximately as good as yours -- i.e., have approximately the same accuracy.  (This is what seems to happen in practice.)

As a result, no obfuscation scheme for obfuscating the weights is likely to be terribly effective, because that doesn't hide the inputs and outputs to the network.  The most you can hope for is a scheme that acts as a "speed bump" that slightly increases the cost of de-obfuscation or that raises the bar a little bit, but nothing you do will provide strong security against a knowledgeable adversary.  So, don't spend too much time or energy or money on trying to make this work.  Instead, you might be better off looking for other ways to deal with this issue.

P.S. Even if you could hide the outputs and reveal only the ultimate classification (i.e., hide the continuous probability values from the softmax and just reveal the highest-probability class), that's probably still not enough.  Revealing the class is still enough for an adversary to label a bunch of instances, create a training set, and then train their own network.


Canny edge detection not working on Gaussian blurred images

You don't have to blur your image before passing it to Canny.
OpenCv's implementation already includes a blurring step. So by passing a blurred image, you're effectively blurring the image twice.  That will suppresses edges.
Read about openCv's implementation here.


The answer to your question is: yes, efficacy of a model depends on scaling. It's very important to scale your variables in the right range and combine them with the right activation function.

The reason is the following: the power of Neural Networks is due to the fact that they can learn any non-linear regularity of your data. This depends on the use of non-linear activation functions (tanh, ReLU, ELU, you name it). However, most of activation functions tend to behave in a non-linear way only around zero. Take the plot of a ReLU, for example. If you move further away from zero (in both directions) the function becomes very "linear" (i.e. its derivative is a constant).

All common activation functions tend to behave like this: non-linear (i.e. very powerful) in the locality of zero, and very linear (or flat) further away from zero. That is why all data are usually scaled in the [0, 1] or in the [-1, 1] range. In this way, activation functions can give their best, and Neural Networks can learn all the most complex patterns in your data.

When you work with CNNs, for example, most of pixel data come in the [0, 255] range. This is very bad for all activation functions, since between 0 and 255 pretty much any of them will look almost completely linear. In this way, your CNN wouldn't be able to learn much.


How to set the number of neurons and layers in neural networks

The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two dimensional cases in order to depict, I use 2-d space. I have used images from the works of a scientist. For understanding other nets like CNN I recommend you taking a look at here.

Suppose you have just a single neuron, in this case after learning the parameters of the network you will have a linear decision boundary which can separate the space to two individual classes.

Suppose that you are asked to separate the following data. You will need d1 which specifies the upper decision boundary and somehow is doing AND operation to determine whether the input data is on the left side of it or on the right side. Line d2 is doing another AND operation which investigates whether the input data is upper than d2 or not. In this case d1 is trying to understand whether the input is on the left side of line to classify the input as circle, also d2 is trying to figure out whether the input is on the right side of the line to classify the input as circle. Now we need another AND operation to wrap up the results of the two lines which are constructed after training their parameters. If the input is on the left side of d1 and on the right side of d2, it should be classified as circle.

Now suppose that you have the following problem and you are asked to separate the classes. In this case the justification is exactly like the above's.

For the following data:

the decision boundary is not convex and is more complex than the previous boundaries. First you have to have a sub-net which finds the inner circles. Then you have to have another sub-net which finds the inner rectangular decision boundary which decides the inputs which are inside of the rectangle are not circle and if they are outside, they are circle. After these, you have to wrap up the results and say if the input data is inside the bigger rectangle and outside of the inner rectangle, it should be classified as circle. You need another AND operation for this purpose. The network would be like this:

Suppose that you are asked to find the following circled decision boundary.

In this case your network would be like the following network which was referred to but with much more neurons in the first hidden layer.


Which learning algorithms to use in what order - dimensionality reduction, bayesian network structure, regression?

Is there any "known" approach to such a problem, i.e. a way to
  determine the formula that connects the initial variables to the later
  variables that determine the outcome, and then further determining the
  predictive power of the resulting network?

This is kinda a question about modelling approach as well as different types of models. I think that it is always best to start with simpler models, such as simple logistic regression. In this case you are trying to predict which runner will win, so that is a 0/1 outcome. So you can put all of your variables into a logistic regression model and see how it performs in terms of accuracy. If you have a lot of predictors, you can add a penalty term like L1 or L2. The simplest model serves as a baseline by which to compare more complex models, which might not always work better.

The formulation you describe seems to like a very common one. These are often called Latent Variable models or others might prefer to call them Bayesian network models. The idea is that we cannot measure a particular quantity directly, but we can use indirect measure to estimate them. So if your sense is that "talent for sprinting" and "current form" are better predictors for predicting who wins the race, then you can use a method for Latent Variables. I have not run those in a while but there are common algorithms like the EM algorithm or Markov Chain Monte Carlo sampling.

So by fitting the two different models you can see how much accuracy you gain by adding the latent variable structure versus making no assumptions about the structure. There are methods to try and estimate the bayesian network structure, but they often require a lot of data. In a case like this you might not have enough data to a neural net or something.

I don't think dimensional reduction would necessarily help in this case. You do seem to have time-series data, which you could include directly into the logistic regression equation. You might even be able to try an LSTM or sequence model if you have enough data.

Those are some ideas off the top of my head, but I think the idea of starting with simple models, benchmarking them, and then trying more complex models is the best way to learn and to avoid getting lost. Haha. Happy running.


is it possible to use the train result as another feature and retrain?

Yes but there are two issues:

Is it really useful? If the prediction is just added to the features on which it's based, it's unlikely to improve performance. However there are cases where this is useful, e.g. stacked learners.
You need to split the training set into two parts t1 and t2:

t1 is the regular training set used to produce the first model
t2 is the data used to predict the new "feature" and train the second model with this new feature.

It would be a bad idea to use the same training set for the two models because the predictions used in the second model would be obtained on the training set, so they would be unrealistically good.


Early stopping and final Loss or weights of models

EarlyStopping

The final weights will be saved, not the weights where your patience parameter is triggered.

Looking at the documentation for EarlyStopping, it seems not to be involved with saving weights at any point - it isn't mentioned.

EDIT

Upon further investigation (reading the source code), it seems you can indeed save the best, using the EarlyStopper callback

The class can be initialised with the aprameters restore_best_weights, as seen here. Then at the end of training, when your waiting period has overshot the patience parameter, the model's weights are returned to be the best weights (weights of the model at the time of loewst validation loss:

if self.restore_best_weights:
    if self.verbose > 0:
        print("Restoring model weights from the end of the best epoch")
    self.model.set_weights(self.best_weights)

It does this by tracking a chosen metric and comparing it to the recorded best value. By default this will be the validation loss. Check out the course code and it's description of the class there.

If these parameters don't work with your code, you will need to upgrade to the latest version (master branch from GitHub). This new parameter was only added 10 days ago!

ModelCheckpoint

If you want to use a callback to prevent overfitting, have a look at the ModelCheckpoint callback class. This has options to save the model weights at given times during training, and will allow you to keep the weights of the model at the end of the epoch specifically where the validation loss was at its minimum. This is selected using the the save_best_only parameter - use it like this:

from keras.callbacks import EarlyStopper, ModelCheckpoint

checkpointer = ModelCheckpoint(filepath, monitor='val_loss', verbose=0,
                               save_best_only=False, save_weights_only=False,
                               mode='auto', period=1)

# your early stopper as before
early_stopper = ...

All callbacks must be placed in a list:

my_callbacks = [checkpointer, early_stopper]

# pass to model along with other parameters as you did already
model.fit(..., callbacks=my_callbacks, ...)

If you don't use save_best_only, the default behaviour is to save the model at the end of every epoch.

You can set the filepath by using certain dynamic variables available during training, so that the filenames have some useful information:

E.g. filepath is weights.{epoch:02d}-{val_loss:.2f}.hdf5

  the model checkpoints will be saved with the epoch number and the validation loss in the filename.

Have a look at the linked documentation for how to use the other parameters of ModelCheckpoint.


Welcome to the site!

  Would my script be more efficient if I was calling my stocks one by
  one as vectors from there instead of picking them directly from
  RStudio?

Yes as you told, calling the specific columns from Database is better than extracting everything. I generally use Dataframes rather than vectors. Dataframes are very efficient and the transformation using Dataframes is much easier and better. You can go through this link, for better understanding on Vector Vs Dataframe.

Currently, I use 1 year data and consists of 100,000 records. Query takes like 8 minutes to extract the data in Rstudio using SQL DB and store it in an R-dataframe. After the extraction is over I don't hit the database meaning no read and write on the database by which the DB Server is not effected. After doing all the modeling and finally I commit the data into SQL DB to store the committed data(results).

  In other words, do queries call the whole dataset and then retrieve
  the requested values or do they follow a kind of shortcut to be memory
  efficient?

It depends on how you write the query, in my experience when I tried using temp-tables and is much efficient but when tried running that query in R but it wasn't supporting so had to stick to nested queries. If you don't write the nested queries in an optimized manner then it takes more time to extract the data(memory lekage).


Since I asked this question, it seems the documentation was expanded:

We need to provide a ".group" file. For example, the file could be

2
3

  This means that, the data set contains 5 instances, and the first two
  instances are in a group and the other three are in another group. The
  numbers in the group file are actually indicating the number of
  instances in each group in the instance file in order.


What inference can we draw from the frequency distribution of thresholds?

I'd say the model 1 is performing really well. If you can use different colors while plotting for positive and negative classes, then you should be able to see the difference. When you are trying to do binary classification, the distributions of negative and positive classes should be dipping in the mid region where as Model 2 is the opposite.


I would recommend using an Alluvial Diagram (Sankey Diagram). You can actually show multiple stages between emotions. So while if you google Sankey you will likely mainly see one two stage demonstrations...that is from A to B, you can easily code it to show from A to B to C, etc.


A weird result from a recommender system

Increasing the latent dimension value was the key here.

My recommendation system was implemented via ML.NET. And the framework's default setting for the latent dimension value was 20, which seems pretty small considering the volume of my training data.

Increasing the hidden feature count makes my system perform better, successfully predicting the X,Y,Z items as false candidates for other existing users except  Andy and Andrew. Below is how to set the value. This is based on the example code in ML.NET

var pipeline = mlContext.Transforms.Text.FeaturizeText(outputColumnName: "userIdFeaturized", inputColumnName: nameof(MovieRating.userId))
       .Append(mlContext.Transforms.Text.FeaturizeText(outputColumnName: "movieIdFeaturized", inputColumnName: nameof(MovieRating.movieId))
       .Append(mlContext.Transforms.Concatenate(DefaultColumnNames.Features, "userIdFeaturized", "movieIdFeaturized"))
       .Append(mlContext.BinaryClassification.Trainers.FieldAwareFactorizationMachine(
           new string[] {DefaultColumnNames.Features}
           , (e) => { e.latentDim = 200; }) // set custom value here.
           )
       );

To my best knowledge

When decomposing matrices, SVD extracts hidden features from the matrix of the training data and the hidden layers will be directly related to each users and a set of items in the middle of each entities, which is referred as dimension reduction here. I guess too small latent dimension generalizes the variety of recommendation items. That's why, I reckon, increasing the value solves my problem.

Any correction or comment would be appreciated. I definitely don't want to deliver false belief.


Calculating Univariate and MultiVariate Logistic Regression with Python

I case somebody is searching for the same thing.  The answer is fairly obvious.  The results as presented are a multivariate analysis.  To achieve a univariate analysis process a single variable at a time.


which algorithms can be used to extrapolate non-linear data?

That function can be decomposed into composite components:

General trend
Repeated frequencies (looks like there are 3 major cycles)
Random noise

A Fourier transform can decompose a function into its constituent frequencies.


Orange (Data Mining) : How to start using "Orange" from Python Anaconda Environment?

Instructions can be found on their Github page: https://github.com/biolab/orange3#starting-orange-gui


Help solving Bigram Model with the following probabilities

I think brute force is the only method. There might be some complicated reductions but that would be outside the scope of the tutorial.

Brute force isn't unreasonable here since there are only 46656 possible combinations. Even python should iterate through it in a couple of seconds.

This is straight forward tree-search problem, where each node's values is a conditional probability. The hardest part of it is having to manually type all the conditional probabilities in.


Updating 1st dataframe columns from 2nd data frame coulmns

Try the below code, it should do the trick for you

df1['Age'] = df1['Name'].replace(df2.set_index('Name')['Age'].dropna())
df1['Sex'] = df1['Name'].replace(df2.set_index('Name')['Sex'].dropna())
print(df1)


Can services such as AWS redshift or Google BigQuery allow you to
  somehow "import" data from many data sources and then you can perform
  SQL queries to join them?

It depends on your data and the type of joins you are performing. But, yeah, databases like Redshift can definitely perform better in your use case as they are column-based databases. Read this post and the associated answers for understanding how columnar data stores handle data.

  How about Hadoop and Hive?

Hadoop + Hive is mostly a DIY hosted/cloud version of what Redshift gives you on cloud.


Which approach for user classification on chat text (classifier, representation, features)?

You're asking what ML representation you should use for user-classification of chat text. User-classification is not the usual text-processing task.

It's not strictly necessary to semantically understand what the user is saying, only how they're saying it; so we look for telltale features indicative of a specific user. And we don't necessarily need to use, or solely rely on, the usual text-processing representations like bag-of-words, word-counts, TFIDF and word-vector.

Here are some features which are predictive of the user:

character length, word length, sentence length of each comment
typing speed (esp. if you have timestamps in seconds)
ratio of punctuation (e.g. 17 punctuation symbols in 80 chars = 17/80)
ratio of capitalization
ratio of numerals
ratio of whitespace
character n-grams (and notice these can pick up e.g. l0ser, f##k, :-) )
use of Unicode (emojis, symbols e.g. stars)
ratio of specific punctuation (e.g. how many '.', '!', '?', '*', '#' )
word-counts, esp. anything statistically anomalous, foreign, slang, insults
anything else you can think of that seems predictive for these two users, e.g. number of misspelled words per sentence (may be actual typos, or come from predictive swiping on a cellphone)


Which features are important in determining fluency of participants measured by weighted composite score?

This is a tentative answer.
One can try to determine first the relative importance of each feature (eg by Factor Analysis or Principal Component Analysis, ..)
Once the more significant features have been identified (or guessed perhaps), then one can try combinations of scores with these features.

Take simply the average as a composite score.
Do a linear regression and use the coefficients as the weights of the composite score.


Orange: Data has no target variable error

You can specify the target by choosing target in the feature column


Algorithm query for bank customer segmentation

So the question asks for other clustering algorithms which can be used in customer segmentation.
In a similar way to using Affinity propagation to identify the "optimal" number of customer segments, to gain more control over the number of customer segments, you can also use agglomerative clustering (https://www.datanovia.com/en/lessons/agglomerative-hierarchical-clustering/), which is a bottom-up clustering method. Here, you can see how customer instances cluster (using a dendrogram) and determine an "optimal" number of customer segments from this.


For outliers treatment, clipping, winsorizing or removing?

On the difference between winsorizing and clipping :

The techniques are very similar. They deal with extreme values (that are not necessarily outliers). Imo you should generally avoid thinking that big values = outliers. Solutions to deal with big values include normalizing your variables by a size factor for more comparability.
Which one should you use is linked to your data and their predictiveness. Sometimes data outside of a threshold will be predictive, sometimes not. Sometimes the underlying process will depend on rank, sometimes on global values. There is no rule for when to do one of those two techniques, there is no rule on which one to choose. In the end, you should choose which one gives you better results. Yeah, windsorisation seems a bit more 'adaptative' as you don't have to look at each of your predictors to choose a threshold, but outside of that, there is no reason it performs better statistically.
One of the main problems of those techniques is that they are univariate. If your predictors are correlated, modifying one value without another can breaks your relationships.
Yes, the answer is model dependent. Having extreme value may be detrimental for the convergence of methods based on gradient descent, as some data points may swing the optimization path. For methods relying on the partition of the space like trees, having extreme values is less of a problem.

Removing outliers is a completely different strategy. The difficulty is to identify outliers confidently... but that's another question.


It means that the model you are using fails at classifying the images. Logistic regression may not be the good one for your classification. Try using another model.


Model for Differing Number of Rows per Observation

You need to create a tidy version of the data with the on_amt and discount_rat encoded as a categorical variable (e.g., one-hot encoded). If they are continuous, they need to binned into categorical variables then encoded.


replicability / reproducibility in topic modeling (LDA)

LDA is Bayesian model. This means the desired result is a posterior probability distribution over the random vectors of interest (probability of topics etc. having seen some data).

Inference for many Bayesian models is done by Markov Chain Monte Carlo. Indeed the wiki on LDA suggests that Gibbs sampling is a popular inference technique for LDA. MCMC draws random samples to provide an approximation of the posterior distribution.

Variational inference methods should typically be deterministic, but I'm not too familiar with the VB inference for this particular model.

Also one can typically replicate runs of random algorithms by setting of random number generation seeds (if your purpose is scientific).

In either case if the results from a Bayesian model show huge variability in the the parameters of interest, it may be telling you that the model is not a good fit for the dataset, or the dataset is not big enough for the model you are fitting.

EDIT: I don't know which inference method (Gibbs, VB etc.) the backend of your software is using, so it's not possible to determine what type (if any) of randomisation is going on.

For scientific purposes, you'll probably want to read up some more on Bayesian inference. Standard software (e.g. the LDA in scikits.learn) will give you a summary of the outputs of the inference (e.g. most coders just want the best assignment of docs to topics). There's more information hanging around behind the scenes which you might be able to get access to, and could be useful.

E.g. (roughly) for scientific applications of Gibbs sampling methods we'd typically run multiple chains, drop the first N samples generated by each, and check if the resulting samples look like they came from the same distribution. If you're concerned about dependence on seeds etc. and your backend is the Gibbs sampler you will want to check out MCMC convergence diagnostics for this model.


Regression: Should "known outputs" be also activated?

The sigmoid function is usually used for classification tasks. Basically, it converts a continuous value into a binary variable (0 or 1) based on a certain threshold. An example is predicting breast tumor as malignant (1) or benign (0) based on whether or not the output is greater than the threshold value (typically, 0.5).

If you want to use ANN for regression, first you'll have to change your cost function into something more appropriate, like mean squared error (MSE). Then adapt the network to provide continuous output, by using a linear activation function, for example.

Source: https://www.quora.com/Can-Deep-Learning-and-Neural-Networks-be-useful-for-regression-problems-where-the-output-variable-has-an-unknown-or-varying-in-real-time-upper-bound


Why is it okay to set the bias vector up with zeros, and not the weight matrices?

As per Efficient Backprop from Lecun (§4.6) weight should be initialized in the linear region of the activation function. If they are too big, activation function will saturate and provide small gradient step to change those weigth. If they are too small they won't really impact the gradient and make the learning too slow.
Yes, if you choose the same weights this will create an artificial symetry that can be problematic. Here 'symmetry' is about neurons of the same layer having the same initial weights, thus being redundant. I think it would be clearer to speak about redundancy than symmetry. This will translate into redundant lines in weight matrices. Of course if all your weights are set to zero, all your weight matrices lines will be the same and you'll have horizontal symmetries in all your weight matrices.
Naturally you want to avoid redundancies in your model, but this is not the main problem. Your main problem is about solving an optimisation problem efficiently, i.e. having a gradient that is sufficiently big relative to your weights to help fast convergence. That's why you set your weights to small but not too small values. The randomness helps to avoid redundancies.
Once you have set your weights to random values, you have some minimal garantee you are in the linear region of the activation function, so you don't really need to add bias.


Would something like the following work?
import re
from typing import List

import pandas as pd

df = pd.DataFrame(
    {
        "codes": [
            "1234A, C (1AAA1), D, E",
            "34240 & 1 & 3",
            "PS1A;D;E",
            "CE11A, C & 2452D",
            "EOPSDCE11A, C & 2452D",
        ]
    }
)

def complete_codes(codes: List) -> List:
    line_code = codes[0][0:-1]
    res = []
    for code in codes:
        if len(code) < 2:
            res.append(line_code + code)
        else:
            res.append(code)
    return res

def clean_up(text: str) -> str:
    text = re.sub(r"\(.*\)", "", text)  # Remove parenthesis
    multi_split = re.split(r",|;|&", text)  # Split text
    multi_split = [item.strip() for item in multi_split]
    codes = complete_codes(multi_split)
    res = " ".join(codes)
    return res

df["output"] = df.codes.apply(clean_up)
print(df)

Which outputs:
                    codes                       output
0  1234A, C (1AAA1), D, E      1234A 1234C 1234D 1234E
1           34240 & 1 & 3            34240 34241 34243
2                PS1A;D;E               PS1A PS1D PS1E
3        CE11A, C & 2452D            CE11A CE11C 2452D
4   EOPSDCE11A, C & 2452D  EOPSDCE11A EOPSDCE11C 2452D


I faced almost exactly the same scenario a year and a half ago -- basically what you have is a variation of the one-class classification (OCC) problem, specifically PU-learning (learning from Positive and Unlabelled data). You have your known, labelled positive dataset (clients) and an un-labelled dataset of prospects (some of which are client-like and some of which are not client-like). Your task is to identify the most client like of the prospects and target them... this hinges on the assumption that prospects that look most like clients are more likely to convert than prospects that look less like clients.

The approach we settled upon used a procedure called the Spy-technique. The basic idea is that you take a sample from your known positive class and inject them into your unlabelled set. You  then train a classifier on this combined data and then run the unlabelled set back through the trained classifier assigning each instance a probability of being a positive class member. The intuition is that the injected positives (so-called spies) should behave similarly to the positive instances (as reflected by their posterior probabilities). By setting a threshold this allows you to extract reliable negative instances from the unlabelled set. Now, having both positive and negative labelled data, and you can build a classifier using any standard classification algorithm you choose. In essence, with the spy technique, you boot-strap your data to provide you with the needed negative instances for proper training.

For starters you should look into the work of Li and Liu who have a number of papers exploring the topic of OCC and PU-learning.


Seems like you overfitted.
I would advise you to get more data if you can and more importantly use a pre-trained model to encode the words to vectors. See for example the answer to this question.


How to approach the PlantVillage dataset?

I think this approach should work. So we need to clasify the plant and its disease. As mentioned, the disease maybe be associated with the plant's species.

Build a standard CNN or Siamese CNN for identifying the plant species. The model should be able classify the name/species of the plant even when it has a disease of some another plant ( The classification should not be baised upon the plant's disease ).
So, now, after the above model is ready, we build a 2nd model for disease classification. But the classification of the disease maybe be dependent on the plant species. So, for the 2nd model we will have 2 features namely, the image ( raw sample ) and the output of the 1st model ( plant name ). So, the 2nd model's prediction will be based on these two features.

The basic idea will be:

Model 1: input = image, output = plant name.
Model 2: input = ( image, plant name ), output = disease name.

So this is a system of models. That's easy if you are working with Keras or PyTorch.

Hope that helps.


What is the hypothesis space of decision tree learning?

As per Tom Mitchell's,

".....For example, consider the space of
hypotheses that could in principle be output by the above checkers learner. This
hypothesis space consists of all evaluation functions that can be represented by
some choice of values for the weights wo through w6. The learner's task is thus to
search through this vast space to locate the hypothesis that is most consistent with
the available training examples....."

Hence , Basically all possible combination of distinct trees makes the hypothesis space.

Lets say if you have chosen to represent your function to be a linear line then all possible linear lines which go through the data (given input, output) makes up your hypothesis space.

Each tree= Single hypothesis , that says this tree shall best fit my data and predict the correct results

therefore combination of all such possible tress= hypothesis space.

Here is the snippet of PPT from lecture


How handle the add of a new feature to the dataset?

What you ask for is known as Transfer Learning in the Machine Learning framework, so you might want to look more into that direction.

An interesting publication regarding Transfer Learning in Decision Trees is this.


Is Max Pooling and Conv used on anything else but images?

As @Aditya mentioned, we can use 1D Convolutions and Max Pooling for text classification as well. It has been used in sentiment analysis and gives quite good performance too. See here and here.

Another useful application is in signal processing. Classifying data from sensors of all kinds is a task for CNNs.

  You can develop a Human Activity Recognizer using 1D Convolutions. See
  here.

But, why not use an RNN instead of a CNN?

RNNs require a higher level of data preprocessing and have low inference speed if you are running them on a smartphone ( or any other IoT device ). CNNs are pretty fast in this case.

Audio classification using MFCC is performed using 1D Convolutional NN. See here.

  2D Convolutions are mainly used in image concerned ML tasks. They could extract spatial features from the 2D arrays ( an image ). In some cases, you can use them on 2D data which is not an image.


What are the best practices for data formatting?

This warning is trying to let you know that some of the columns of your DataFrame failed to resolve to a single data type (at least from pandas' perspective, it didn't know what to interpret the columns as!). Since it's a warning, pandas is happy to give you the DataFrame after finishing the read_csv but it's not advisable to proceed without addressing your mixed type columns in some fashion.

1) You can avoid this warning by explicitly identifying the dtype. The parameter in read_csv is conveniently called dtype. The usage specified by the doc is either single data type (e.g. dtype = str) or a dictionary of columns and data types (e.g. dtype = {0:str, 1:int}). Some options for the data type are:

the pandas specific data types (object)
the built in Python types (str, int, float)
or numpy data types (numpy.int32, numpy.float64)

The nice thing about specifying per column data types is that Python will throw a hard error if something breaks that type convention. It's a great, immediate check that your data is in a format you expect and it's always nice for Python to break loudly. However, it's possible you want to just read all your data in and then fix it in the DataFrame. You generally can get away with specifying dtype=str and pandas will (usually) perform the read_csv without complaint.

2) Since this question is unfortunately best answered as "it depends," I would advise looking into data cleaning methodologies (since that's what this problem largely is). Some specific (but generic) advice is: think hard about how you want to deal with your missing/malformed data, be explicit about the transformations you'll apply to the data to handle those cases, and keep track of the fact you've made those changes.

I hope this was helpful!


Recommender system that connect users with each other , should I go for content based or collaborative filtering?

You can use collaborative filtering with implicit features. However, I would first start with an even simpler approach. Maybe you could start using a distance metric, such as cosine similarity, or search for nearest neighbours using KNN.


You could try a "Mask R-CNN" The github page contains a keras model that could be trained by yourself. And there is also a link to an article how to train the model yourself step-by-step.


In this case, you might want to convert these into pair-wise relationships (e.g. item1 < item3), put together what you get from the different methods, and find a ranking which agrees with them the most.

You can look at the answer in the comment for some ideas - I would also suggest another paper that might help you: Aggregating inconsistent information: ranking and clustering


XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

As I understand it, iterations is equivalent to boosting rounds.

However, number of trees is not necessarily equivalent to the above, as xgboost has a parameter called num_parallel_tree which allows the user to create multiple trees per iteration (i.e. think of it as boosted random forest).

As an example, if the user set num_parallel_tree = 3 for 500 iterations, then number of trees = 1500 (=3*500) rather than 500.


How is Data Science related to Machine learning?

Data science is much broader concept than machine learning.
It starts from simple data visualization and descriptive statistics to get insights, manipulations like cleansing to prepare data. Before you can use some ML algorithms.

Basically such huge stacks as bigdata, visualization and data preprocessing are out of machine learning scope. And they are all integral parts of "Data Science".

Large resolution image:
https://whatsthebigdata.files.wordpress.com/2013/07/datascientistmap.png


Why Sckit's fit_transform causes a huge drop in accuracy and all other evaluation metrics?

Without seeing the actual data its kind hard to say.
I do have one speculation, when using the scaler on all data(before the train/test split) you create a data leakage.
Meaning, some of the data the model fitting should not see(the test set) is effectively included in the train set(because the scaler seen it and used if for setting the scales).
This data leakage can cause overfitting and thus lower accuracy score.
Trying doing the scale part on each train/test split(use fit_transform on the train and only transform on the test).
This is a better practice for model research and obviously closer to how the trained models will perform live(on not seen yet data).


ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

With np.isnan(X) you get a boolean mask back with True for positions containing NaNs.

With np.where(np.isnan(X)) you get back a tuple with i, j coordinates of NaNs.

Finally, with np.nan_to_num(X) you "replace nan with zero and inf with finite numbers".

Alternatively, you can use:

sklearn.impute.SimpleImputer for mean / median imputation of missing values, or
pandas' pd.DataFrame(X).fillna(), if you need something other than filling it with zeros.


How many times is backprop used in epoch?

It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "seen" all training data. Now to do so you have at least two options (let $n$ be the number of samples in your training data):

You can either run backprop after each forward pass of an example and update your weights accordingly. This is usually implemented as stochastic gradient descent (SGD), has a batch size of 1 and means you run backprop $n$ times per epoch.
Alternatively, you can first run the forward pass for all your training data and then run backprop and update your weights accordingly. This is called batch gradient descent, has a batch size of $n$ and means you run backprop once per epoch.

Usually a mix of these two is applied so that $1 < \text{batch size} < n$, which is called Minibatch Gradient Descent. Following the above explanations, it means you'll run backprop $\frac{n}{\text{batch size}}$ times per epoch.

This blog post provides a good explanation as well.


Replace Values in Vector on Specific Place in R

You have a data frame with 4 columns, each columns has 25 rows. You want to replace every 5th element with an outlying value (the mean of the column).

apply(x,2,function(i){
    i[seq(5,25,5)]=i[seq(5,25,5)]+mean(i)
})


printing theano.tensor without having any idea about the shape and type of the tensor

For evaluating and tensor, you have to pass the value of the input tensors as shown in the below code snippet.

import numpy
import theano.tensor as T
x = T.dscalar('x')
y = T.dscalar('y')
z = x + y
print(z.eval({x : 16.3, y : 12.1}))


What is "Policy Collapse" and what are the causes?

A web search for "policy collapse" "reinforcement learning" finds this question, a related one in stats.stackexchange.com and the comments section where you found the phrase. There are two other results on unrelated subjects where the words happen to appear next to each other. Then that's it - 5 results total from Google.

A google books ngrams search for policy collapse finds no references at all.

It is hard to prove a negative, but I think this is not a widely used term.

However, the comment does appear to be referring to a real phenomenon. That is where a reinforcement agent, instead of converging on the value functions for an optimal policy as it gains experience, actually diverges (and the parameters of the approximator will diverge too).

This can happen when using non-linear function approximators to estimate action-values. More generally, it tends to happen when you have the following traits in your problem:

Using a function approximator, especially a non-linear one (although even linear function approximators can diverge)
Using a bootstrap method, e.g. Temporal Difference (TD) Learning (including SARSA and Q-learning), where values are updated from the same value estimator applied to successive steps.
Off-policy training. Attempting to learn the optimal policy whilst not behaving optimally (as in Q-Learning).

In Sutton and Barto's book this is called the "deadly triad". If you do a web search for "deadly triad" "reinforcement learning" you will find many more results. It is an ongoing area of research how best to combat the effect. In the paper that introduced the DQN model learning to play Atari games, the researchers applied two things that help stabilise against the effect:

Experience replay, where transitions are not learned from immediately, but put into a pool from which mini-batches are sampled to train the approximator.
Bootstrap estimates are made from a "frozen" copy of the learning network, updated every N training steps - i.e. when calculating the TD target $R + \gamma \hat{q}(S', A', \theta)$, use this old copy of the network.

From the comment section you linked, it appears even applying these things is not a guaranteed fix and takes some judgement. In that case it was increasing the mini-batch size for experience replay that helped to stabilise an agent playing a variant of the video game Pong.


Is there any way to collect categorical features quickly in Julia DataFrames?

I think you can use the eltypes function in DataFrames.

categorical_indices = eltypes(X_train) .== String
categorical_columns = names(X_train)[categorical_indices]

This should provide a vector of Symbols for each of the categorical columns.


Scale out to between 0 and 1 before computing loss fucnction in keras

Use Lambda layer
def scale(tensor):
    tensor = (tensor - K.min(tensor)+1e-20)/(K.max(tensor)-K.min(tensor)+1e-10)
    return tensor

scaled_density_pred = Lambda(scale, name="scale_layer")(density_pred)


Can I add features that are parts of another feature?

First, I recommend NOT binning continous variables in any way. The amount of information lost and numerical problems makes it rarely a good decision. This is especially true in your case where you will be sacking a lot of degrees of freedom in your model by binning (which are necessary if you want statistical power, i.e. rejecting the null when you should). If you want to account for non linearities, either use splines or better yet, use an algorithm that doesn't assume a linear relationship between (some function of) y and your predictors.

Second, I would be extremely cautious with doing any sort of statistical inference (determining importance/significance of predictors) with machine learning algorithms that primarily (if not fully) were created for predictive, rather than explanatory reasons. In other words, I would stick with logistic regression in this case, since most ML algorithms don't allow for much, if any, proper statistical inference.

You are right to consider multicollinearity in this case. Severe collinearity will adversely affect p values and confidence intervals, and make inference hard, if not impossible. But this is why you need to have a hypothesis first as to what could possibly influence your target, to carefully specify your model, and to check the assumptions related to your chosen model (and fix or note any violations). Simply throwing a bunch of predictors into a model won't work here since you actually care about interpreting the model and its statistical output.

To be honest, I recommend going to the cross validated SE for more information rather than here, which primarily focuses on predictions, not inference.


Data leakage occurs whan you have in your training set information which you are not supposed to have. This might be, for example, having in the training set information contained in the test set. Generally speaking, data leakage can provide a model which is "too good to be true", but not to be trusted as it was build using "cheatful" information. Find more information here.


Influence of a data point on the regression result?

It is hard to quantify exactly what a single point is doing, but you would benefit from plotting data to look for suspicious points and then measuring differences in statistics if you remove it.

Don't forget about the value of plotting data! Anscombe's quartet is a great reminder of how important this is. https://en.wikipedia.org/wiki/Anscombe%27s_quartet


Which deep learning framework have support for gtx580 GPU?

Unfortunately you can't use GTX 580+Tensorflow.

TensorFlow GPU support requires having a GPU card with NVidia Compute Capability >= 3.0, while GTX 580 has only 2.0 compute capability.

However, caffe supports CUDA compute capability 2.0. You can have a try.

References

https://www.tensorflow.org/versions/r0.10/get_started/os_setup#prepare_environment_for_linux

https://developer.nvidia.com/cuda-gpus


Why does batchnorm1d in Pytorch compute 0 with the following example (2 lines of code)?

BatchNorm works along dim = 0. You might want to use LayerNorm instead.


How to get model attributes in scikit learn (not hyper parameters)

I believe you are trying to access "labels_" before fitting the data.
from sklearn.cluster import KMeans
import numpy as np
X = np.array([[1, 2], [1, 4], [1, 0],
               [10, 2], [10, 4], [10, 0]])
kmeans = KMeans(n_clusters=2, random_state=0).fit(X)

def get_properies(model):
  return [i for i in model.__dict__ if i.endswith(‘_’)]

get_properies(kmeans)

['n_clusters',
'init',
'max_iter',
'tol',
'precompute_distances',
'n_init',
'verbose',
'random_state',
'copy_x',
'n_jobs',
'algorithm',
'cluster_centers_',
'labels_',
'inertia_',
'n_iter_']


Targeted information extraction / focused extractive summarization

What about splitting the text into different paragraphs and select only the important paragraphs based on the keywords or patterns and then apply the summarization techniques only on the extracted paragraphs.


How to use a CNN to predict free space in a 2d environment with obstacles?

Instead of predicting free space, you can detect shapes/obstacles with a detector  or a mask builder and after determine free spaces locations.

In my opinion, this is a better approach to predict where are obstacles instead of predict where there are no obstacles (free space).


Handling categorical variables in linear regression and random forest

For linear regression, we have to do one hot encoding and it creates one less number of variables then levels of the categorical variable. In latest tools you don't have to do it manually it automatically does I have tried in R. One hot encoding has no impact on which level you choose, it is basically a binary way of representing the level True or False. This will create whole new variables which with just 1 and 0.
For tree-based algorithms, you basically don't need to apply the one hot encoding. These algorithms are capable enough to handle the categorical variables.

Reference: https://gerardnico.com/wiki/data_mining/dummy


Why is my CNN activation maximization split vertically?

I'm answering this myself as it may be helpful to others.
The error is in this line:
outputs = [o[0] for o in output]

I had copied and pasted this from dense-layer visualization code, where it is correct, selecting class 0 for visualization. Here, it is not wrong, but what it does is select the top row of that conv layer across all filters (and not the first filter of that layer). Hence the many small-scale features near the top row, more large-scale features near the middle, and nothing more near the bottom. I believe this may nicely visualize the perceptive field of the network.
For the record,
outputs = [o[:, :, 0] for o in output]

yields expected output.


Why normalize when all features are on the same scale?

For neural networks, there is another reason. Sigmoid function provides values between 0 and 1; if the task is binary classification, you'd use a sigmoid function at the output. For another task, you might have used tanh function at some layer, a centered input to that neuron works computationally well.

The reason is as follows; outside the range of some input values, the derivatives of the activation functions will be close to zero. At those points, the gradient descent steps will be extremely slow due to the small weight updates. Directly think of the 2-D function graph of the related activation function and think of the range of input values where the derivatives are close to zero. This is why Rectified Linear Unit (ReLU) or Leaky Rectified Linear Unit (Leaky ReLU) outperforms the others in most of the tasks, with the constant derivative of 1, when the input is greater than 1.

Also refer to: https://stats.stackexchange.com/questions/51012/must-i-normalize-inputs-into-a-perceptron-that-uses-a-sigmoid-activation-functio


Word embedding of a new word which was not in training

Generating word embeddings for "OOV"(out of vocabulary) words is one of the major limitations of many standard embeddings like Glove and word2vec. However, fastText circumvents this problem to some extent.

Instead of the traditional approaches which have distinct vectors for each word, they take a character n-grams level representation.
For  instance,  a  word  with n=  3,  will  be  represented  by  the  character  n-grams:

<wh, whe, her, ere, re>

and  the  special  sequence:

< where >

Here,<>are part of the n-grams.

$$
s(w,c) = \sum_{g\varepsilon G_{_{w}}} z_{g}^{T} v_{c}
$$
Here, $G$ represents the size of a dictionary of n-grams, and given a word $w$, then $G_{w}\subset \left \{ 1, ..., G \right \}$ represents the set of n-grams appearing in
$w$. They associate a vector representation $z_{g}$ to each
n-gram $g$ and represent a word by the sum of the
vector representations of its n-grams.

This helps them tackle OOV words via knowing some representations of a subword.
For an instance, an OOV word:
sechero

The 3-gram:

<se, sec, ech, che, her, ero, ro>

since, these 3-grams may have been encountered during learning, through other known words, like:

<se - section
che - cheer
ro> - hero

Hence, it can form at least some sensible embedding, instead of returning a useless <UNK>

Fastext in fact is an extension to word2vec, with majorly the feature explained above.


Why can't my neural network learn how to predict the squares of natural numbers?

Decrease the number of hidden layers; you can omit the dense layer with $50$ neurons. Furthermore, train your network more. You should also provide more data. It is not much at the moment.

Your current architecture is very deep for such a relatively easy task. Consequently, it needs more train time. You can just decrease the size of the current model by diminishing the number of hidden layers and neurons. For instance, use the following setting to see how you can train very fast and have a good accuracy.

model = Sequential()
model.add(Dense(20, activation = 'relu', input_dim = 1))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(1))


That would depend on the exact goal of the task and the specifics of the dataset, but in general I would say that it's always better to use the information specifically provided with the data if it's relevant for the task. In this case the rating for the product is indeed very likely to reflect the sentiment of the text, so I would go with it. Notice that you could also do both and compare the cases where the predicted sentiment differs from the one derived from the rating.

Given that the ratings are provided as 1-5 scores I would also consider the option of treating the task as a regression problem, instead of the standard binary classification setting.


GridSearchCV with Random Forest Classifier

You can definitely use GridSearchCV with Random Forest. In fact you should use GridSearchCV to find the best parameters that will make your oob_score very high.
Some parameters to tune are:
n_estimators: Number of tree your random forest should have. The more n_estimators the less overfitting. You should try from 100 to 5000 range.
max_depth: max_depth of each tree. You should specify certain max_depth so that your model don't memorise train examples.
min_sample_split: the minimum number of samples to have before splitting into new nodes.
and many more...
But these are the main hyperparameters we tune to get a forest that works well.And to get these hyperparams you should use GridSeachCV.


(Newbie) Decision Tree Classifier Splitting precedure

sklearn doesn't know that your feature is categorical; it's treating it as continuous, for which only splits of the form $x \leq \alpha$ are checked, so your second listed split candidate isn't actually a candidate.

In general, sklearn doesn't support categorical variables (yet?), and you'll need to encode it differently (one-hot?) if you want different behavior.

See also
https://datascience.stackexchange.com/a/52103/55122
how to make a decision tree when i have both continous and categorical variables in my dataset?


Is the multilayer perceptron only able to accept 1d vector of input data? If yes, why is this so?

You could say every type of neural network gets 1d input data. It's just more convenient to think about 2d-CNNs taking 2d data because the convolution operation is best illustrated by moving squares across a grid, and similarly for max-pooling.

But you could easily write out all the multiplications, additions, and max operations you performed in one line of algebra. And in this line you could easily flatten the 2d input indices into 1d indices. The information passed in and the calculations and output of the network would be exactly the same, but this representation would lose its real-world interpretation so there's no reason to think of it this way.

The point is that there are different ways of representing exactly the same information, and we choose a 1d representation for FNN inputs and sometimes higher dimensions for CNN inputs because it corresponds to the physical structure of the real-world problem and it is easiest think about. This is why most libraries have you input the data in these shapes. But if you think about what's happening at the lowest level of computation on your computer you won't necessarily find the same structure.


How can i increase the memory of Jupyter?

Assuming you cannot add more memory to your computer (or free up some of the memory), you could try 2 general approaches:

Read only some of the data into memory e.g. a subset of the rows or columns.

reduce the precision of the data from float64 to float32.

From your error, it looks like you are loading data into a numpy array, so somewhere in your code, you would need to add this argument to the array creation step e.g.  np.array(your_data, dtype=np.float32).
EDIT:
I don't think this is a rate limiting problem, or a max_buffer_size issue from Tornado (the library behind Jupyter).
You can try to see if your machine is actually running out of memory by using a tool called htop - just execute htop in a terminal (or first sudo apt install htop if it isn't already installed).  That shows the total amount of memory (RAM) available on your machine, it looks something like this:

This example shows the machine has 16 cores and 62.5 Gb memory - 6.14Gb of that is occupied.
Watch that view while you run your code. If the memory bar becomes full before the crash, you know you ran out of RAM.


Implementing spatio-temporal convolutions in pytorch

import torch as t

class Net_1(t.nn.Module):
    def __init__(self):
        super(Net_1, self).__init__()
        self.conv3d = t.nn.Conv3d(3, 8, (2, 3, 3))
        self.relu = t.nn.ReLU()

    def forward(self, x):
        x = self.relu(self.conv3d(x))
        return x

net_1 = Net_1()

video_1 = t.empty(1, 3, 6, 72, 108).normal_()

%%time
net_1(video_1).shape

Wall time: 18 ms
torch.Size([1, 8, 5, 70, 106])

class Net_2(t.nn.Module):
    def __init__(self):
        super(Net_2, self).__init__()
        self.conv2d = t.nn.Conv2d(3, 8, (3, 3))
        self.conv1d = t.nn.Conv1d(8, 8, (2))
        self.flatten = t.nn.Flatten(start_dim=2)
        self.relu = t.nn.ReLU()

    def forward(self, x):
        x = self.relu(self.conv2d(x))
        x = self.flatten(x)
        x.transpose_(0, 2)
        x = self.relu(self.conv1d(x))
        x.transpose_(0, 2)
        x.transpose_(0, 1)
        return x.view(5, 8, 70, 106)

net_2 = Net_2()

video_2 = t.empty(6, 3, 72, 108).normal_()

%%time
net_2(video_2).shape

Wall time: 58 ms
torch.Size([8, 5, 70, 106])


How to cluster government census data in order to group Metropolitan statistical areas

If you want to identify the distance between MSAs. Then yes, I think it would be best to first aggregate your features such that each instance (row) represents an MSA. From there you will have an $n\times m$ matrix where $n$ is the number of MSA, and $m$ is the number of features you end up with.
You can then apply your clustering algorithm, there are many to choose from, among my favorites I always try are:

K-means
K-nearest neighbors
Spectral clustering
DBSCAN

Others can be found here.
Once you train the clustering algorithm then you will get an associated cluster values for each of the $n$ rows in your input matrix. With this you will know what MSAs are similar in nature given the selected set of features.


Choosing between Regression and Classification

If you are trying to build a neural network you should stick with a the orientation as a numeric value since neural networks only accept numeric data as input.  If you want to use the class values, you will need to one hot encode them.  It will also result in a loss of information.

Of course, it depends what kind of neural network you are trying to implement. In most cases, I don't think this step is necessary and I think it may complicate the architecture of your model.

If you are open to suggestions, instead of building a regression model, I would investigate Principal Component Analysis.  It could alleviate the need to have the photographs labeled before input into the neural network.


In your first equation w is weight matrix $(w_1 + w_2 + \ldots + w_j)$ and a is the input vector $(x_1 + x_2 + \ldots + x_j)$. Writing wa in component form means taking the dot product of these two vectors, which is $\sum_{j}w_ja_j$. Now, you just apply the sigmoid function to get your final answer:$$\sigma (z) = \frac{1}{1 + \exp(-z)}$$ $$\Rightarrow\sigma (\textbf{wa} + b) = \frac{1}{1 + \exp(-\sum_{j}w_jx_j - b)}$$


Which metric should I use for classifying an imbalace data with fewer labels for the negative class?

The names of the classes don't matter, you might as well call them class A and class B. In binary classification the typical choice is to evaluate using precision, recall and F1-score. There are other options, but that depends on the task.
Assuming you choose F1-score, the choice of which class you select as the "positive" class for evaluation also depends on the task. Usually it's recommended to use the minority class because it's the most challenging one for the classifier.
The only problem here is the possible confusion of calling a class "negative" and calculating F1-score using it as the "positive" class, but that's just a naming issue. You could easily add this point to the explanations, or avoid any confusion by calling your classes A and B for instance.


How to improve LSTM accuracy on multiclass text classification?

50% is quite decent because you have five labels and random guessing model would have achieved only 20% accuracy. So you know your model is learning something.
The other thing you want to check out is whether this is suited to be a regression problem more than classification. For e.g, misclassifying a 5 (ground truth) into a 4 is better than misclassifying into a 2. Your current loss function treats both situations as equally bad. So, it's possible that your current accuracy measure is underestimating the models true performance.
You should also try out simple naive bayes model on top of bag of words. It will be interesting to compare how that compares with the more complex lstm.


How to evaluate sequence to sequence models?

Regarding your concern, there is no reason for you to choose only one evaluation metric. If there are several values that give you different views of the performance of the system, then compute all of these values. The evaluation should depend on your specific use case, so the important thing is that the values correlate with a good or bad performance of the system in the real world problem.

Even though I don't know exactly what your system is expected to do [written before question edit, see below for addition], maybe you could take as reference how performance is measured in speech and text recognition tasks. There you have a reference and a predicted sequence composed of characters and these characters form words. The length of the reference and the prediction is not necessarily the same. The performance is measured both at character and at word level (character error rate (CER) and word error rate (WER)). When measured at word level, even if only one character is wrong, the whole word is wrong. In both cases the main idea is to compute the levenshtein/edit distance (either at word or character level) between the reference and the prediction and then divide by the length of the reference.

In your example

Y=[["1","2","2"],["3","2","2"],["1","3","2","2"]]

Ŷ=[["1","3","2"],["3","3"],["1","3","2"]]

At inner bracket level the edit distances for the three inner brackets would be 1+2+1=4, since in the first a "3" is substituted by "2" (one edit) in the second a "3" is substituted by a "2" and a "2" is inserted (two edits) and in the last case again a "2" is inserted (one edit). The total length of the reference is 3+2+3=8. So the final performance measure is 4/8.

I am not sure if each of the inner brackets in your case are different samples (i.e., Ŷ is the whole dataset) or Ŷ is a single sample and you have something similar to characters and words. But anyway, if you do the edit distance between Ŷ and Y and divide by the length of Ŷ then you have 3/3, since all three brackets are different.

Additional response to question edit: If the task is number OCR, then you would compute character error rate. Computing an error value for each of the 10 digits is also possible with the edit distance. It would be by counting how many edit operations correspond to errors in each of the digits. For this in python you can use the editops function from the python-Levenshtein package (https://rawgit.com/ztane/python-Levenshtein/master/docs/Levenshtein.html#Levenshtein-editops).


What is the num_initial_points argument for Bayesian Optimization with Keras Tuner?

The Bayesian optimization algorithm selects points to test based on a balance between exploring uncertain regions and exploiting high-performing regions.  But before you've tested very many points, there's not much information to go on.  So, in this implementation you can specify a number of completely-at-random points to evaluate to start, and after that the actual Bayesian exploration begins.
Setting a high number of random points gives you guaranteed "exploration" points; indeed, in the documentation for the package bayesian-optimization, we find:

init_points: How many steps of random exploration you want to perform. Random exploration can help by diversifying the exploration space.

(The default is 5 in that package, and 3 times the dimension in keras-tuner.)
That said, you can also make the algorithm focus more or less on exploration/exploitation directly, using the beta parameter (kappa in bayesian-optimization, see this example notebook).


How to transform features order to make their distribution normal?

Just looking at a pairplot isn't enough to suggest changes, in my opinion. Moreover, what are you trying to test? What is your null hypothesis, what is the alternative hypothesis? What kind of data is it, how neat was your collection process? Are there any co-variables?

With the info provided, all I can suggest is to perform parametric and non-parametric tests. May try Box-Cox transformation!


Can you interpolate with QLearning or Reinforcement learning in general?

Since the convergence of QLearning is so slow I am wondering if it is possible with QLearning to interpolate the QValue of unexplored states since QLearning does not use a model?

When Q learning is described as "model free", it means that the agent does not need access to (or use) a predictive model of the environment. It cannot refer to state transitions and rewards in advance, but has to experience them in order to learn.

This does not mean that you have to avoid using a learning data model (such as a neural network) in order to generalise to new unseen data.

So, yes, Q learning can interpolate from unseen states and predict their Q value. To do this, you replace the state/action table with a supervised learning method based on descriptions of state $s$ and action $a$ as inputs, that you train as a regression model to predict $Q(s,a)$ (as a variant you can also have just state as input and predict $Q(s,a)$ for all possible actions as a vector in one go).

However, Q learning with a neural network suffers from instability. See Deep Mind's DQN paper for example of a system that solves that instability. In short:

Use experience replay - store S, A, R, S' data for each step and run the Q learning update on random mini-batches of the stored data, instead of online.
Keep two copies of the Q estimator neural network. Train one continuously, and copy it to a "frozen" version every now and then (e.g. every 100 mini-batches). Use the "frozen" copy to calculate the new $Q(s,a)$ targets.

This still might not match your learning scenario. If you want to solve mazes, think carefully about what data is truly available to the agent and how you might use it. For instance if you are using Q learning to solve a maze where you have a map, it is very inefficient approach. This is often shown as a toy problem, because it is possible to view the learning data very easily. But in that toy problem, the agent is not given the map, nor any knowledge of what a grid is.

Here are a couple of suggestions that may still help, separate to using a neural network value estimator:

If you do have a model of your environment (but not a map or other data that could be directly analysed for a solution), joining Q learning with a planning algorithm might work better for you than Q learning, as in Dyna-Q. This is relevant where you have an agent exploring in real time that would benefit from "looking" ahead before taking actions.
If your problem is very sparse rewards (due to larger maze, and only getting different reward at the end), then a worthwhile improvement is to look into multi-step TD learning, where the rewards are propagated back to previous steps more efficiently. Maybe look into $Q(\lambda)$


Modeling is a process where you invent hypotheses and then try to reject them. Without seeing your data, there is no guarantee any of these would go far. But here are some thoughts.

90 days of daily observations seem too few, which won't lead to much confidence no matter what model you use.

1.1 If you treat each user as one sequential observation, yes your dataset is sufficient for models like RNN. But ask yourself the question, are there users out there who are unique and different and you don't know about how they differ from the rest.

You seem to start with a belief that there are three types of users. I would suggest don't. Let the data tells you how many types are there. And for the purpose of predicting future usage, you don't care about this type information.
Recurrent neural net is a good choice if you believe the user behaviors are temporally dependent individually and have cross-sectional dependence. Otherwise, a time series model like ARIMA may do a better job and require less resources to fit.
Lastly, if you want to normalize the data, you should do so by subtracting each row by the row mean and divide by the row standard deviation. By adding the mean as the first observation, you're going to confuse the neural net and it's not smart enough to treat this data point differently. Unless there is a huge variance across users, I would start without normalization, fit a model, figure out some sort of out-of-sample MSE, then repeat on normalized data.

EDIT : There is no reason to train multiple RNNs on this set of data if you do not have a good segmentation argument.


XGBoost Log Loss different from GridSearchCV Log Loss

Your modelfit prints the training score, but GridSearchCV bases its decisions on the out-of-fold average (and in particular best_score_ is an out-of-fold average score).  This is an unfair comparison, and in particular your 0.577 is probably quite optimistically biased.


Time-series multi-step generalization from single step model

Welcome to the community Fra, below you can find a worked out example implementing a multivariate several input features (as I think is your case) time series forecasting, predicting multiple future steps (multi-step forecast), applying bayesian hyperparametrization.
It is based on a from simpler to more complex approach, so you can see there are few layers in the architecture to begin with, and what is being hyperparametrized is the units in such layers. This is a whole worked out example, in case you can almost directly apply it and try out modelling first with a single output node for a single-step prediction, and then add more future step predictions to check if you still run on the same issue as you described (beggining with n_output = 1 and then incrementing this, e.g. in my case it is 72 future values to predict).
For a dataset of this type:

and trying to predict several future time steps (the deep blue dots) for the temperature values:

we have some code like:

train-validation dataset splitter functions:
  def multivariate_data(dataset, target, start_index, end_index, history_size, target_size, step, single_step=False):
    data = []
    labels = []

    start_index = start_index + history_size
    if end_index is None:
      end_index = len(dataset) - target_size

    for i in range(start_index, end_index):
      indices = range(i-history_size, i, step)
      data.append(dataset[indices])

      if single_step:
        labels.append(target[i+target_size])
      else:
        labels.append(target[i:i+target_size])

    return np.array(data), np.array(labels)

This is what we get by applying that function (in your case, you do not need to apply the sampling interval of 6 timesteps, you can set it to 1; this is in case you want to downsample a maybe too long history):

function implementation to build the train-eval sets:
  past_history = 720 # historic values to consider for each prediction
  future_target = 72 # number of future values to predict
  STEP = 6 # sampling frequency (it makes taking 720/6 historic points for each sample)

  x_train_multi, y_train_multi = multivariate_data(dataset_standardized, dataset_standardized[:, 1], 0,TRAIN_SPLIT, past_history, future_target, STEP)
  x_val_multi, y_val_multi = multivariate_data(dataset_standardized, dataset_standardized[:, 1], TRAIN_SPLIT, None, past_history, future_target, STEP)

This way you have:

Single window of past history : (120, 3)
Target temperature to predict : (72,)

For the training phase, we define the batch size and other values and build our training and validation sets:
BATCH_SIZE = 256
BUFFER_SIZE = 10000

train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi, y_train_multi))
train_data_multi = train_data_multi.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()

val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))
val_data_multi = val_data_multi.batch(BATCH_SIZE).repeat()

and defining a simple architecture to begin with, we have:
multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                          return_sequences=True,  # "Boolean. Whether to return the last output. in the output sequence, or the full sequence"
                                          input_shape=x_train_multi.shape[-2:]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
multi_step_model.add(tf.keras.layers.Dense(72))

multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')

for which we can check the performance on the validation set as:
EVALUATION_INTERVAL = 200
EPOCHS = 10

multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,
                                          steps_per_epoch=EVALUATION_INTERVAL,
                                          validation_data=val_data_multi,
                                          validation_steps=50)

having the predictions in red below, after de-scaling the values to its original scale:
for x, y in val_data_multi.take(3):
    x_denormalized = (x[0])*temps_std+temps_mean
    y_denormalized = (y[0])*temps_std+temps_mean

    multi_step_plot(x_denormalized, y_denormalized, (multi_step_model.predict(x)[0])*temps_std+temps_mean)

and maybe the most interesting part for your use case, you can try implementing hyperparametrization of some params as hidden layers units as follows with the keras bayesian tuner:
from kerastuner import BayesianOptimization

def build_model_2(hp):
    model = keras.Sequential()
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=16,
                                        max_value=32,
                                        step=16),
                                        #activation='relu',
                                        return_sequences=True,
                                        input_shape=x_train_multi.shape[-2:]))
    model.add(keras.layers.LSTM(units=hp.Int('units',min_value=16,
                                        max_value=32,
                                        step=16),
                                        activation='relu'))
    model.add(keras.layers.Dense(72))

    model.compile(loss='mae', optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0),

                   metrics=['mae'])
    return model

# define model
bayesian_opt_tuner_2 = BayesianOptimization(
    build_model_2,
    objective='mae',
    max_trials=3,
    executions_per_trial=1,
    directory=os.path.normpath('C:/keras_tuning'),
    project_name='timeseries_temp_ts_test_from_TF_ex_multivar_multistep_2',
    overwrite=True)

EVALUATION_INTERVAL = 200
EPOCHS = 10
bayesian_opt_tuner_2.search(train_data_multi,
             epochs=EPOCHS,
             validation_data=val_data_multi,
             validation_steps=50,
             steps_per_epoch=EVALUATION_INTERVAL)

# and selecting the best model:

    best_MULTIVAR_MULTISTEP_LSTM_model_2 = bayesian_opt_tuner_2.get_best_models(num_models=1)[0]

we have our new model predictions:

which gave an improved MAE on the validation set.


Prevent overfitting when decreasing model complexity is not possible

is this about trying to have the model learn the sum function? Because if it is you can always initialize the weights to be 1 and then make the entire model untrainable.
If all weights are one and the model has linear activations it will just compute the sum of your inputs. If its important to you to train the model to do this I would add a Dropout layer (?).

Your model is so simple it can be written down in closed form as it is $\sum_{i=0}^{255*350}w_i x_i$ and therefore I would just set the $w_i$ to be 1.

(In general initializing the weights to 1, the ideal solution, should make training very fast)


Understanding output probabilites of xgboost in multiclass problems

What you were told is a worst case scenario. With 5 labels, 20.01% is the lowest possible value that a model would need to choose one class over the other. If the probability for each of the 5 classes are almost equal then the probabilities for each would be approximately 20%. In this case, the model would be having trouble deciding which class is correct.
About the 30% comment, while it seems low, keep in mind that the model mainly uses the probability to classify the most likely label, calculating a highly accurate probability value is not the first priority.
Here is a good reference. This paper discusses a few of the problems with probability accuracy and mentions some theories on how to improve the accuracy:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.381.4254&rep=rep1&type=pdf
It mentions (see excerpts below) how a probability estimation in classification is a complex and difficult problem, and that some models can be very accurate at classification but poor at probability estimation, and how multi-class models are much more difficult than a binary classification model.
Excerpt 1:
"Probability estimation is known to be a quite hard problem, especially in comparison to standard classification. This comes at no surprise, noting that proper probability estimation is a sufficient but not necessary condition for proper classification: If the conditional class probabilities (1) are predicted accurately, an optimal classification can simply be made by picking the class with highest probability"
Excerpt 2:
"More generally, the Bayes decision can be taken so as to minimize any loss
in expectation. On the other hand, a correct classification can also be obtained
based on less accurate probability estimates. In fact, the classification will remain correct as long as the estimated probability is highest for the true class. Or, stated differently, an estimation error will remain ineffective unless it changes the result of the arg max operation in (2). This is also the reason for why methods like naive Bayes show competitive performance in classification despite producing relatively inaccurate probability estimates [7]."
Excerpt 3:
Methods like naive Bayes and decision trees are multi-class classifiers and
can in principle be used to produce probability estimates in this setting. In
practice, however, one often prefers to estimate probabilities in the two-class
setting, especially because estimating a single probability (of the positive class) is much simpler than estimating K − 1 probabilities simultaneously
Example with probabilities close to 1:
from sklearn import datasets
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
iris = datasets.load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)
from sklearn.tree import DecisionTreeClassifier
dtree_model = DecisionTreeClassifier(max_depth = 2).fit(X_train, y_train)
dtree_predictions = dtree_model.predict(X_test)

print(dtree_model.predict_proba(X_test) )
[[0.         0.02564103 0.97435897]
[0.         0.91666667 0.08333333]
[1.         0.         0.        ]
[0.         0.02564103 0.97435897]
...


Predicting future order dates and order amounts for customers with online retail data

To answer your points in order:

I can generally understand what you are trying to do. However, a few considerations you should bear in mind:

1.1. Given that you are trying to make time series predictions for all customers, then you are most likely interested in a panel data modelling solution - i.e. one that takes into account the fact that the data is both cross-sectional (at one point in time) and time series (across several time periods). The alternative is simply running a separate time series for each customer, which could prove to be highly inefficient.

1.2. This would be considered a multivariate time series forecast - provided that price is in fact dependent on variables other than itself, i.e. quantity, country, etc. Are you planning on incorporating these as explanatory variables into a time series model?
Hard to say how many models should be used off the bat, but I might be inclined to construct a single panel data model for this case - you could always construct it on a subset of the data as a proof of concept and then work from there. Construction of multiple LSTM models would appear too much effort for too little reward (in terms of forecast accuracy).
I would be inclined to take a look at linearmodels - it is a Python library designed for working with panel data.
These techniques are used for representing categorical data - you might find the following answer useful.

Hope this helps, and please let me know if I can offer any further clarity on the above.


Binary classification problem with imbalanced dataset, how to compare to random classifier

Since you are interested in different decision thresholds, your random model should produce scores.  In that case, a reasonable base-line model assigns a score uniformly at random in $[0,1]$.  Such a model will, at threshold $t$, have

$$\begin{align*}
\operatorname{precision} &= \frac{2\%\cdot N\cdot (1-t)}{N(1-t)} = 0.02,\\[1em]
\operatorname{recall} &= \frac{2\%\cdot N\cdot (1-t)}{2\%\cdot N} = 1-t.
\end{align*}
$$

(Perhaps a very simple model will serve as a better baseline.)


You do want to model the traffic, at least over a work day, otherwise it wouldn't matter what time you traveled! Absent any data, I'd assume there isn't much variance over the working week, but that's one thing the data will quickly confirm or refute. If it is varying, you can use a different model for each day.

You have two variables; the departure times from home and work, respectively. Let's call them t_h and t_w. Let's call the commute time T_c(t), where t is the time of day. You can estimate this function from the data, so I'll assume it is given.

You want to maximize c t_h - (1-c) t_w subject to the constraints t_h + T_c(t_h) < 9.5 and t_w > t_h + T_c(t_h) + 8

where c is a constant you can set to adjust the relative importance of leaving home early relative to leaving work early. You should be able to solve this numerical optimization problem with Mathematica, MATLAB, or something similar. I would not recommend Java; it's not meant for this. The only tricky part is estimating T_c. You know that it's a non-negative function, so you could use the standard trick of estimating it's logarithm (say, with kernels) and exponentiating. For implementation with Mathematica see Smoothing Data, Filling Missing Data, and Nonparametric Fitting and Constrained Optimization.


There is no definite source on how to do feature engineering. It is often dependent on the problem you are trying to solve. Some say it is more of an art than it is science.

But I would go through some of the high scoring kaggle kernels / winning solutions if available. Just head over to kaggle and browse through the competitions. There is a lot of very useful material in there.

Also the journal of machine learning research has a lot of papers about feature engineering. Just search on their site http://www.jmlr.org/.

The following links are useful and to long to paraphrase:

Some information about some best practices of feature engineering can
be found on Quora, see this
link
At machine learning mastery there is this
page
about feature engineering.


Price optimization for tiered and seasonal products

You should be able to use linear regression to find correlation between the factors which cause your products to sell better (or worse).

There are many correlations you can test against in this data set. Some examples are:

If a product has been marketed aggressively, does it sell more quickly?
If a low tier item is available, do fewer high-tier items sell?
If multiple high-tier items are available, are fewer sold of each item?

Keep in mind that correlation does not necessarily imply causation. Always think about other factors which may cause sales to go up and down. For example, you may sell more high tier items in a season one year than another year. But, this could be due to changes in the overall economy, rather than changes in your pricing.

The second thing you can do is perform A/B tests on your product sales pages. This gives you clear feedback right away. Some example tests could be:

Show the user one high-tier product and one low-tier product (A). Show the user two high-tier products and no low-tier products(B). Which page generates more revenue?
Send out marketing emails for a seasonal sale 5 days in advance to one group of users (A). Send the same email to a different set of users 1 day in advance (B).

There are many possibilities. Use your intuition and think about previous knowledge you have about your products.


Standardization, Oversampling and Evaluation

Let's discuss the second variant:

If a standardization is considered as a function f(data) and the second standardization after oversampling is considered as a function g(f(data)), then you should use the same transformation for your test data. However, in the second option you just implement the function g(data) without transformation the data using function f().

Therefor, it seems to me, that the first variant is better. Moreover, it is enough to standardize data just once, so the first option is ok.


Really depends

Why? updating everything in production (pre-processing, fitting etc) can get extremely expensive. If you have some complex architecture it is not worth it.

Alternatives

Approximate covariate shift if you know distribution of your future data you can adjust all your, for example normalisation parameters, in advance.
Save your you future data every time you make prediction, it could be cheaper to quickly save your data in DB and depending on your system do updates weekly,monthly


I do biological network analysis and here are some networks.


Recommendation for boolean dataset with apache mahout

The code and data combination is too small for predicting a solution. But note that you will not get any answer if there are no recommendations made. In this case there will be no exception either. Looks like your code

List<RecommendedItem> recommendations =
     recommender.recommend(userid, noOfRecommendations);

returned no recommendations. You should try with a variety of data.