Semi Supervised Learning without label propagation

One strategy that seems good here is instance-level constrained clustering. These methods are semi-supervised algorithm that have "must-link" and "cannot-link" constraints between instances of known labels. So in your example, you would bind the 4 pairs (red, blue), (red, yellow), (blue, yellow), and (man, woman) as "must-link", and the 6 pairs (red, man), (red, woman), ..., (yellow, woman) as "cannot-link".

The results are similar to unsupervised clustering. For example, if you were to use DBSCAN (ignoring the labels/constraints), you would not need to specify the number of clusters/groups you're trying to achieve, and the algorithm would even find "outliers".

In fact, there is a version of DBSCAN that supports instance-level constraints, called C-DBSCAN. It is described in the work "Density-based semi-supervised clustering" by Ruiz et al (2010).

I do not know of any out-of-the-box implementations available, but I have a working version of C-DBSCAN I implemented for an experiment. However it is not documented nor is it performant/production level. You can find it at my lab's repository if you're interested (also contains the C-DenStream, which is the data streams version of it, but it does not seem to fit your problem).


Actually, it depends on the boosting algorithm you used.

In the original boosting algorithm (Schapire 1990), three classifiers are used (say $C_1$, $C_2$ and $C_3$). The training dataset is randomly divided into three subsets, $D_1$, $D_2$ and $D_3$. The training process is as follow:

Train $C_1$ using $D_1$, and use $D_2$ to test $C_1$
The instances in $D_2$ which misclassified by $C_1$ (say m instances) are taken as training data for $C_2$ along with m instances that are correctly classified by $C_1$
Train $C_2$ using the 2*m instances taken from step 2
use $D_3$ to test both $C_1$ and $C_2$
Train $C_3$ on the instances that $C_1$ and $C_2$ disagree with each other (namely, the instances that $C_1$ and $C_2$ predicted differently)

For Adaboost, the training subsets are selected by sampling from the whole training set with probabilities.

At the beginning of running the algorithm, the probabilities are uniform. After a classifier is trained and tested, the probabilities of each successfully classified instances are decreased. Then, all the probabilities are normalized after adjustment. The next classifier is trained using dataset sampled with the adjusted probabilities.

reference: Ethem Alpaydin: Machine Learning


How can I get my trained model ready for production

You should first cross validate your pipeline, making sure that you get an homogeneous y_pred result.

Then you can retrain a model with the same parameters on your full dataset. Pickle the model as well as any preprocessor tools, and reuse them to predict on new data.


What are some recommended visualisations to represent how attributes in a data-set influence each other?

Using seaborn in python, I usually plot the pairwise distributions with sns.pairplot. seaborn provides a wide variety of data exploration tools in a high level API, it is quite powerful for data exploration.

Here's what sns.pairplot looks like on the popular iris dataset:

Sometimes plots like the following (a specific kind of cobweb plot) can be useful too.

Those represent the distribution of each parameter on each vertical line, with a web of links joining values for a single entry in the dataset.


How Natural language processing and elasticsearch are related

Natural language processing (NLP) is a field of computer science, artificial intelligence, and computational linguistics.
ElasticSearch is a search engine based on Lucene.

ElasticSearch uses parts of NLP (e.g., tokenization and stemming). ElasticSearch also includes software engineering elements to ensure a search solution is performant.
Since NLP is entire discipline, there are many parts of NLP that are not related to ElasticSearch.


Can Precision-Recall be improved for imbalanced sample?

I am not sure if you are asking how to improve the power of the models or how to better represent the effectiveness of the models on the minority class.  These are information retrieval measures that are sometimes adequate in other circumstances.  If you are working on an information retrieval process, this is probably fine.  If it is more of a statistical analysis you are working on, perhaps sensitivity (which is the same as recall) and specificity would be better metrics to describe your results.  Maybe your model is useful in that it can rule out the negative. These metrics can also be evaluated relative to the prior probability of the minor class in the data.

Sensitivity and specificity have an additional benefit if you are trying to communicate your results with an audience that reviews a lot of medical test results as these measures are used most frequently in that field.

HTH


doubt in improvement of performances given by a layer in a deep neural network

Yes, there is a threshold where additional layers are too much.
Look at  residual networks


how to impute missing values on numpy array created by train_test_split from pandas.DataFrame?

Can you just cast your np.array from train_test_split back into a pandas.DataFrame so you can carry out your same strategy. This is very common to what I do when dealing with pandas and scikit. For example,
 a = train_test_split
 new_df = pd.DataFrame(a)


What statistical method should i use to find Correlation between number of days and AmountEarned

Whether to use pearson's correlation coefficient or Spearman's correlation coefficient depends on what you want to measure ?
Pearson's correlation coefficient measures the strength of a linear relationship between two variables. The variables should be on an interval or ratio scale. Ordinal variables cannot be used in pearson's test. The variables should satisfy certain assumption before the pearson's product moment correlation is applied. They are:

The variables should be approximately normally distributed
There should be no significant outliers in the data
The variables should have a linear relationship

In case, the variables do not satisfy these assumptions they can either be transformed to meet them or we can use the spearman's rank correlation coefficient to measure correlation.
The Spearman's method is primarily used when the data can be ranked, therefore correlation between ordinal, interval and ratio variables can be measured using this method. Also, this method measures the strength of a monotonic relationship between two variables.


What does localist one-hot vector mean in cs224n NLP course?

One-hot vector is called "localist" because it contains information only about a single data point, and does not give clues about other points, in contrast to a distributed representation (e.g. result of an embedding algorithm) that contains information about other data points too.

For example, one-hot encoding of two animals and two flowers is as follows

dog   = [1, 0, 0, 0]
cat   = [0, 0, 1, 0]
rose  = [0, 1, 0, 0]
tulip = [0, 0, 0, 1]

Distance between any pair is the same, and effectively by knowing the vector of "rose" we gain no information about the related "tulip" or unrelated (or less related) "dog" and "cat", hence the name "localist".

But if we happen to find a distributed representation like

dog   = [1.0, 0.1]
cat   = [1.1, 0.2]
rose  = [0.0, 1.2]
tulip = [0.1, 1.3]

The vector of "rose" alone gives us clues about other flowers like "tulip", which are closer, and also about other entities like animals, which are farther, hence a name like "globalist".

Two notes:

Being "localist" is not a zero / one thing. The more clues a representation gives about other data points, the less it is localist.
The notion of "relatedness" is task dependent. For example, in the task of "grouping things with similar colors", a red car is more related to a red rose than a blue car.

Here is a random image of those entities for fun!

If all pairs look equally different, your brain works with "localist" vectors, thus has no clue ;)


Feature has a pattern in relation to class but does not enhance classifier to predict class

There are multiple things at play here.

First of all, if your training and testing data are different then it becomes difficult to learn to generalize, because it will learn the relationships in your training data and if those relationships are significantly different then it will learn the wrong things. It could be that the data density is just different in your testing set which is not as much of a problem.

Second of all, if the relationship that you see is not linear, your logistic regression might not be able to pick up the relationship due to the underlying linear assumption.

Third of all, maybe other features are highly (co)correlated and adding the new feature does not add any signal but does add the potential for fitting additional noise, thus explaining the performance reduction.


Can I test my already trained model using the data it was trained on?

It is possible to re-use the train+test dataset.

You used the train dataset to train your parameters and the test to check if there was or not overfitting in your model.

The train/test split is used to check your if your model is likely to suffer from overfitting or not, more than to obtain the "real parameters".

So, in that order of ideas, you discarded your model's overfitting and then you could use your complete dataset to recalibrate your parameters.

Remember the decision of wheter an individual belongs to train or test is arbitrary.


You can save the encoding and use them to encode the new data. Only thing to make sure is not to have new data which was not there in saved encoding. You can save them as pickle files if you are using Python.


Does a Keras checkpointer save the best weights when using chekpoints with restore_best_weights?

It will if you set the save_best_only flag in your checkpoint callback definition:

ModelCheckpoint(filepath, monitor='val_loss', save_best_only=True)

From the docs:

  save_best_only: if save_best_only=True, the latest best model
  according to the quantity monitored will not be overwritten.


It might be, or it might not. It depends on how it performs.

The key to a wrapper in the more general sense is that it provides for the interoperability of two distinct classes of object which are not inherently able to work together.

In the specific case you are talking about, simply creating a script to scrape the web is not necessarily a wrapper. Once you format the xml into a recognized format such as JSON, pandas dataframe or  in a SQL schema that your program can operate over, it becomes a wrapper.

The wrapper 'repackages' the data to work in your current system's class transparently so that you do not see the underlying nature of the incompatible structure.


Forecasting one time series with missing data with help of other time series

Not a fully complete answer, but some inputs.

Your time series are correlated.
I assume that the measure you want to forecast for a region is an aggregation of units forecasts.

To address the first point, I usually use Vector Autoregressive Model (VAR) that forecast all time-series at once (each one being expressed as a regression using the others)
The second point involves the concept of hierarchical forecasting and reconciliation. You can exploit the fact that the regional forecast should/must equal the unit- forecasts. There can be a process to adjust forecasts to take that into account.

There are both packages for VaR and hierarchical reconciliation in R but as far as I know no direct code to handle both at the same time...

You may find this paper providing some details on the proposed approach:
https://mpra.ub.uni-muenchen.de/76556/1/MPRA_paper_76556.pdf


We already know that the images from the generator are fake and those from our training set are real. So, we assign them labels manually, real images as $0$ and fake ones as $1$, which gives us labelled data. Check the definition of loss function in any implementation of a GAN and you'll find that we assign the labels ourselves. For example, take a look here.


Number and size of dense layers in a CNN

First of all:

  There is no way to determine a good network topology just from the
  number of inputs and outputs. It depends critically on the number of
  training examples and the complexity of the classification you are
  trying to learn.[1]

and Yoshua Bengio has proposed a very simple rule:

  Just keep adding layers until the test error does not improve anymore.[2]

Moreover:

  The earlier features of a ConvNet contain more generic features (e.g.
  edge detectors or color blob detectors) that should be useful to many
  tasks, but later layers of the ConvNet becomes progressively more
  specific to the details of the classes contained in the original
  dataset.[3]

For example, in a method for learning feature detectors:

  first layer learns edge detectors and subsequent layers learn more complex features, and higher level layers encode more abstract features. [4]

So, using two dense layers is more advised than one layer.

Finally:

  The original paper on Dropout provides a number of useful heuristics to consider when using dropout in practice. One of them is:
   Use dropout on incoming (visible) as well as hidden units. Application of dropout at each layer of the network has shown good results. [5]

in CNN, usually, a Dropout layer is applied after each pooling layer, and also after  your Dense layer. A good tutorial is here [6]

References:

[1] https://www.cs.cmu.edu/Groups/AI/util/html/faqs/ai/neural/faq.html

[2] Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural networks: Tricks of the trade. Springer Berlin Heidelberg, 2012. 437-478.

[3] http://cs231n.github.io/transfer-learning/

[4] http://learning.eng.cam.ac.uk/pub/Public/Turner/Teaching/ml-lecture-3-slides.pdf

[5] https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/

[6] https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html


Yes, it is possible. Provided the dataset contains the objects that you actually want your model to detect during run time.


It sounds like the chi-squared test confirms your suspicions about age being correlated with the response.

As far as I am aware, the null-hypothesis for the chi-squared test is "there is no relationship" between the two variables. The test statistic is calculated based on the assumption that all observations are evenly distributed amongst the cells in the contingency table, so the test should work for most kinds of relationships.

A word of warning - the test is sensitive to data imbalances.


Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

You gave an exemple of a value function, but not an optimal value function, that would verify :
$$ V^{*}(s) =max_\pi(V^{\pi}(s)) $$
Such a value function would allow to use one-step look ahead approach to get the optimal policy. Because the optimal value function would be defined somehow iteratively to take neighbor value into account.
In other words, the problem with your value function is that it is not optimal, notably  because of the significant step from 0 to 20. If it was optimal you would have a value nearer to 20 near it and the look-ahead would follow this path.


I was also looking for something similar and found a couple of libraries (https://matrixprofile.org/libraries/) that might work but I haven’t tried them out much yet.

I hope that helps and if anyone has any other suggestions I’d be interested as well.


How to perform one hot encoding on multiple categorical columns

LabelEncoder is not made to transform the data but the target (also known as labels) as explained here. If you want to encode the data you should use OrdinalEncoder.

If you really need to do it this way :

categorical_cols = ['a', 'b', 'c', 'd']

from sklearn.preprocessing import LabelEncoder
# instantiate labelencoder object
le = LabelEncoder()

# apply le on categorical feature columns
data[categorical_cols] = data[categorical_cols].apply(lambda col: le.fit_transform(col))
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder()

#One-hot-encode the categorical columns.
#Unfortunately outputs an array instead of dataframe.
array_hot_encoded = ohe.fit_transform(data[categorical_cols])

#Convert it to df
data_hot_encoded = pd.DataFrame(array_hot_encoded, index=data.index)

#Extract only the columns that didnt need to be encoded
data_other_cols = data.drop(columns=categorical_cols)

#Concatenate the two dataframes :
data_out = pd.concat([data_hot_encoded, data_other_cols], axis=1)

Otherwise :

I suggest you to use pandas.get_dummies if you want to achieve one-hot-encoding from raw data (without having to use OrdinalEncoder before) :

#categorical data
categorical_cols = ['a', 'b', 'c', 'd']

#import pandas as pd
df = pd.get_dummies(data, columns = categorical_cols)

You can also use drop_first argument to remove one of the one-hot-encoded columns, as some models require.


Facing a difficult regular expression issue in cleaning text data

To remove multiple white space matches, you will need [\s\n]+, note the inclusion of the + (match one or more).

Code:

Here is a function which will build the regex automatically from a text snippet:

def remove_words(to_clean, words, flags=re.IGNORECASE):
    regex = r'[\s\n]+'.join([''] + words.split() + [''])
    return re.sub(regex, ' ', to_clean, flags)

Test Code:

st = "Management's decision  \n \n and analysis\n " \
     "and  \n comments is to be removed"
print(remove_words(st, 'decision and analysis and comments'))

Results:

Management's is to be removed


How can i find trend time for my articles?

Based on your comment and what I can understand from your question, you could identify the "trend" you are talking about as a series of Outliers. From Wikipedia, an outlier is:

  In statistics, an outlier is an observation point that is distant from
  other observations. An outlier may be due to variability in the
  measurement or it may indicate experimental error; the latter are
  sometimes excluded from the data set.

How can you do this in excel? There is a nice tutorial in this link. In short, based on the data you added, you need to do

Calculate the 1st and 3rd Quartile with the formula =QUARTILE(B2:B9,1) and =QUARTILE(B2:B9,3) respectively
Then get the IQR which is the difference of the above two =F3-F1
Calculate the L Bound as =G2-(1.5*G4), where G2 is the 1st Quartile and G4 the IQR
Calculate the U Bound as =G2-(1.5*G4)
Then next to the column with the data, add this =OR(B2<$G$5,B2>$G$6) which check if your data is lower or higher the L and U Bound. If it is, that row is marked as Outlier.

You need to play around with the multiplier in 3rd and 4th step to get you the right results in your test data. With 0.8, the last 3 rows are marked as outliers and you can say that the desired outlier or "trend" as you called it period is on 17 to 19th January 2018.

There are more sophisticated ways to do it in Python, but I am not aware of a similar one in Excel. You could look for Anomaly Detection in time series with Python.


How can autoencoders be used for clustering?

Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar.  Using an autoencoder lets you re-represent high dimensional points in a lower-dimensional space.  It doesn't do clustering per se - but it is a useful preprocessing step for a secondary clustering step.  You would map each input vector $x_i$ to a vector $z_i$ (not a matrix...) with a smaller dimensionality, say 2 or 3.  You'd then use some other clustering algorithm on all the $z_i$ values.

Maybe someone else can chime in on using auto-encoders for time series, because I have never done that. I would suspect that you would want one of the layers to be a 1D convolutional layer, but I am not sure.

Some people use autoencoders as a data pre-processing step for classification too.  In this case, you would first use an autoencoder to calculate the $x$-to-$z$ mapping, and then throw away the $z$-to-$\hat{x}$ part and use the $x$-to-$z$ mapping as the first layer in the MLP.


Different hidden layer architectures deliver the same classification results, is that normal?

There are several things I would like to mention :

I do not think that much change on your architecture will impact a lot. Try comparing 10, 20, 50, 100 or more depth. Difference will be most likely to be slightly more noticeable.
You are comparing accuracy which is based on labels instead of the probabilities. Try comparing logloss or auc score which is more sensitive to small changes.

Hope this helps.


How to use correlation matrix when the dataset contains multiple columns with text data?

The problem is that the correlation matrix has to be done with numerical values. So what you have to do is to transform the texts into numerical vectors. There are several ways of doing this, there are libraries like gensim that can make implementation easier.

Bag-of-words
A vector of the most frequently used words across all texts is created. Then for each text sample the occurrence of each word in that sample is counted.

"John likes to watch movies Mary likes movies too"

Would give

{"John":1,"likes":2,"to":1,"watch":1,"movies":2,"Mary":1,"too":1, ... other words in corpus}

And then the values of this could be used to calculate the correlation matrix

[1,2,1,1,2,1,1, ...]

This method can also be improved by using TF-IDF.

Word embeddings and averaging
Word embeddings are when a word is mapped to vectors that try to encode the information in a word. These can trained from scratch for the task at hand, but there are also pretrained word embeddings that can input a word and output a embedding vector. To get a single vector from an entire text the average word embedding can be used.

Doc2Vec
Doc2Vec expands on word embeddings and is speficially used for embedding entire chunks of texts.


What software is being used in this image recognition system?

I'm pretty sure that the software you're referring to is a some kind of internal research project software, developed by Enlitic (http://www.enlitic.com), where Jeremy Howard works as a founder and CEO. By "internal research project software" I mean either a proof-of-concept software, or a prototype software.


Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint

I also have been looking for answers regarding this before, and after several days of trying with my friend, we manage to do it. I attach some code snippet that we use, you may modify it according to your use case
image = '763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.2.0-cpu'
container = {
    'Image': image,
    'ModelDataUrl': model_data_location,
    'Mode': 'MultiModel'
}

sagemaker_client = boto3.client('sagemaker')

# Create Model
response = sagemaker_client.create_model(
              ModelName = model_name,
              ExecutionRoleArn = role,
              Containers = [container])

# Create Endpoint Configuration
response = sagemaker_client.create_endpoint_config(
    EndpointConfigName = endpoint_configuration_name,
    ProductionVariants=[{
        'InstanceType': 'ml.t2.medium',
        'InitialInstanceCount': 1,
        'InitialVariantWeight': 1,
        'ModelName': model_name,
        'VariantName': 'AllTraffic'}])

# Create Endpoint
response = sagemaker_client.create_endpoint(
              EndpointName = endpoint_name,
              EndpointConfigName = endpoint_configuration_name)

# Invoke Endpoint
sagemaker_runtime_client = boto3.client('sagemaker-runtime')

content_type = "application/json" # The MIME type of the input data in the request body.
accept = "application/json" # The desired MIME type of the inference in the response.
payload = json.dumps({"instances": [1.0, 2.0, 5.0]}) # Payload for inference.
target_model = 'model1.tar.gz'

response = sagemaker_runtime_client.invoke_endpoint(
    EndpointName=endpoint_name,
    ContentType=content_type,
    Accept=accept,
    Body=payload,
    TargetModel=target_model,
)

response

also, make sure your model tar.gz files have this structure
└── model1.tar.gz
     └── <version number>
         ├── saved_model.pb
         └── variables
            └── ...

more info regarding this


One-class classification in Keras using Autoencoders?

If you have images of cats only, you could create boundary boxes (BB) of your images. Some BB will have cats an others won't. You will label those BB with cats inside as 1 as the others as 0.
This way you can set up a dataset with a binary class. It will be much easier if you already have boundary boxes for the cats in each image since this way it will be possible to generate the dataset automatically (generate random BB and label them as 1 if the intersection with the cat is big enough and 0 otherwise).
There is some software that helps to create boundary boxes.


If i understand you correctly,To join two outputds layers into nextn layer input, use the following design:

out1 = keras.layers.YourLayer1(**params**)
 out2 = keras.layers.YourLayer2(**params**)
 NextLayer = keras.layers.YourLayer3(**params**)[out1, out2]


need explanation on how an equation is being converted to cvxopt logic in solver.lq

Welcome to the site!

As Media has mentioned values of A,b,c are passed through those matrices.
To understand it better I'm naming the constraints from top to bottom i.e., 1st constraint as constraints-1,......constraint-4.

Firstly, lets talk about matrix b which consists of all the values on the right hand side of the constraints i.e., 1,2,0,4. He converted all the constraints to $⩽$ by multiplying with -1 on both sides. So he multiplied with -1 for constraints-2 and 3.

Now the constraints-2 has changed from $x_1 + x_2 ⩾ 2$ to $-x_1 - x_2 ⩽ -2$

Values of Matrix b would be 1,-2,0(-0),4

Now lets go through the constraints i.e.,

$-x_1 + x_2 ⩽ 1 $ for the coefficient of $x_1,x_2$ are -1 and 1. The values of Matrix A row 1 values are -1,1
Similarly for row 2 to row 4 as you have 4 contraints.

Finally our objective function coefficients, are the values of c i.e., 2 and 1.

Do let me know if you have any more doubts.


How to properly compare these two confusion matrix?

First, about interpreting these confusion matrices: the sum of every row is 1, which implies that every value is a conditional probability p( predicted label | true label ), i.e. the probability of a given true label to be a particular predicted label. Example: the top left cell in both matrices is 0.01, which means that when the true label is 5 the probability that the system predicts label 1 is 1%.
The two confusion matrices show the results of the predictions by two different systems. This means that in general there is not one which is correct and the other which makes errors, just two different ways to predict labels which might lead to different types of errors.

The diagonal shows the True Positive cases, i.e. cases where the predicted label is the same as the true label (this is important since these are the "correct" cases). The probabilities show very little difference between the two systems on the diagonal, the top one being slightly better for labels 3 and 4 and the bottom one slightly better for label 5.
In general it's important not to look only at the diagonal especially for ordinal values, because if the predicted value is far from the true value it's a more serious error than if it's close to it. Example: if the true value is 4 it's better for a system to predict 3 than 1 (both are errors but the latter is worse). However here again there is very little difference between the two systems, they appear to have a very similar behaviour.

In order to quantitatively summarize and compare the performance of the two systems, confusion matrices are too complex. Typically one would use an appropriate evaluation measure, for instance micro or macro f-score (classification evaluation) or Mean Absolute Error (regression evaluation).


Understanding declared parameters in my Conv2d layer of my convolutional neural network

Bot the name and input_shape come from the Layer class which Conv2D inherited. In the doc you provide, they are implicitly in **kwargs


Here's my take on each of your points.

You have very sparse data. Are you storing these data optimally in a sparse object, for example a csr sparse matrix if you're using Keras? This is more likely to affect training time than accuracy, but is something to think about.
5000 training examples and 500 testing examples sounds alright to me, except you may have too few data to fit the type of model you have (you may be severely underfitting). Try a simpler model and see if you can improve results with that (try something stupidly simple like an MLP with one layer and see what happens).
This seems the most likely problem. My first guess is this has to do with the unbalanced classes. If you're using a metric like accuracy to evaluate training at each epoch, I'd recommend instead using something like average_precision or roc_auc_score or f1, all available through scikit-learn. If you're  using Keras, try using class_weights as well, which will weight underrepresented classes higher in the loss function, essentially biasing the algorithm to consider underrepresented classes on an equal playing field. If you're not using Keras, try implementing some similar class weighting scheme.
Batch normalization is less popular now than it used to be. It's worth trying the architecture with and without batch normalization to see if it provides a clear benefit.
Yes, if your data are purely random then you will not detect a signal at all. This goes back to point number 2 (try a simpler model and see if you can detect a pattern). You can also try visualizing some of your samples to see if you can visually see a difference between the classes.

With regard to Xavier initialization, I don't think that's the likely cause of the effect you're seeing. The type of initialization can affect results, of course, but from what you're describing, I strongly suspect this is due to too little signal from the data or the unbalanced classes problem.

Hope that helps!


You could think of histogram as one way of plotting a distribution of values. Another way of plotting such distribution is a KDE (Kernel Density Estimate), but after all, these inform about the same concept, which is how frequent is each value (or values interval) of your values series (x-axis).

I like this picture from seaborn distplot to have a one shot info about the (univariate) distribution of your values, both with continuous (kde) and discrete (histogram) representations. https://seaborn.pydata.org/tutorial/distributions.html


What do you think of Data Science certifications?

As a former analytics manager and a current lead data scientist, I am very leery of the need for data science certificates.  The term data scientist is pretty vague and the field of data science is in it's infancy.  A certificates implies some sort of uniform standard which is just lacking in data science, it is still very much the wild west.

While a certificate is probably not going to hurt you, I think your time would be better spent developing the experience to know when to use a certain approach, and depth of understanding to be able to explain that approach to a non-technical audience.


Why not continue next epoch with best weights

A new epoch starts with the same weights as the previous epoch but with different mini-batch of data. Model performance will have small oscillations across mini-batches for that reason.

Always starting each epoch with the best overall weights might result in staying in a local optimum. Gradient descent is already prone to staying in a local optimum, at the expense of finding a better optimum. Encouraging continued exploration of weights is one of the goals of training.

There is always a chance that both training and validation loss will do down. It is okay that training performance gets worse, there is no need for monotonic improvement in training.

At the end of the training, you should select the weights that have the lowest validation loss.


Multiple Features with the same categorical value

The increase in dimensionality due to 'No Internet Service' could be handled maybe with a 2 layer model. If 'No Internet Service' is true, you run the first level model with a handful of other variables (excluding those which depend on internet service availability). The second layer model runs only when internet service available, with all the variables.


Is the resulting $\hat{L}(h_*,V)$ an unbiased estimate of the true loss?

No. You have taken multiple measurements, each with some uncertainty, and chosen the maximum or minimum value.

  How can I bound the true loss using $\hat{L}(h_*,V)$ ?

In the general case you cannot. It will depend on how much over-fitting is occurring within the model on the training set, size of cv set, amount of times it has been used, and how similar the model's performance was on each use. There is also sampling bias in the cv set, and that interacts with the selection process.

What is generally done if you need unbiased estimate at the end of production is a train/cv/test split. The cv set is used for model selection, and once you have a single model selected, you estimate its loss - or other key metric - on the test set. It is important to use the test set minimally and not in order to select models, if you want it to be an unbiased measure. Otherwise you repeat the problem.

Another approach which maintains confidence in cv-based metrics is to use k-fold cross validation. Taking taking the mean of a metric in k-fold cross validation is still biased once you have used it a few times, but the bias is reduced somewhat. You can take that idea further with nested cross-validation, which allows you to get an unbiased estimate of model performance in a general fashion (i.e. using the same hyper-parameters) from more of your data.


Developing an Empirical Model for Non-Linear Data

A couple of thoughts...

First as a (former) physicist:

As a scientist/physicist I would likely never be satisfied to analyze the data that you have presented because there are some obvious issues with it.  I was initially thinking that it was way to schizophrenic, but then I noticed that you referred to the time units as seconds, in which case this is a pretty slowly varying time series.  Are the units really seconds?  If they happen to be milliseconds or microseconds then you might want to think about a probe that is less noisy (i.e. a larger probe with more of a temperature sink) and check weather you are seeing the effects of ground loop feedback in your system or just EMF interference from your electrical system.

Secondly, there are clear patterns in the data and these do not line up very well, so I would rerun the experiment with things timed much more accurately.  For instance, the event that occurs midway through the experiment that drops the temperature (looks like an icicle or dagger) occurs 5-8 minutes apart in comparing the dark blue experiment to the purple experiment.  It would be great if these could be synced better.  The point being that if you can't sync them better, then the noise of the time series renders most phenomenon apart from a flat line, almost useless.

Now as a data scientist:

This data is really noisy or at least fluctuates significantly!  You should stay away from averaging minima and maxima as you suggest in your post.  extrema are very sensitive to noise where as you are trying to reduce the noise in your system.

You should perhaps think about applying some sort of smoothing method to reduce the jitter in your data.  Perhaps a second-order exponentially weighted moving average like Holt-Winters.

Finally, you could probably just average the three signals to produce a mean signal.  Means are more suseptable to outliers than medians,so the median signal is also an option.

Some other options include separating the trend from the general behavior using some type of de-seasoning or double ARIMA procedure.  But, it seems like these are too extreme given that the dagger/icicle event looks significant and you don't want to transform it away.  So I would only use these if there is some type of periodicity to your data.

Hope this helps!


Genetic Algorithm to find best parameter values of an estimaor

Having used genetic algorithms extensively in both professional projects and for my own entertainment, I think it is best to just code your own set of functions/classes for the specific project.  As you have seen, there are quite a few well-intended open-source GA packages out there. However, many of the fine details and tweaks necessary for a good GA are problem-specific.  It is also more convenient to work directly with your existing data structures, rather than adapt to someone else's worldview.  In other words, cost-benefit does not seem to work out when applying someone else's GA package.  You need a function for creating a random instance, a function for checking it is valid, a function for scoring it, and a mating function.  That's pretty much it!


Beginner math books for Machine Learning

Although you need book, I recommend the following courses respectively for understanding statistics which are used for machine learning and other tasks in data science. They are free.

Learn Statistics - Intro to Statistics Course
Intro to Descriptive Statistics
Inferential Statistics: Learn Statistical Analysis

If I want to recommend a book, I would recommend the following book which is free under CC license. It has nice examples and is so much practical; moreover, there are lots of codes in it which help you feel statistics in real world examples.

Think Python by Allen B. Downey
Python Data Science Handbook

Also the following link may help:

From Google Itself Good And Concise


Tattoo Image Recognition - Should I Crop Training Data Background

It is really hard to give an answer on what would work best, as Deep Learning is often based on experimenting with different things. But I would highly recommend you start quick and dirty. This is an advice given at all universities or by the famous Andrew Ng. Just start with the easiest model and then iterate. For example if your simple model doesn't work try to create a more complex pipeline by preprocessing the images as you suggested.

In Deep Learning, time is most often the biggest factor. Don't waste it by trying to create a highly complex model to then just observe that it didn't help to improve your model.

I would like to add, if you decide to add more parts to your pipeline, for example 1) Find the human in the image 2) Find the tattoo 3) Classify tattoo etc. you should take a look at this video as it explains how to improve your pipeline in the most efficient way.


Are Convolutional Neural Networks a one model, multiple output layers.?

Convolutional neural networks implement the convolutional filter technique. This is more or less a sliding window, usually (but not limited to) convolving across the width, heights and depths of images.

If you have a single task or multi learning learning task depends on what you want this network to output. Convolution rather describes the behavior of a single layer processing some inputs into some intermediate outputs. The characteristics of your last layer determine if it is used for single or multi tasks. To my understanding this is rather independent of the convolution.


On a (cough) safer bet, I was planning on using PCA after a log transform and the scaling of the data, since it would help to orthogonalize it and, I suppose, improve K-Means's performance. Is this better or worse than simply applying PCA without the log transform?

This is your solution, this is the normal, most used and best practice on what to do with skewed data


Why do we scale down images before feeding them to the network?

Because it saves lots of computation time. Yes, we do loose image details, but it depends on your task whether the details are really important. Let's say your task is to detect and count the total number of circles in an image, So you are only concerned with the shape circle, not its dimension. Hence, reducing image resolution a bit would be more computationally efficient.


There's a few problems in your script actually, and growing objects (whether using rbind() or bind_rows() ) seriously messes up your performance. So if you know how many rows you have, start by creating an object with that many rows.

Next, you rely on a data frame. If you just store the info in separate vectors and only combine them in the end, you remove again a ton of overhead. Take a look at following three mockup examples:

library(rbenchmark)
library(dplyr)
# The naive way
f1 <- function(n){
  out <- data.frame(x = numeric(), y = numeric())
  for(i in seq.int(n)){
    tmp <- data.frame(x = rnorm(1), y=rnorm(1))
    out <- bind_rows(out, tmp)
  }
  return(out)
}
# Preallocating memory
f2 <- function(n){
  out <- data.frame(x = numeric(n), y = numeric(n))
  for(i in seq.int(n)){
    out$x[i] <- rnorm(1)
out$y[i] <- rnorm(1)
  }
  return(out)
}
# Using vectors
f3 <- function(n){
  outx <- numeric(n)
  outy <- numeric(n)
  for(i in seq.int(n)){
    outx[i] <- rnorm(1)
    outy[i] <- rnorm(1)
  }
  return(data.frame(x = outx, y = outy))
}

On my machine, I get the following timings:

benchmark(
  f1(100),
  f2(100),
  f3(100),
  columns = c("test","elapsed","relative","replications")
)
##     test elapsed relative replications
## 1 f1(100)    2.16    27.00          100
## 2 f2(100)    0.34     4.25          100
## 3 f3(100)    0.08     1.00          100

Just changing this can seriously cut your running time already.


Backpropagation Through Time (BPTT) of LSTM

Doesn't this dependency lead to the exploding/vanishing gradients problem again?

Absolutely, and you better have vanishing gradients otherwise you have a training problem.

Vanishing gradient in this case is not a bad thing, it is a good thing (unlike in feedforward). Let $C(t)$ be the cost function evaluated at time $t$ and $ W(t)$ be some weight of the network at time $t$. What vanishing gradient in this case means is that $ dC(t)/dW(t-u)$ becomes smaller and smaller as u becomes bigger and bigger. That is good because $ dC(t)/dW =  \sum_{u=0}^{num\_steps} dC(t)/dW(t-u) $, so if the gradients didn't vanish in time, then the gradients would explode. So $W$ gets a proper non-vanishing gradient even if the gradients in time vanish because the gradient for $W$ is the sum at all times of the gradient for $W$.

In LSTMs the gradients are sure to vanish in time because the activation functions are sigmoids and tanh's so their derivatives are less than or equal to one, so as they get multiplied they slowly become smaller.

This compares to what is normally called the vanishing gradient problem which occurs when gradients vanish while passing from top layers to bottom layers, because that means that $ dC/dW $ for $W $ of the lower layer is vanishing and so the lower layers don't get trained, only the upper layers get trained.

Also, as mentioned in the comments, the above applies to any RNN, not only LSTMs. What sets LSTMs appart from vanilla RNNs in with regards to this question is the gating functions which allows the LSTM to control what it remembers and what it forgets and how much of the new input it takes in. While the above is true in practice for LSTMs (and is true on average also in theory), in theory, one could have a time step $t$ where the output has ignored the last 10 inputs and only depends on the input 11 timestep back ($t-11$), in which case the gradient for the weights 11 timesteps ago will not have decayed. Of course that means that at the next time step ($t+1$) the gradients for 11 steps ago ($t+1 -11 = t-10$) will be zero because the input was totally disregarded at $t-10$. So on average it averages out and you still have the same situation for LSTMs.


Difficulty interpreting word embedding vector similarity (spaCy)

Why are the vector similarities so high for unrelated words for the embedding?

For the specific example you give, I would argue that it makes sense that car and plant  have high similarity. This is likely due to phrases such as car manufacturing plant

  Also I am able to get vectors for non-words like "asdfasfdasfd" or "zzz123Y!/§zzzZz", and they differ from each other. How is this possible?

For your specific case, since you use the en_trf_xlnetbasecased_lg, the answer is straightforward. Embeddings provided by XLNet are contextual, meaning that even if the word itself isn't a word, you'll get an embedding given the words in its context. Also, it is likely that HuggingFace's implementation uses Byte-Pair Encoding as tokens, making it much more robust to out-of-vocabulary situations.


ML that learns to predict and play a simple wagering game

There are a few different data science and ML techniques you could throw at the numbers output by this game. You could try to analyse human players' style. You could generate a table of expected gains/losses by ignoring actual amount gained or lost and estimating the probability distribution of $x$ even though the player did not see it, just from know whether $x \gt w$ or $x \le w$ from enough examples.

If your end goal is to find out how a computer might play such a game, then one clear machine learning choice would be Reinforcement Learning (RL). It is not the only way for a computer to play a game - there are very many optimisation techniques for this. However, it is a machine learning approach - it learns from data observations - and in many variations it includes numerical analysis that you may find interesting, such as chance of winning from a certain position, or predicted future rewards.

RL is not a single algorithm, or even a single approach. Instead it is a way of framing a problem, and all ways of solving that kind of problem are considered to be RL. For RL to work, a problem needs to be framed as a Markov Decision Process (MDP). The good news is that your example game, and similar ones, are good examples of MDPs already.

In RL, the way that an agent plays a game is called its policy. The usual goals of RL include measuring the performance of a specific policy, or discovering the best policy.

So, where to start? First, if you want to try an RL process on your game you should probably do a few things:

Understand what in the game is the State, what are the Actions and what are the Rewards.

The state is anything that the agent knows about and can affect the outcome. Probably the only thing to worry about here is the current funds $f$, although this may not have a huge influence unless $w$ can always go up to $f$. Note each value is treated as a different state.
The action is the agent's choice of both $w$ and $g$ - note each specific combination is considered a different action. If there are a large number of actions, this can make learning harder, requiring more advanced RL algorithms, so I recommend initially you try variations of your game with limited number of choices.
The most obvious choice of reward is the change to $f$ at the end of each turn. It doesn't have to be though, it depends on what you consider a "winning condition" to be for the game. See below.

Simplify the game options a little. Set a small limit to possible ranges of $x, w, g$. This will make it easier to try things out initially.
Set a clear goal that you want to achieve, and make sure that the game structure works for that. You may need to adjust the game representation for even simple change such as "get the most reward in 10 turns" - because that might encourage large wagers at certain steps, and the agent will need to know how many turns it has left (so your state becomes combination of $(f, t)$. Another variation that may work is to set the goal of getting to a certain amount of funds, such as 1000. In which case, the reward would not be the increase in $f$, but actually be +1 for getting to that target, and 0 for any other result. That might radically change the behaviour of the agent, which could be interesting to experiment with.
Look up simple Reinforcement Learning algorithms. I suggest you start with tabular methods - perhaps Monte Carlo Control, or Q-Learning. They can typically be implemented in a few tens of lines in Python/Numpy.

A useful resource for RL is Sutton & Barto's Reinforcement Learning: An Introduction. The draft of the second version is free to download.


What is the meaning of "The number of units in the LSTM cell"?

Most LSTM/RNN diagrams just show the hidden cells but never the units of those cells. Hence, the confusion.
Each hidden layer has hidden cells, as many as the number of time steps.
And further, each hidden cell is made up of multiple hidden units, like in the diagram below. Therefore, the dimensionality of a hidden layer matrix in RNN is (number of time steps, number of hidden units).


Recommended Language/Framework for Building a New Recommendation Engine

I am developing a recommendation engine for stack overflow (personal project). Check it on http://recommender.im .

It is still a working in progress, but I have a quite functional website working. I am putting there most of the code I used through python notebooks.

Basically I used:
- Frontend: angularJS
- Website backend: flask + scikit-learn
- machine learning and data preparation: python, pandas, scikit-learn

I really like python for data science as the community and libraries are really good.


How can I show the relations between travel destinations?

You can try Graph databases (Neo4j/Orient DB etc). Store location and connection between the location as nodes and edges. Then do analysis over graph data. Based on your need, you can use additional attributes (like count) and assign weights for edges etc. Neo4j supports collaborative filtering also.


Recommender System for mostly unique user and items

There are a lot of different ways you could approach this problem. Sort of depends on what kind of results you're hoping to get for your particular situation. I'm going to break up my answer into sections, as this will be a longer reply because I want to provide as much context as possible to the problems you will face with building a job/candidate recommender as they are quite numerous.
Some suggestions on how to deal with your current problem:
Your biggest challenge is data sparsity, so a few suggestions:

If you're not already, use a Factorization Machine as your core model. At least until you're ready to implement a more complicated neural net or something. FMs deal with sparse conditions really well and can take in the wide variety of data & embeddings that you're going to be using in this kind of task. Just doing this alone I think will get you on track to solving the problem you're talking about.
You need to define features to make impressions more dense. An example could literally be consolidating companies with sparse histories into segments:

['Netflix', 'Facebook', 'ComputerVisionCompaniesUnder10Employees', ...]
Then from now on any company that falls under the category of ComputerVisionCompaniesUnder10Employees will only be represented as that label.
You then need to one-hot-encode normalized job title as a separate feature from company and candidate.
Given how FMs work, you shouldn't need to do this, just including the feature should accomplish the same thing, but I've noticed that sometimes explicitly replacing companies labels like that does boost performance quite considerably.
So now variance in recommendation for companies under the same company label would depend on the job title and contextual features about that particular job.
If two different companies under that same label had similar job postings, then they would get recommended similar (if not the same) candidates and that's OK.
As far as candidate abstractions go, you're probably going to want to find some way of normalizing and grouping skills as the skills matrix can get quite sparse on its own.
Additionally adding in aggregations of features of companies that a candidate previously worked for could help. Like how many "Large companies" has the candidate has worked for as an example.
The more categorizations you can feed to the FM the better, the FM will calculate  interaction embeddings between every set of features. The important bit is to separate the different features about the jobs/candidates and abstract separately. Don't calculate abstractions for job postings, calculate abstractions for companies, job titles, and skill descriptions, and let the FM figure out how all those things relate.
Data issues in Job/Candidate Recommendation:

Data sparsity of impressions.

As you implied in your question, the only knowledge you currently have on candidates' job preferences is their past jobs and content related features. Getting a measure of whether or not the candidates liked their past jobs is also difficult.

Lower than expected information entropy in job-related data points.

Almost all resumes, LinkedIn profiles, and job postings are all designed with brevity in mind. It's a well known fact that recruiters spend very little time reading resumes, so it's become a standard that people should aim to keep their resume under 1 page. This makes information content lower than one may think. The same is often true of job postings, in fact most candidates don't even read job postings due to the prevalence of "1 click" apply features.

Non-trivial normalization of different job related entities

There aren't really any good open source standard job title sets out there last time I checked. Good skill ontologies are also hard to come by. These hurdles are a big pain, but you can overcome them with a lot of hard work. I'd start by picking some standard sets as truth (I personally recommend checking out the data referenced here). There's a couple of ways to try and do this automagically, but I personally haven't found a great way of dodging a lot of manual cleanup.
Notes on candidate recommendation:
Candidate recommendation is really only something that recruiter platforms like LinkedIn have had to think about all that much, as most job platforms recommend jobs to candidates and then candidates hit apply and that's when they show up in the employer's applicant pool.
From my experience, the features that recruiters care the most about when being recommended candidates is whether the technical skills match, whether or not they went to a prestigious university, and years of experience. As mentioned earlier, it's difficult to get any measure of what one might call "personality" from resume/profile data, so they just want to get candidates that match on a call to further vet them. Another note here is that most candidates don't actually read the JD, so getting candidates that are genuinely interested in the position beyond the paycheck/title saves recruiters a lot of time.
Notes on job recommendation:
There are actually very few platforms that do true job recommendation in the way that's been described here. Most just do job search. When you go to indeed and search for a job, indeed knows very little about the visitor on it's site beyond what they're typing in the search box. Indeed's success hinges on its search ranking far more than its recommender performance. Another note is that in my experience senior candidates like to have a lot of control of what jobs get thrown their way and junior candidates like to shotgun apply to every job that vaguely resembles what they're looking for as fast as possible. The junior case is at odds with what recruiters want, so the goal is make sure that candidates are only shown jobs that they are fit for.
My experience:
I've actually started a recruitment company before and have had to build a job recommender for candidates. Now this company didn't work out for reasons unrelated to the recommender performance. The job-search market is a very saturated one and surviving/innovating in it given the problem's constraints is very difficult but that's a whole other subject on it's own.


xgboost Predictions from R and Python don't match

Solution for posterity: In Python, xgboost requires np.array as input. So this line:

d = xgb.DMatrix(data=[x], missing=np.nan)

should be:

d = xgb.DMatrix(data=np.array([x]), missing=np.nan)


Spyder 4: changed behavior or "run cell" / run selected code

[Ctrl]+[Enter] is for debugging mode and F9 is what you are looking for.
See also this stackoverflow question.


Caffe net.predict() , predict same probability

I solved this problem. The problem was use_global_stats setting in deploy.prototxt. In training step, use_global_stats has to be set as false because mean/var need's to be update. But when I predict using deploy.prototxt use_global_stats has to be set as true.


Regression coefficients vs feature_importances_ vs none

The main difference between Linear Regression and Tree-based methods is that Linear Regression is parametric: it can be writen with a mathematical closed expression depending on some parameters. Therefore, the coefficients are the parameters of the model, and should not be taken as any kind of importances unless the data is normalized.

On the other hand, non-parametric methods have very different ways of measure the importance. In layman terms, measuring the importance of variables on a tree can be done by checking how close they appear to the root node. In ensembling methods, like bagging, one can compute the importance of a variable as the average among the ensemble, like in this stackoverflow answer.

The main difference is, then, the fact that parametric models have, through their parameters, a way of showing the importance of the variables, while non parametric models need some extra work.


What is this type of plot and how to interpret it?

The shaded area likely shows the dark green line plus or minus some error/uncertainty estimate. Common error estimates may be based on the standard deviation, a confidence interval, or the interquartile range depending on the data and the analysis being done. Without more information, we cannot know what the shaded area represents.


Improve Pandas dataframe filtering speed

The concept to understand is that the conditional is actually a vector.  So, you can simply define the conditions, and then combine them logically, like:

condition1 = (df.col1 == 10) & (df.col2 <= 15)
condition2 = (df.col3 == 7) & (df.col4 >= 4)

# at this point, condition1 and condition2 are vectors of bools

df1 = df[condition1]
df2 = df[condition2 & ~condition1]
df = df[~ (condition1 | condition2)]

This will be considerable faster as it only evaluates the conditional once.  Then it uses them to perform indexed lookup to create the new smaller dataframes.


Sentiment Analysis of Movie Reviews using Python

Given that exact case, I would assume that you are getting negative decision due to names, mentioned in the review (in your training dataset actors were more often met in negative reviews). You should, probably, remove all non-relevant words from reviews, and that includes not only stop words, but all person names (since they are less of a sentiment marker except, maybe, justin bieber, who is a really negative marker to anything :) ).


Parameters used in GridSearchCV Slow Down in skLearn Tutorial

If you aren't sure whether the code works or not, you could turn on gridsearch output:

grid = GridSearchCV(SVC(), param_grid=param_grid, cv=cv, verbose=2)

I just run the script in your link - it worked fine for me. Are you sure you didn't change something while working on the code?


I am going to agree with @user1683454's comment. After importing, your DATE column is of either character, or factor class (depending on your settings for stringsAsFactors). Therefore, I think that you can solve this issue in at least several ways, as follows:

1) Convert data to correct type during import. To do this, just use the following options of read.csv(): stringsAsFactors (or as.is) and colClasses. By default, you can specify conversion to Date or POSIXct classes. If you need a non-standard format, you have two options. First, if you have a single Date column, you can use as.Date.character() to pass the desired format to colClasses. Second, if you have multiple Date columns, you can write a function for that and pass it to colClasses via setAs(). Both options are discussed here: https://stackoverflow.com/questions/13022299/specify-date-format-for-colclasses-argument-in-read-table-read-csv.

2) Convert data to correct format after import. Thus, after calling read.csv(), you would have to execute the following code: dateColumn <- as.Date(dateColumn, "%m/%d/%Y") or dateColumn <- strptime(dateColumn, "%m/%d/%Y") (adjust the format to whatever Date format you need).


How to feed tuples of multiple features to the input layer of a Neural Network

This sounds like an ideal situation for a complex valued neural net. A readable introduction is https://makeyourownneuralnetwork.blogspot.com/2016/05/complex-valued-neural-networks.html


How to calculate accuracy on keras model with multiple outputs?

It's going to take a bit of engineering - since you have a variable size output, you need to encode the length into the output in order to evaluate the accuracy of the model overall. If instead of outputting "up to 5 digits", you output an array of 5 predictions, where some non-digit (such as -1) operates as indicating that there is no digit present, you can better evaluate your network. If you retrain your network as such (where $X$ is the array of images and $Y$ is an array containing arrays of form $[1,4,3,-1,-1]$, for example), then model.evaluate($X_{test}$,$Y_{test}$) will work as expected.

If you don't want to re-train your network, you can write a simple function to take the output from model.predict($X_{test}$) and encode it into the corresponding format. This encode function will simply go from $[1,4,3]$ to $[1,4,3,-1,-1]$. You can then calculate the  accuracy by sklearn.metrics.accuracy_score($encode$(model.predict($X_{test}$)),$Y_{test}$), where $encode$ is the aforementioned function.


How to conclude, regression results can't be improved anymore?

In general, it is very hard to prove a negative. So, concluding that a result cannot be somehow improved with some unspecified technique tends to be almost impossible. Often the best one can do is to follow state-of-the-art good practices and try out several sensible approaches. Then it is at least unlikely that the results can be easily improved.

For example, using p-value cut-offs from univariate regressions to eliminate variables and using stepwise model selection is known to be a poor practice, so I would assume that one could improve over your current results. For example, without knowing more details of your data, I would expect that elastic net without pre-screening variables with the hyperparameters selected using cross-validation (within the training set) would perform better on a hold-out test set in many cases.

Additionally, if you primarily care about predictive performance then linear regression is unlikely to achieve the best predictive performance. Some form of boosted tree or similar approach will likely be substantially better, variables may tend to be non-linearly related to the outcome. With some subject matter knowledge/good feature engineering you may also achieve pretty good results with penalized (i.e. ridge/elastic net or similar) linear regression - but perhaps the outcome needs to be transformed (e.g. to the log-scale), perhaps some predictors need transforming (e.g. also to the log-scale) and perhaps we can expect certain interactions to be potentially important.

It is also not clear how you are evaluating performance (cross-validation? a hold-out set?). Evaluating model performance on data that directly entered the modeling is likely to be overoptimistic, particularly when you have many potential predictors and presumably not a lot of data. You mentioned a split, which I speculated might be a train-test split, but then it seemed like you kept changing the split to find good performance metrics. If you do that, you can no longer trust the resulting performance metrics.


Predicting or patron find of a binary variable over time

Since you say you are a beginner, I suggest using a simple binary classifier: logistic regression. It takes the input vector, and transforms it by taking its dot product with a weight parameter (estimating which is the goal) before passing it through the logistic function:

$\hat y(x) = \dfrac{1}{1+\exp(-\left< w, x \right>)}$ and $y=\begin{cases}1, \text{user present} \\ 0, \text{user absent} \end{cases}$.

For details read the wikipedia article. For features, I would use the time of day encoded as $\left( \sin (2\pi t/T), \cos (2\pi t/T) \right)$, a hot-encoded categorical variable for the day of the week, a binary variable to indicate weekends, and so on. It's a crude model that leaves a lot of data on table, such as correlations between users, but as a beginner's exercise it is appropriate.


Support vector regression and paremeters

It looks like that you are using scikit-learn. In this case use Grid Search Cross Validation  or Randomized Search Cross Validation to find the best parameters.

sklearn.grid_search.GridSearchCV(estimator, param_grid, scoring=None, cv=None, ...)

In these approaches you basically loop over possible sets of your parameters, specified via param_grid. For each set you perform a cross-validation (default is 3-fold, but you can specify it via cv parameter). Cross-validation gives you the mean value and deviation of your 'scoring' parameter (e.g. 'accuracy' for classification, 'r2' for regression). The set of parameters with the best 'scoring' is the winner.

See example here. It also show how to output not only the best set of parameters, but also "top-N". I find it very useful in building intuition about my model.


anomaly detection in vehicle sensor data

Anomaly Detection in sensor data can be approached in many ways. A thorough introduction is available in these books:

Anomaly Detection Principles and Algorithms by Mehrotra et.al (2017)
Outlier Analysis by Aggarwal et.al (2017)

These cover the different approaches like distance-based, clustering-based et.c, along with time-series models as well as ensemble models.


Scientific explanation of Skip Connections

I recently read this paper which proposed a new technique without using skip-connections.It is proposed that the same accuracy can be achieved by increasing the width of the network rather than depth.


Decision Tree Regressor: domain of the y variable

Decision Tree Regressor won't predict values outside the range of values they are given in the train set. If your extreme values are -4 and +10 the predicted values will be between -4 and +10. The reason for that is that a decision tree splits the training data in groups. The prediction associated with a node is then the mean value of the training data of that node (or if you choose something else than MSE, whatever value that minimize the cost function for this group).
An example of how a regression tree works : https://www.saedsayad.com/decision_tree_reg.htm The output for Windy and Sunny is 26.5 as the training data has 2 data points that fit these criteria (23 and 30)


Deep Q-Learning for physical quantity: q-values distribution not as expected

One guess is is that network isn't lucky enough to encounter (sufficient number of times) an example that leads to the reward.  The issue is mentioned here pointing out that RL-agent might favor higher though incorrect estimates too early during learning. Link hypotheses that perhaps "step-annealing"  could be better, instead of gradualy annealing the epsilon-greedy parameter.

In other words, every time you reduce the epsilon-greedy parameter, you reduce the network's chance to take a random action and actually encounter your (already rare by default) reward. Consider reducing the epsion-greedy parameter further in time, or maybe use a step function to anneal it.

There is also Hindsight experience replay. It tells you to concatenate to your input-vector the state in which you wish to ultimately end up (best-case scenario), and feed it into the network, for forward prop.

After it fails to encounter your reward, you take the state where the network ended up, and backpropagate as if that state was what you actually aimed for originally.

According to the above link you can also give it part of the state vector, if the exact destination state is unknown.

HER teaches your network to achieve smaller goals instead of "fixating" on the best-possible goal. Unlike the traditional approach "don't care how, just give me reward", HER describes how the final state looks like. Now, the network is able to learn how states relate to each other, and ultimately how they relate to the destination state the network was meant to go. (instead of mere "I failed" or "I succeeded")

Traditionally, your network won't get a chance to form an impression of the  environment if encountered rewards are zero. With HER, you get such a chance even if the end goal is not visible on the horizon and rewards would be zero or uniform.

Here is a great blog about HER

Video of Hindsight experience replay


Benchmark datasets for collaborative filtering

The obvious answer would be the Netflix prize dataset, there is a lot of research into it and most CF algorithms have known scores in it.

There are other available datasets that are usually used as benchmarks:

Movie lens Dataset: a 20 million ratings dataset used for benchmarking CF algorithms;
Jester Dataset: a joke recommendation dataset with more than 6 million ratings;
You can find many more datasets in this link


why is the model prediction output in keras lstm imdb, a vector?

x_test is of shape (25000, 80), that means 25000 samples with each sample corresponding to a sentence with a maximum of 80 tokens (see maxlen).
Accordingly x_test[0] is a vector of size 80.

model.predict does work on batches of samples, so you can call all model.predict(x_test) you will get a vector of n predictions where n is the number of test samples, i.e. 25000 in that case.
You could also predict the outcomes for just a part of the test set with slicing, e.g. model.predict(x_test[:10]) for the 10 first samples.

What happens when you are calling this function on just one sample x_test[0] is that the input is interpreted as a batch of 80 samples each with just one token.
(To the best of my knowledge, this "interpretation" happens somewhere down the function tree in keras.engine.training._standardize_inpupt_data.)
The model "doesn't know" the length of the input samples beforehand and thus doen't complain about these very short samples.
So you get 80 predictions for all these samples as if they were all sentences with just one token.

To get a prediction for the first sample, you could use model.predict(x_test[:1]). So you predict a batch containing only one sample and get the corresponding batch of one prediction.


Transformer masking during training or inference?

The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet.
At inference time, it is no longer a problem because there are no tokens from the future, there have not been generated yet.


Data with similar mean, min and max across all columns. What could I do to build a classifier

Your datapoints are too close to each other and hence it is really tough for any ML model to learn this inputs as it doesn't know how to differentiate almost same data to 1 and 0 label. That's why the result is random and you are getting around half accuracy.


Use trained non linear regression model to identify variables that maximize the predicted value

For non linear regression are you using SVM Kernel approach or deep learning?  One of the articles I have referred to in the past is this research paper from computational biology where in they have defined a measure for feature importance. I had come across this in the past when preparing for an interview so didnt actually try it out and see if it works.The link to the article :- [https://arxiv.org/pdf/1611.07567.pdf] . Let me know incase you do practically try this out


What kind of plot will be helpful for total count group data ( data grouped by month and year )?

If you like using Seaborn to construct your plots, sns.countplot is probably the best for plotting data where you want to show the differences in value counts.

You can find the link to the documentation page here which might be helpful for you to construct it in the  particular way you want.

As an example, here is a version of a "multiple" count plot like you suggest. (directly from the Seaborn Documentation page)


What courses / subjects are most important to the field of Data Science?

Probably the most useful (and fun) answer is provided by the excellent Swami Chandrasekaran ... in the form of a subway map:

I would add just one comment to that:  school (or learning for the sake of learning) doesn't provide the same kind of experience as solving a real problem does.  Therefore, to learn, find real problems you can tackle.  To stay motivated, make them problems you really care about.  Kaggle competitions are a great place to start.  Even reproducing a winning solution (perhaps with some variations, or on a new dataset) will be a huge learning experience.


Why is the testing set standardized using fit values from the training set

You could do that. Assuming your train/test data are from the same distribution, which is in fact what you assume, they should have a similar mean and standard deviation. The result would be similar, in theory, so by the same token, why would you refit the scaling on test data?

In practice, the mean/variance estimate from the presumably smaller test set is probably not as accurate. You will be transforming the test data differently than the training data, and the model may not fit as well.

Taking it to a logical conclusion, how would you then predict for a single future instance? you can't scale one instance based on stats from one instance; it has no variance.

The principle here is to fit a model on training data and then apply that fixed model, of which scaling may be a part, to future data. So there is no particular upside to refitting this specially, and some potential downside.


Creating an "unclassified" class in Random Forest

It seems you can use RandomForest to get probabilities of being in both class by using predict_proba(X).
You could just get thoses probs, get the higher one (which is currently the class assigned to your data), and manually set a threshold, for example setting that if the most probable class is at less than 75%, then you manually classifies it as "Unknown"


I would say it depends on your images themselves and the task. So my short answer would be: for your task at hand, would this help a human being to perform the task? A similar answer may apply to your algorithm.

To be more specific: as far as I understand HDR compared to traditional images, the aim is to increase the dynamic range of the pictures to make it more similar to what the human eye sees. This means that for pictures with complex lighting conditions (strong backlighting for instance), there will be a real difference between classic / HDR pictures, so that if you aim at detecting objects in theses conditions, HDR will help.

On the other hand, if you have standard lighting conditions, I believe @mikalai answer fully applies and there will be not much difference.

I know this answer relates more to the "photography" side of the question rather than the machine learning side, but I think it is important to consider both in your case.

Does this help?


Finding groups of friends in social network data

Community Detection and Clique Percolation:

This is a community detection problem.  Here is a very detailed review article surveying the state of the art.

The Clique Percolation Method is also useful to explore as it pretty much solves what you may need to know.  You can also go through what Matching is, and link the concept with Blossom Algorithm. Though, this finds the maximum matching, you may slightly modify the algorithm to get the minimum matchings of 5 at least.


What are the output shape of LSTM with "return_sequences" equal to "True" or "False"?

Assuming your output state size is h. You will generate nxh for n data points if return_sequences is false. On the other hand you would be having nxtxh dimensional array as output if return_sequences is set to true. Here t is the number of timestamps in each data point.


Looking for similar items in a large data set

Consider the R package poLCA. Given a set of people with attributes that have categorical values, it can cluster the set into a fixed number of groups. As Anony-Mousse says above, it's not quite the same problem. Similarity would be to find people that are close to the one in the query. Clustering is close to answering that question for everybody.

The name of the package gives some deeper insight into those differences. LCA is for latent class analysis. The idea is that there is some characteristic of the people in the dataset that can be used to group them into clusters, but it's not explicit in the variables you have. It's latent.


Share Jupyter Notebook with a non programmer

GitHub has built in support for showing a notebook. You will just need to run the notebook yourself, then upload the file to Github like all other file types.

Your viewer will be able to see your notebook without any installation.


Get an equation from the relationship among variables

This seems like a problem that can be fixed with any regression problem. Here are models I would recommend to get started with:

If you need your model to be explainable (i.e. show which factors contributed to the score most):

Decision trees / random forests
Linear regression

If you only care about giving a final score

Essentially any other model, which can include:

Support vector machines
Neural Networks
K-Nearest Neighbours
Gaussian Processes
and more


What are the advantages / disadvantages of off-policy RL vs on-policy RL?

This was answered in cross-validated and stackoverflow:

The reason that Q-learning is off-policy is that it updates its Q-values using the Q-value of the next state $s′$ and the greedy action $a′$. In other words, it estimates the return (total discounted future reward) for state-action pairs assuming a greedy policy were followed despite the fact that it's not following a greedy policy.
The reason that SARSA is on-policy is that it updates its Q-values using the Q-value of the next state $s′$ and the current policy's action $a′′$. It estimates the return for state-action pairs assuming the current policy continues to be followed.

These slides offer some insight on pros and cons of each one:

On-policy methods:

attempt to evaluate or improve the policy that is used to make decisions,
often use soft action choice, i.e. $\pi(s,a) >0,  \forall a$,
commit to always exploring and try to find the best policy that still explores,
may become trapped in local minima.

Off-policy methods:

evaluate one policy while following another, e.g. tries to evaluate the greedy policy while following a more exploratory scheme,
the policy used for behaviour should be soft,
policies may not be sufficiently similar,
may be slower (only the part after the last exploration is reliable), but remains more flexible if alternative routes appear.

For reference, these are the formulations of Q-learning and SARSA from Sutton and Barto seminal book:

Q-learning:

SARSA:

P.S.: I referenced and quoted the original answer from a different stackexchange site, as indicated in this meta question.


How to classify features into two classes without labels?

As you know, clustering is Unsupervised learning algorithm. Since you don't know the number of clusters, it becomes hard to find the best possible separation (number of clusters). There is a very good paper published on validating clustering techniques. There are 3 criteria defined in this paper for validating your clusters. You can take a look at that.

Also, the link below gives you some code in R for cluster validation. If you want you can try this too-

http://www.sthda.com/english/articles/29-cluster-validation-essentials/97-cluster-validation-statistics-must-know-methods/


How do different models take in account independence/dependence of features?

It depends on the different families of the learning algorithms. For example, Naive Bayes and all the semi-Naive Bayes models assume that the features are indpendent, which is not true in most real world problems. The latter is a disadvantage of Naive Bayes learning algorithm, although in some problems as in natural language process these models usually perform well. Most of the machine learning algorithms do not assume such thing (feature independence). Finally, there is no holistic rule for that.


What is the difference between the 3 architectural components of Microsoft R Server?

RevoScaleR package & ScaleR are same (I think after MS acquisition its became scaleR)
A new file format especially designed for large files such as :
XDF File Format (block based data format)
RxDataSource (designed to support the use of external memory algorithms with .xdf files.)
PEMA  based Algorithms
External memory implementations of the statistical algorithms most commonly used with large data sets such as : rxSummary, rxCrossTabs, rxLinMod. rxKMeans, rxGlm etc.
An extensible programming framework that allows R programmers to write their own external memory algorithms that can take advantage of ScaleR Enterprise’s new Big Data capabilities.

DistributedR for supporting big data framework
ConnectR It has all connectors to interact with other data sources
DeployR is used to integrate R with any application programming


You can convert input to vectors of labels + day number . E.g:

(First column is day-number, rest of the columns indicate presence/absence of a label)

[[ 1 0 1 0 0 0 0 .......]
 [ 2 1 0 0 1 0 0 .......]
 [ 3 1 1 1 1 0 0 .......]
 [ 4 1 0 1 1 0 0 .......]
 [ 5 1 1 0 1 0 0 .......]]

(5000, 201)

Output should be a vector of probabilities for individual labels.

This article has an example :

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/


Visualizing Variance / Standard Deviation for categories

I know this may not exactly correspond to what you need, but my first idea would not be to visualize the variances. Instead, I would define a metric for each method.

For instance, considering all results obtained with a given, you have a vector of outputs (one Variance for X for each Category), then you can simply compute a p-norm of this vector, and compare it to the norm obtained for other methods:

$$\left\lVert x\right\rVert_p = \left( \sum_{i=1}^n x_i^p \right)^{1/p} $$

$p=2$ gives you the euclidian norm, you can increase or decrease $p$ depending on if you want to raise attention on maximal values or the average.

In terms of visualization, you could simply plot (in a bar plot for instance) a few norms that you selected (orders 1, 2, and infinite for instance).

For a pure visualization, you could plot the data in violin plots. For instance in python:

import seaborn as sns
sns.violinplot(x="Method", y="Variance for X", data=your_data_as_df)

Each violin would give you an idea of how the data is distributed over all categories.


Have you tried using the pandas.read_json method? (documentation)

And it looks like your json is structured like 'records' so use

pd.read_json(_, orient='records')


Transformation from Datawarehouse into Big Data structure

Based on your comments transitioning big data sets from on-premises systems to a cloud-based system is cumbersome and fraught with challenges. However, you can use Amazon RedShift:

  Amazon Redshift is a fully managed, petabyte-scale data warehouse
  service in the cloud. You can start with just a few hundred gigabytes
  of data and scale to a petabyte or more.

The first step is to create a data warehouse is to launch a set of nodes, called an Amazon Redshift cluster.
After you provision your cluster, you can upload your data set and then perform data analysis queries. Regardless of the size of the data set, Amazon Redshift offers fast query performance using the same SQL-based tools and business intelligence applications that you use today.

  OR

Use a Hadoop environment as the landing zone to pull in data from
various sources, process it, and transfer the processed data to the
existing data warehouse or other repositories.
Explore scenarios of different ways to implement a landing zone. Learn about the architecture of the zone and the tools and techniques for integrating it with various environments.


Why am I getting misaligned values while plotting heatmaps using seaborn?

There seems to be an issue with the matlpotlib package.
3.1.1 update has caused issues with heatmaps.
Downgrading to v3.1.0 fixes this issue.
Use the following code to downgrade.

pip install matplotlib==3.1.0

Refer this github discussion for details


I don't think there's any definitive answer for this and it will depend on your particular domain. Here's how I go about it:

The English language contains about 20,000 words (or at least the most common) so I use that as a baseline
I expand this number to account for some common misspellings
Does my data contain special things like emojis? Emojis can still convey meaning, so I expand my vocabulary to include de-emoji'd text
Does my data contain specialized text like scientific and/or academic terms? I expand my baseline number based on this.

Finally, you can always check  your token index to find how  many words you have out-of-vocabulary. If that number seems reasonable enough to you to proceed then you move forward, otherwise you expand your baseline number a little more.


Using One Hot Encoding VS Label Encoder for Football Match Outcomes

I would expect some ordering, since the expected outcome is probably depending on the team's strengths. However, there may also be a tendency to/away from draws if teams are equally strong. I would guess that the one-hot encoding is the better choice if you have sufficient data to train your model.
Your prediction vector would typically give e.g. [0.1 0.7 0.2] as probabilities, and you select the highest probability as the outcome. Either by hand or in a (final) layer in the model.
It shouldn't. Typical models are dimension agnostic. E.g. a fully connected first layer loses all positional/dimensional bias. For trees similar arguments hold, however, I don't know how well tree models would perform on such sparse data.
The bookies' odds should highly correlate to the expected outcomes, e.g. bookies will likely have odds that reflect relative strength. In such a case, the odds could more or less 'vanish' in the model.
With such sparse data, you should definitely tune your parameters for forest models carefully.

Just some added notes: If you want to improve on your model, you could also include a few calculated parameters, e.g. current rank, win/draw ratio for both teams (in general, or just the respective home/away versions), most recent (home/away) results. Or even travel distance if you have the data.


Building a multiclass classifier that can handle classes it has never seen?

You basically want your model to say - I dont know.
For the test set created out of the original dataset with labels, plot a histogram of the probabilities of the predicted class. Are they generally higher than some threshold say 0.2 ? If so experiment using that as a threshold to flag items from your unlabeled dataset for a manual inspection. The hope is that the probabilities are low because the model is unsure.
The manual review of these flagged items will allow you to verify if this approach even works. If it works, then use that as a labeling strategy for the "I dont know" category and retrain along with this new label.


Train object detection without annotated data/bounding boxes

Yes, there are models that do this. This link points to one of the first papers I believe. The main idea is called weakly supervised object detection.

The paper essentially makes three modifications.

They treat the typical hidden fully connected layer as a convolutional layer. This works because convolutional layers can be thought of as convolving the same fully connected network about the image.
They add a global max pooling later at the end of this convolutional layer. This is the operator that will "highlight" the area of this final conv layer that has learned the pattern of objects it is trying to classify. Using a threshold on the weights of this global max will ensure a region is significant. Then, they use an algorithm to create a bounding box from this region.
They suggest a new loss function that lends itself to an object existing or not. I think they assume a Bernoulli for each class which lends itself to multiple logistic regression instead of softmax.

Take a look because it's pretty sweet and has been cited by a lot of other new exciting papers.


Compound Annual Growth Rate Function in R Statistics

Instead of depending on a library, you can create your own vectorized function:

Code:

library(tibble); library(dplyr)

example_data <- tribble(
  ~Values_2009, ~Values_2014,
  10000, 19500,
  10500, 15000,
  25000, 35500
)

CAGR_formula <- function(FV, PV, yrs = 4) {
values <- ((FV/PV)^(1/yrs)-1)
return(values)
}

formula is (FV/PV)^1/n -1

Results:

example_data %>%
  mutate(CAGR = CAGR_formula(Values_2009, Values_2014, 4)*100)
#> # A tibble: 3 x 3
#>   Values_2009 Values_2014      CAGR
#>         <dbl>       <dbl>     <dbl>
#> 1       10000       19500 18.170386
#> 2       10500       15000  9.326511
#> 3       25000       35500  9.162151

# If Percentage is needed use the below mutate instead
#mutate(CAGR = CAGR_formula(Values_2009, Values_2014, 4)*100)


Is the ultimate challenge in ML simply computational power?

There is something known as the VC dimension of a hypothesis class. This refers to the maximum number of datapoints with arbitrary binary labels that can be correctly classified by a model from the hypothesis class.

https://en.wikipedia.org/wiki/VC_dimension

If your number of datapoints is larger than the VC dimension for the hypothesis class you have chosen (say set of 2d hyperplanes), then no matter how much you tune your model using cross validation you can never achieve complete accuracy. Hence, the analyst has the important job of choosing the correct hypothesis class while making sure it doesn't overfit. In the case of deep learning this would mean coming up with a specific architecture and is often one of the most difficult tasks.


How to save multi-output predicted masks into two different folders after using model.predict_generator

solve it, I just did this change:

from:

saveResult(conv11,"data/membrane/test/results/road")
saveResult(convc11,"data/membrane/test/results/cl")

to :

saveResult("data/membrane/test/results/road", conv11)
saveResult("data/membrane/test/results/cl",convc11)

and Ive got each output saved into different folders


How can positional encodings including a sine operation be linearly transformable for any offset?

I elected to ask this question on the Mathematics Stack Exchange and I thought it prudent to add the answer here:

https://math.stackexchange.com/q/3119882

From what I have learned from @Servaes, who was kind enough to answer the question, there is a function

$\operatorname{PE}(\text{pos}+k,2d)=\operatorname{PE}(\text{pos},2d)\cos(k/c^d)+\operatorname{PE}(\text{pos},2d+1)\sin(k/c^d)$

that allows for the property of transforming any positional encoding into one with a given offset. However, due to the required use of the sin() and cos() functions this is not a linear transformation.


which scoring function for validation_curve (regression)?

I'd use r2 for validating regression. It'd be much easier to compare different models. Here's wiki, here's sklearn.


Math PhD (Nonlinear Programming) switching to Data Science?

Data science jobs cover a wide range of different activities so any answer is likely to be subjective. I'm in academia so my knowledge of the job market is limited, but from what I can see:

The current context is very favorable to data scientists looking for a job, so anybody with some basic knowledge of ML has a chance. You're already above this level so no worries on this front imho. The trend is reasonably likely to continue like this, but nobody knows the future.
Ideally to increase your chances you should be able to demonstrate that you have some hands-on experience: for instance github projects, participation to a ML competition, this kind of thing.
Don't neglect your maths PhD: you seem to have a lot of free time to learn data science, good for you... But make sure the PhD is your top priority, it's going to be a great asset on your resume even if it's not related to data science.


Sampling n data points from high dimensional data

One option is to use something like seeding technique of k-means++ to pick representative sample of data points. K-means++ picks each subsequent cluster center from the remaining data points with probability proportional to its squared distance from the point's closest existing cluster center.


Null values as useful information for feature engineering

It would depend on the type of model you want to train.
If you are doing a classification model ( Yes/No, 0/1 type of class) then I suggest you use dummy variable generation functions in python.

I would also recommend that you convert all your other categorical variables into dummy variables in order for your model to take every bit of information into account.

one popular technique in feature engineering is one-hot-encoding to change the nature of your data.

So generally you would create a new variables ( as you have mentionned) that has 2 values :

1 if the variable is NULL
0 if the variable has a non-Null value
Below are some links with nice examples:

https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

https://jamesrledoux.com/code/dummies

Good luck !


Bayes Optimal Decision Boundaries for Gaussian Data with Equal Covariance

When class-conditional distributions are gaussian with equal covariance matrices, the optimal decision boundary is a hyperplane.  This is the core concept behind Linear Discriminant Analysis (LDA).

For any data point $x$, the probability that $x$ comes from class $\omega_1$ is:

$P(x|\omega_1) \sim N(\mu_1,\Sigma) = (2\pi)^{-1}|\Sigma|^{-1/2}\exp\left\{ (-1/2) (x-\mu_1)'\Sigma^{-1}(x-\mu_1)
\right\}$

Similarly, $P(x|\omega_2) \sim N(\mu_2,\Sigma)$.

Let's denote the prior probability of class 1 and 2 as $P(\omega_1)$ and $P(\omega_2)$, respectively.

Equal Misclassification Cost:

If the cost of misclassification is equal, we want to assign new data points such that the probability of misclassification is minimized. This decision rule assigns points to class 1 when $x$ satisfies:

$P(x|\omega_1)P(\omega_1) > P(\omega_2)(P(x|\omega_2)$

We can establish a similar rule for assigning points to class 2.

o find the decision boundary, we need to find the values of $x$ that satisfy

$P(x|\omega_1)P(\omega_1) = P(x|\omega_2)P(\omega_2)$

The values of $x$ that satisfy this equality here lie on a line (or hyperplane for higher dimensions).

Unequal Misclassification Cost:

For the second question, you will need an additional term.  Let's call $C(\omega_1)$ the cost of making an error when the data point was actually from class 1 and $C(\omega_2)$ the cost of making an error when the data point was from class 2.

To minimize the cost of an error, your new decision rule would be to assign $x$ to class 1 when $x$ satisfies:

$P(x|\omega_1)P(\omega_1)C(\omega_1) > P(x|\omega_2)P(\omega_2)C(\omega_2)$

As before, to find the decision boundary, you need to solve for $x$ when:

$P(x|\omega_1)P(\omega_1)C(\omega_1) = P(x|\omega_2)P(\omega_2)C(\omega_2)$

This decision boundary will still be a line / hyperplane, however it may have a different offset or orientation from the solution in the case of equal misclassification cost.


Is it a good idea to normalize the outputs of a Neural Network for Regression, when the different outputs vary in magnitude?

Yes, if proper weights were not introduced in cost function or target variables were not normalized, optimization process would be driven by the target variable which had the largest scale.

Therefore it's a good practice to normalize outputs if they have huge differences in terms of scale.


I also recommend MLXtend library for frequent itemsets.
usage example:
dataset = [['Milk', 'Onion', 'Nutmeg', 'Kidney Beans', 'Eggs', 'Yogurt'],
           ['Dill', 'Onion', 'Nutmeg', 'Kidney Beans', 'Eggs', 'Yogurt'],
           ['Milk', 'Apple', 'Kidney Beans', 'Eggs'],
           ['Milk', 'Unicorn', 'Corn', 'Kidney Beans', 'Yogurt'],
           ['Corn', 'Onion', 'Onion', 'Kidney Beans', 'Ice cream', 'Eggs']]

te = TransactionEncoder()

te_ary = te.fit(dataset).transform(dataset)

df = pd.DataFrame(te_ary, columns=te.columns_)

frequent_itemsets = apriori(df, min_support=0.1, use_colnames=True)

print frequent_itemsets


NLP - extract sentence parts related to people

This is not a standard problem but you should be able to roughly do this using two basic kinds of tools that usually go together anyway:

Use an NER system to identify people (as opposed to organizations) in sentences. Most systems have a default model that flags people.
Use an Open Information Extraction system to get relation triples from sentences like (Sarah, has, brown_hair). See OpenIE for an example. You could also use a dependency parser.

So you take the relationships you get from the tools in part 2 and throw out any where the referrent/noun isn't a person. If you have to deal with relative clauses like in your first example ("John, who was..."), you'll also have to deal with coreference resolution, which is its own complicated problem.


Can Single Node Hadoop Cluster be installed on a system with 1gb RAM

Based on this serverfault answer and this recommendation page it seems you need more RAM. The only way to know for sure is to just try.