System Requirements to train a Image Recognition Neural Network

This is not really a data science specific question, so you might want to ask elsewhere; in any case, you need to provide more information!

What error message are you receiving? Are you running out of memory? What size are the images?

Unless your images are really really large, I would not expect you to be running out of memory, given you have 32Gb. however, if you have built a massive network, where e.g. many ConvNet filters are then passed to an extremely wide FullyConnected layer (Dense layer), you will have a huge number of weights, which may result in a memory error.

Try creating your model (or a similar one) using Keras instead of pure Tensorflow, and then using model.summary on the compiled model. This will show you a nice overview of the number of weights in each layer.


How to use a one-hot encoded nominal feature in a classifier in Scikit Learn?

I think you should try pd.get_dummies to code the categories; which will create new columns in dataframe and then use that df to pass it to the classifier.


It sounds likes you have non-stationary data which can be a challenge to model.
One option is to completely discard all older data and only train/test on newer data. This way the model will only capture the newer relationships. This approach assumes there is enough newer data to train a model.
Another option is heavily regularize the model. The goal of regularization  is to increase generalization.


Multi-inputs Convolutional Neural Network takes different number of images

You need to define what your training set is in this case. If a single input to your network is a pair of images $(a, b)$, and you only have 1500 examples of $b$, then you only have 1500 training examples.

If each image in $A$ can map to any arbitrary image in $B$, you can potentially re-use images in $B$ to create these input pairs. If that's not the case, which images in $A$ correspond to the image of interest in $B$?

Alternatively, if you don't want to (or it's not appropriate to) simply re-use the images, you could consider augmenting the smaller set by, for example, rotating, cropping, or otherwise manipulating the images in that dataset.

To re-initialize the generator, you would just execute this code again :

genX2 = generator.flow_from_directory(dir2, target_size = (img_height,img_width), class_mode = 'categorical', batch_size = batch_size, shuffle=False, seed=7)

When genX2.next() doesn't return anything. Alternatively, and more pythonically, you could use itertools.chain() to form a generator capable of returning the right number of observations:

n_times_through = math.ceil(4000/1500)
genX2_new = itertools.chain(*([genX2] * n_times_through)


Keras: Dimension Error with Sparse Categorical Crossentropy

I thing you've misunderstood what the difference between categorical_crossentropy and sparse_categorical_crossentropy is. The sparse part doesn't refer to the sparsity of the data but the format of the labels.

If your labels are one-hot encoded: use categorical_crossentropy
If your labels are encoded as integers: use sparse_categorical_crossentropy


What are the standard parameters values for SMOTE technique?

First of all, if all your variables are numerical then you can you SMOTE; otherwise you should use SMOTENC.

Answers:
1- Your classes are 150-50, then SMOTE gives you 150-150. So, YES; your total dataset sample will be 300.

2- You can use the default parameters; it works well in most of the problems. However, you can play with random_state and k_neighbors. All parameters have been explained here: imblearn.over_sampling.SMOTE


Based on your post (my emphasis):

The problem I'm facing is that the some of the predictors have changed over time, for example the pressure in the machine was increased. This influenced some of the other predictors, but hasn't influenced my target.

So to be clear, the input parameters changed but made no impact on the output parameter?
There are two possibilities:

The data is part of a single population, then the change in the input parameters (over that range) has little or no impact on the output parameter.  This can be addressed by creating a new feature, which is a delta from some baseline and see if that new feature is a better fit for your model.

Pressure
Piston Position
Leakage Flow (Out)

10
20
30

11
22
30

12
23
33

The inputs into row 2 have changed relative to row 1 but no change in output. Therefore a small change in parameter 1 and 2 has no impact on output (physical examples include say overcoming hysteresis or localised energy storage or data buffer or a bucket that has to be filled with water before spilling over).
In this case the output isn't sensitive to the small changes and perhaps they're a function of the original value. So new features could be:

Pressure
Piston Position
Leakage Flow (Out)
F1
F2

10
20
30
0
0

11
22
30
0
0

12
23
33
1
1

So F1 only increases if the difference between row 1 and the other rows is greater than 1 otherwise it's 0 etc.

The data is part of more than one population. As an example, say you're taking measurements of a product. Over time the measurement device loses calibration and you see a drift in the measurement. The only real fix is to determine the drift due to a loss of accuracy and apply a corrective factor as function of time or base value.


How to load the pre-trained BERT model from local/colab directory?

You can import the pre-trained bert model by using the below lines of code:
pip install pytorch_pretrained_bert

from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForNextSentencePrediction

BERT_CLASS = BertForNextSentencePrediction

# Make sure all the files are in same folder, i.e vocab , config and bin file
PRE_TRAINED_MODEL_NAME_OR_PATH = '/path/to/the/files/containing/models/files'

model = BERT_CLASS.from_pretrained(PRE_TRAINED_MODEL_NAME_OR_PATH, cache_dir=None)


With dropout, the sum of activations feeding each neuron during training and testing needs to be roughly the same. The problem is that during training some neurons are masked out, and this does not happen during testing. Therefore there has to be a scaling difference between train and test phases using dropout, otherwise each neuron would be receiving larger magnitude inputs and the network as a whole could behave very differently.

This is not mathematically the same as averaging the weights - although the effect is compared intuitively to bagging many related neural networks and averaging their output. Instead, it is usually explained that dropout scales the neuron activations to have same expected total magnitude when dropout is active (during training) vs inactive (during testing/prediction).

Applying L2 or any other kind of parameter-controlling regularisation technique will not help with this, because these do not change any of the activations between train and test phases.

You either treat training activation values as canonical and adjust activations to be lower at test time (multiply by keep probability $p$), which is "classic dropout" or you treat test activation values as canonical and multiply activations by $\frac{1}{p}$ during training, which is called "inverse dropout". The latter approach is nowadays more common, as it is possible to completely ignore dropout layers during prediction that way.


I don't use PIG, but from looking online I think you should be able to use the Replace function built into PIG.

Assuming your Date field is already a string:

Data_ID = FOREACH File GENERATE(int) REPLACE(Date,'-','');

Here's the reference if it's helpful: https://pig.apache.org/docs/r0.9.1/func.html#replace


Question about bias in Convolutional Networks

Bias operates per virtual neuron, so there is no value in having multiple bias inputs where there is a single output - that would equivalent to just adding up the different bias weights into a single bias.

In the feature maps that are the output of the first hidden layer, the colours are no longer kept separate*. Effectively each feature map is a "channel" in the next layer, although they are usually visualised separately where the input is visualised with channels combined. Another way of thinking about this is that the separate RGB channels in the original image are 3 "feature maps" in the input.

It doesn't matter how many channels or features are in a previous layer, the output to each feature map in the next layer is a single value in that map. One output value corresponds to a single virtual neuron, needing one bias weight.

In a CNN, as you explain in the question, the same weights (including bias weight) are shared at each point in the output feature map. So each feature map has its own bias weight as well as previous_layer_num_features x kernel_width x kernel_height connection weights.

So yes, your example resulting in (3 x (5x5) + 1) x 32 weights total for the first layer is correct for a CNN with first hidden layer processing RGB input into 32 separate feature maps.

* You may be getting confused by seeing visualisation of CNN weights which can be separated into the colour channels that they operate on.


Which (naive) NLP method for correlating human messages in chatrooms?

This is a stylometry task, more exactly some form of author verification/identification task. In case you want to dig deeper, the PAN workshop series is a good source of datasets and methods.
About your method:

Your intuition is correct about selecting the most frequent words, in particular stop words: writing style is better characterized by the patterns in frequent grammatical constructs than by the choice of content words. However depending on the size of the data you might want to be more flexible about the number of words: if you have enough data, you should probably take more than the top 100 frequent words. If you don't, well... it might not work very well. Note also that some successful methods don't use the top frequent words but rather a range of middle frequency words.
Using TFIDF is controversial for style identification. It's often done simply with the term frequency instead.

Other features you could consider:

words n-grams, typically bigrams or even trigrams if there is enough data.
characters n-grams (usually trigrams) have the surprising property to be very robust features for style detection. If you use these do not apply any tokenizer or remove punctuation signs.


I assume you are performing an independent samples t-test. That the Ns are different isn't necessarily a problem -- the mean is an unbiased estimator -- but just how different are the sample sizes?
As you describe, you are probably violating the homogeneity of variance assumption.

T-tests tend to be fairly robust to this violation, but you might consider having some outlier treatment and/or using a nonparametric test.


Matplotlib: plot indicating the precision of measurements

One can use the matplotlib.axes.Axes.errorbar class.

From this example in the documentation:

# example error bar values that vary with x-position
error = 0.1 + 0.2 * x

fig, (ax0, ax1) = plt.subplots(nrows=2, sharex=True)
ax0.errorbar(x, y, yerr=error, fmt='-o')
ax0.set_title('variable, symmetric error')

Acknowledgments: Thanks to Emre!


Extract datatable column by indexing a character vector

You need [[]] as otherwise you are attempting to index the row.

a[[col_names[1]]]


Interesting question.  I haven't worked with this sort of data much, but it seems to me that the bulk of the job is likely to be feature engineering.  Every "supervised" statistical method that I know of requires that you shoe-horn your data into "outcomes" and "covariates."  $\mathbf{Y}$ and $\mathbf{X}$.  Once your data is in this form you simply find an appropriate algorithm that can estimate

$$
\mathbf{Y} = f(\mathbf{X})
$$

But taking data on how people play a video game and turning it into a 2-dimensional matrix isn't necessarily obvious.  One of the most difficult things is that you'll need to make observations of different users comparable.

Say you're playing quake 3 and you're trying to predict whether the user is going to go for the BFG.  How do you define $\mathbf{Y}$?  Is it got BFG? Or should it be came within some distance of BFG?  (maybe people messed up the rocket jump, for example).  What if they got fragged on the way to the BFG?  The point is that defining the outcome involves choices.

Likewise what are the covariates?  Say I get spawned in some location, and I know the map, and immediately move toward the BFG.  I'll have a different set of w, a, s, and d keystrokes than someone who got spawned elsewhere on the map.  So is it n-grams that you want to look at?  Maybe rather it is change in euclidian distance between spawning location and BFG location over some time interval.  if so, what is the appropriate time interval?

In general, what you're doing is taking raw data and turning it into abstractions of that data.  This is actually what a neural net does.  It takes raw data $\mathbf{X}$ and forms a lower-dimensional representation of that data, $\mathbf{V}$, where $\mathbf{V}$ has fewer columns than $\mathbf{X}$.  When neural nets work well, those columns of $\mathbf{V}$ can be uncannily similar to things that a human would pick out of a stream of raw data.  Akin to this:  https://www.youtube.com/watch?v=3vAnuBtyEYE

Those $\mathbf{V}$ are then just related to your outcome $\mathbf{Y}$ in a linear model or a logistic regression.

But you don't turn raw keystrokes into blonde, brunette and redhead without a lot of data and long training times.  It'll go faster if you can prespecify functions of your raw data, which requires domain knowledge.


Why predicted proababilities from this binary classifier does not sum up to 1?

You are absolutely right to be skeptical about the results you are getting. I've been using boosted trees (not necessarily C5.0) for over 5 years, and I've never encountered results like what you've described here. Excluding the possibility that you simply had a typo in your code, I can think of two reasons:

Low decision threshold: As is often the case in class imbalance problems (e.g., fraud detection, marketing response, bankruptcy, etc) a low probability could still indicate the minority class if the decision threshold is also low. In your case, if the model might have automatically decided to pick a threshold of 0.05, then 0.1 would be flagged as C1.
Cost function: This is what you mentioned -- if your model has a high cost associated with missing true positives then 0.1 could be classified as C1.


Most people are using Python or R for data-science, I like Python more because python have enormous amount of libraries for datascience related tasks (for data-analysis, plotting, out of the box machine-learning & deep learning algorithms) and huge open source community to support/help.
You can read more about individual language capabilities everything can be done using any of these languages (Python/R/Matlab) it's just the matter of perspective.


Do I need to square a column if I want a neural network to try using that?

You don’t necessarily need to, according to the universal function approximation theorem.

It is easier for a neural network to learn an identity function than some other function, so if one of the inputs definitely needs to be squared your network will learn faster if you pass the input already squared
If your network is sufficiently large it should work out that squaring that input is helpful and approximate the squaring function as part of the overall learning process


What are the drawbacks of traditional clustering algorithms?

Check out this great answer for K-means in particular:
How to understand the drawbacks of K-means
My favorite resource for comparing the strengths and weaknesses of clustering algos is from sklearn's docs:
Comparing different clustering algorithms on toy datasets


How to deal with millions or rows of data for analysis/visualization purpose

Pandas load everything into memory before it starts working and that is why your code is failing as you are running out of memory. One way to deal with this issue is to scale your system i.e. have more RAM but this is not a good solution as this method will eventually fail to scale.

Other option will be to use big data libraries like spark or flink for this on a distributed system. You can try the installation on windows sever but that also wont scale well for you.


Python and data analytics how to get the unique number of orders per seller using groupby?

Based on the information available above I think the solution below should work

# Number of unique orders per seller
a = pd.pivot_table(df, index = ['seller_id'], values = ['order_id'],
                   aggfunc = {'order_id' : pd.Series.nunique})

# Products that are being ordered the most per seller
# Products that are being ordered the most in general
b = pd.pivot_table(df, index = ['seller_id', 'name'], values = ['product_id'],
                   aggfunc = 'count')

# Sum of the quantity field for each product to check how many times has been order
c = pd.pivot_table(df, index = ['name'], values = ['product_id', 'quantity'],
                   aggfunc = {'product_id': 'count', 'quantity': 'sum'})

# Number of orders for each customer
d = pd.crosstab(df.customer_id, df.order_id)


Keras functional api explanation of activation() layer?

As stated in the docs, the activation layer in keras is equivalent to a dense layer with the same activation passed as an argument.

This would be equivalent

x = Dense(64)(x)
x = Activation('relu')(x)

is equivalent to

x = Dense(8, activation='relu')(x)

As per your example if the activation layer is used as a layer, this will act as a transformation of the outputs of the previous layer.

As you can see in the model

inputs = Input(shape=(784,))
x = Dense(32, activation='tanh')(inputs)
x = Activation('relu')(x)
predictions = Dense(8, activation='softmax')(x)

The output of the first layer is the result of a densely connected layer with a tanh function. Then these outputs will each be transformed by a relu function. You can see how this is simply a transformation and does not introduce any new model parameters using

model.summary()


A dead ReLU pretty much just means that its argument value is negative such that the gradient stays at 0; no matter how you train it from that point on. You can simply have a look at the gradient during training to see whether a ReLU is dead or not.

In practice you may simply want to use leaky ReLUs, i.e. instead of f(x) = max(0,x) you set f(x) = x if x > 0 and f(x) = 0.01x if x <= 0. This way you always allow a small non-zero gradient and the unit should not get fully stuck in training anymore.


Your line results = model_selection.cross_val_score(ensemble, X, y) just returns an array of scores for each run of the cross validation, so the error is telling you that it's just an array of numbers, not a model itself and has no defined method for fitting things.

To use predict, you need to call it on a model object, which I'm assuming you're going for the VotingClassifier. Your ensemble variable has a fit method, so you want to call

predicted = ensemble.predict(X_new_tfidf)


decision trees on mix of categorical and real value parameters

Unfortunately scikit-learn does not handle categorical features well - they must be encoded.  One-hot-encoding is great and can be implemented in sklearn very easily.  So if you have an array of feature columns, X, and a class label vector, y

from sklearn import OneHotEncoder
ohe = OneHotEncoder(categorical_features=[0])
X = ohe.fit_transform(X).toarray()

will perform the encoding for you (The categorical_features attribute gives the index of the feature you want encoded).  More to your point though, if your feature has many levels, one-hot encoding can leave you with a sparse and inefficient X array.  In this case it may be beneficial to do a count transform, possibly replacing each level with its corresponding log-odds ratio.

Count Transforms

The link above does a more thorough job explaining it, but, in layman's terms, for each categorical feature you:

count the number of times a level(category) belongs to class0 or
class1
add 0.5 to each count (this takes care of instances when a level belongs strictly to one class)
calculate the probability that it belongs to either class.
calculate the log of the odds ratio
replace each level with its corresponding log odds ratio

In this manner you get numerical values representing each of the feature's many levels without creating a dummy variable for each level.  Still, 20 variables is a lot, and you may want to use PCA for data compression and dimension reduction.

One of the great things about tree based methods in general is that they do not require standardization.  That being said, from a data science perspective, I would train several different models (tree based methods, svms, logistic regression, knn, etc.), and see which one yields the best results.  All methods besides CART benefit from scaling, so try

from sklearn.preprocessing import StandardScaler
stdsc = StandardScaler()
X_train_std = stdsc.fit_transform(X_train)
X_test_std = stdsc.transform(X_test)

I hope that answers most of what you're asking.  I'm not sure why you would want to concatenate two very different variables like cities and urls, but if you encode them or do a count transform your X array should be just fine and you can leave them separated.  Also, you shouldn't have to worry about the length of a given variable - python is object oriented (if you're getting error messages complaining about 'length', you've likely just confused your indices).


Association rules: Find the recipe for a list of ingredients

It doesn't seem like an association rule problem. It seems that what you suggest could be simply solved using set operations.

First, create mappings between 1 ingredient and all recipes containing it. Then, if you want to find all possible recipes based on a list of ingredients, it would be identical to the union of all of their "recipe sets".


MCMC vs Network Theory for Big Data and Data Science applications

Both MCMC methods and network analysis play an important role in data science. I think you should go for the project you like more. However, in my experience, community detection in networks is a niche (applied math/graph theory) of network analysis, while MCMC methods involve lots of statistical and computational concepts. Personally, being a statistician, I would go for the MCMC methods.


How to train NLTK Sequence labeling algorithm for using custom labels/Train set?

Have a look on conditional random field. Conditional random fields are a probabilistic framework for labeling and segmenting structured data, such as sequences, trees and lattices. The underlying idea is that of defining a conditional probability distribution over label sequences given a particular observation sequence, rather than a joint distribution over both label and observation sequences. The primary advantage of CRFs is their conditional nature, resulting in the relaxation of the independence assumptions. Additionally, CRFs avoid the label bias problem. It is often used for labeling or parsing of sequential data.
CRF++ is a simple, customizable, and open source implementation of Conditional Random Fields (CRFs)


Check out this paper.  It also addresses question of how many clusters to use.  The R package mclust has a routine which will try different cluster models/number of clusters and plot the Bayesian inference criterion (BIC). (great vignette here).  It's a general method, meaning, something you can do without being domain/data specific.  (It's always good to be domain-specific if you have the time and data.)

The chart is from the vignette by Lucca Scrucca.  MClust tries 14 different clustering algorithms (represented by the different symbols), increasing the number of clusters from 1 to some default value.  It's finds the BIC each time.  Highest BIC is usually the best choice.  You could apply this methodology to your own stable of clustering algorithms.


Do I load all files at once or one at a time?

Since the process can be run independently on every file/batch, I would tend to recommend processing each file one by one for the sake of scalability:

Depending on the details of the task there could be some minor optimizations which can be done only by processing the whole data at once, like things related to I/O and memory usage. I can't think of any significant improvement obtainable this way.
By itself processing the files independently won't bring any performance improvement either, but it has a massive advantage in terms of scalability: the process can be distributed easily by using multiple cores and processing any number of files on each core. This option offers flexibility for processing the current data (if you can afford multiple cores/machines, the process can be faster) and importantly any future version of the data, even if it grows very large.


How can I find similarities in two graphs?

Well, you need to first define what your threshold for 'similar' is, and also what length of similarity is meaningful to you.

One way of achieving this is by taking a 'slice' of the first set of coordinates, and comparing them against each slice of the same size in the second set. If all values are within a certain threshold distance, bingo.

You can then repeat this with the next slice of coordinates from set #1.

e.g. here is an O(n2) implementation:

slice_len = 10
thresh = 2
overlap_x1 = []
overlap_x2 = []

for i in range(len(x1)-slice_len):
    for j in range(len(x2)-slice_len):
        # checking the y coords are all at most 'threshold' far away
        if max(abs(y1[i:i+slice_len]-y2[j:j+slice_len])) < thresh:
            # Adding the similar x-coords to the containers
            overlap_x1.append(x1[i:i+slice_len])
            overlap_x2.append(x2[i:i+slice_len])

# Converting arrays to ordered sets to remove duplicates from overlap
# Since they are x-coords, they are monotonic increasing, order is preserved
overlap_x1 = OrderedSet(overlap_x1)
overlap_x2 = OrderedSet(overlap_x2)


why always predict same class for all input

I solve this problem with adjusting learning rate and adding regularization and some optimization


How to create ROC - AUC curves for multi class text classification problem in Python

First check out the binary classification example in the scikit-learn documentation. It's as easy as that:
from sklearn.metrics import roc_curve
from sklearn.metrics import RocCurveDisplay
y_score = clf.decision_function(X_test)

fpr, tpr, _ = roc_curve(y_test, y_score, pos_label=clf.classes_[1])
roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()

In the case of multi-class classification this is not so simple. If you have 3 classes you could do ROC-AUC-curve in 3D. Have a look at the resources here.
What you can do and what is much simpler is to make 4 one-vs-all curves. You basically have a binary setting for each class.
In your case:
import matplotlib.pyplot as plt
# all the same up until now
prediction = learn.predict(x_test_tfidf)
proba = learn.predict_proba(x_test_tfidf)
print("Accuracy is of Multinomial Naive Bayes Classifier", accuracy_score(test.target, prediction) * 100)

for i in range(len(categories)):
    y_test_bin = np.int32(test.target == i)
    y_score = proba[:,i]
    fpr, tpr, _ = roc_curve(y_test_bin, y_score, pos_label=0)
    plt.subplot(2,2,i)
    roc_display = RocCurveDisplay(fpr=fpr, tpr=tpr).plot()


How can I get rid of the white space in matplotlib?

This stackoverflow answer should solve your issue. You can use either plt.margins(x=0) or ax.margins(x=0) to remove all margins on the x-axis.


How to handle features which are not always available?

If the feature doesn't make sense in a subset of the samples, doesn't this mean that this is (or should be) a separate dataset, that needs a second model? That's one approach I'd think about.

The second would be to work with the data (feature) itself. It's probably best to use neutral value.

In case of numerical value:

try using a mean or median value, calculated on all entries
try using an extreme value, e.g. -1 if your feature has only positive values. This should indicate that the feature is missing and the network should be able to handle it.

In case of textual value, e.g. word embeddings, replace the value with a placeholder like N/A that doesn't have an embedding


First my understanding of your problem. You want to find the best hyperparameters for a Random Forest.

For that, you want to first adjust n_estimators parameter and then the rest of parameters in different runs.

Before answering to your question, you will only want to do a thorough search of hyperparameters when you want to have an improvement of around 1%. So it will be a small improvement. If you want to improve your model, probably feature engineering or data engineering will give you a better improvement. Or even a different algorithm.

You can print the results of your GridsearchCV with

pd.DataFrame(clf.cv_results_)

The answer to your question:

No you shouldn't run a GridsearchCV in different runs you have to explore the whole parameters if you want to find the global minima. A small change in one parameter can affect other. At the end you are exploring a search space.


Using t-SNE to track progress of a word vector embedding model. Pitfalls?

T-SNE is extremely useful for visualizing high-dimensional data in lower-dimensional space. However, t-SNE can have several gotchas, including comparing cluster sizes. The t-sne algorithm tries to even out cluster sizes by expanding dense clusters and contracting sparse clusters. Thus, it is not straightforward to directly compare clusters across different runs.

"How to Use t-SNE Effectively"  goes into greater detail about common pitfalls of using the technique.


Better code completion in python notebooks

If you are working locally (running e.g. starting up a Jupyter notebook from a terminal on localhost) code completiong should work just fine - I have never had any problems.

If you are running a Jupyter notebook attached to a Python process in a Docker container, there can be issues with code completion (the two-way communication may need to be activated).

You can try running the following ipython "magic command" that makes the interpreter greedy - this has worked for me in the past:

%config IPCompleter.greedy = True

Simply run that in the first Jupyter cell.

Here is a list of all built-in magic commands.

I am afraid I don't know of any differences in these cases between jupyter notebooks and Spyder.


Regularization hyperparam tuning during training

As mentioned above you can try a couple of things, depending on which framework you are using and how you want to go with hyperparameter optimisation.
sklearn (with Keras wrappers):

GridSearchCV: slow but sure to find optimal hyperparams
RandomizedSearchCV: faster and almost as good as GridSearch

Keras (with Keras Tuner):

BayesianOptimization: tuning with Gaussian process
Hyperband: Variation of HyperBand algorithm (Li, Lisha, and Kevin Jamieson. Journal of Machine Learning Research 18 (2018): 1-52.)

Pytorch (with RayTune):

Population Based Training: trains a group of models (or agents) in parallel (https://deepmind.com/blog/population-based-training-neural-networks)
HyperBand: as above
ASHA:Compared to the original version of HyperBand, this implementation provides better parallelism and avoids straggler issues during elimination
Median Stopping Rule: implements the simple strategy of stopping a trial if its performance falls below the median of other trials at similar points in time.
FIFOScheduler: Simple scheduler that just runs trials in submission order


How to go about text mining for suggestions/Tips in reviews for restaurants?

So the way I thought about it is - abstractly you are attempting to identify which sentences are, Imperative Mood (IMP) and which are not. One option is to put together a training set/validation set, where each row is a phrase's pos tags (can use some encoder to convert to integers) with a target value of 'isImperative' (binary) then train a model and restructure each review so that they resemble the training set format and have your model assign the values. Seems the easiest and most sustainable method from where I am sitting. Sorry but I don't know any easily accessible phrase data sets but I bet there is one out there somewhere. Hope this helps.


What is the difference between Freidman mse and mse?

As Sergey has pointed out on the original question, going through the following link one finds the reference to the original article. So, we're going to proced by unraveling the components of each formula (explaining each of them) and then doing a proper interpretation of the splitting criterion that Jerome Friedmann proposed.

1. Probabilities $p_{k}$

First, let's consider
\begin{align}
    p_{k} = exp(F_{k}(x)) / \sum_{l = 1} ^{K} \exp(F_{l}(x)),
\end{align}

where $F(x)$ is a logistic transformation. The term $p_{k}$ determine an exponential way to consider probabilities to belong to a particular $k$ class, that gives a mayor probability to the events that are more likely to happen and a lesser probability to the ones that are less likely to happen. For example, consider that the values that $F(x)$ could take are 1, 7 and 49, thus

\begin{align}
     \frac{\exp(49)}{\exp(1) + \exp(7) + \exp(49)} \approx .9999 > \frac{\exp(7)}{\exp(1) + \exp(7) + \exp(49)} \approx 0.
\end{align}

In other words, greater values of $F(x)$ are more likely to happen.

2. Weights $w_{l}$

Once we know that $p_{k} \in [0,1)$, we arrive at the definition of the weights

\begin{align}
    w_{l}  = \sum_{i \in R_{l}} w_{l}(x_{i}) = \sum_{i \in R_{l}}p_{k}(x_{i})(1 - p_{k}(x_{i})),
\end{align}
for the $x$'s in the region $R_{l}$. In the other words, the weights are the sum of the probabilities, that in the given region $l$, every $x_{i}$ belongs to a certain $k$-class. This is fundamental to understand the criteron proposed by Friedman, as we would see next.

3. Least-Squares Improvement Criterion

The least-squares improvement criterion is

\begin{align}
      i^2(R_{l}, R_{r}) = \frac{w_{l}w_{r}}{w_{l} + w_{r}}(\overline{y_{l}}-\overline{y_{r}})^2  \:\:\:\:\:\:\:\:(1),
\end{align}

considering the fact that we would like to split the current region $R$ into two subregions $(R_{l}, R_{r})$. Remember that at the end we would like to do the split as clean as possible, that is to say, the outcomes are close to the average outcome of all $k$ cases at that node. Therefore, this splitting criterion allow us to take the decision not only on how close we're to the desired outcome, but also based on the probabilities of the desired $k$-class that we're going to find in the region $l$ or in the region $r$.

Disclaimer: This is what I understood by reading the original paper by Friedman, any errors should be pointed out. Thanks


How to properly resize input images for transfer learning

This can be accomplished using the PIL library in Python.

One thing to note - if you are resizing the images to 224x224, you might want to keep the proportion of your image, e.g. if the height and width is significantly different, then you might lose perspective by resizing in this way.

If proportion is not an issue, you can resize as follows:

from PIL import Image
baseheight = 224
img = Image.open('image.jpg')
width = 224
img = img.resize((width, baseheight), Image.ANTIALIAS)
img.save('resizedimage.jpg')

That said, if you wanted to keep the same proportions as the original image and only wanted to set height to 224 for instance while keeping a proportional width, you could also use PIL as follows:

from PIL import Image
baseheight = 224
img = Image.open('image.jpg')
hpercent = (baseheight / float(img.size[1]))
width = int((float(img.size[0]) * float(hpercent)))
img = img.resize((width, baseheight), Image.ANTIALIAS)
img.save('resizedimage.jpg')


What does "baseline" mean in the context of machine learning?

A baseline is the result of a very basic model/solution. You generally create a baseline and then try to make more complex solutions in order to get a better result.
If you achieve a better score than the baseline, it is good.


Why is it bad to use the same test dataset over and over again?

In fact this will introduce some bias to your model because you will distinctly choose a model that performs well on the testing set. This is indeed what bias consists of. Ideally you would want to have a training set, a validation set and a testing set. You will only use the testing set right before writing the spec of your model. If the performance is much poorer than expected at this very last step, then you need to restart from the top. Get new data to reduce "the polluting" of the testing set and retrain and validate until you are satisifed then use the testing set.

Consider if your testing set does not generalize well on the actual distribution, this may be improbable but it is possible. Then a good model will perform poorly on this test set, however it would otherwise be a good model if you did not bias your decision using your test set.


Random Parameters to fix in ML to perform controlled experiments

Scikit uses numpy for pseudo-random number generation. So to fix random state in various scikit calls, you use numpy.random.seed(12345) and then use scikit. You would want to record the random seed when you log the model so you could reproduce the same run later.

If your code (or something you call) also uses Python's random number generator, you would set random.seed too.

How to set the seed depends on the library. For example I believe most xgboost APIs expose a seed parameter instead. Not sure about catboost.

You're also depending on the library even exposing a way to seed any pseudo-random choices it makes. It's possible some library doesn't quite do so. (Or in the case of Spark, sometimes the result can even conceivably depend on order of distributed execution, which is hard to control.)


Can linear regression or any other models be used to judge if Y increases as X increases?

I don’t think linear model is a good idea for your problem, because it can capture only linear pattern. Furthermore, if I understand you correctly the results for different sites will not be comparable due to different scales/ slopes.

Instead I suggest using the Spearman rank correlation coefficient.


How do we go about imbalanced data for prediction problem?

In addition to post linked by @Wes:

You should to check other metrics                than accuracy like
precision, recall (especially), F1 score or similar.
After a certain threshold (lower than the one provided in your
example definitely), it might be worthwile to use anomaly detection
and use your lesser present classes as such.

I would advise against downsampling as it's leaving informations about data, and IMO you would usually want to avoid that.


How to predict the sentiment of the entities form the tweet?

If you have a fixed number of entities and a labelled dataset, then you can build a CNN multilabel classification model. The final layer of model has Nx3 binary outputs, where N is the number of entities. Each entity has 3 outputs, representing positive, negative and a NA sentiment. The NA is important, an entity may not be represented in a tweet.
The CNN is recommended over a standard dense network so that the model can take advantage of signal words in the neighborhood of entities to estimate the sentiment.


LDA is a unsupervised technique which identifies the set of words that compromise of a topic.
This is done by assuming that there is a generator that generates the sentences using the words.
To simplify the assumptions on the number of topics are made which ideally has to be learnt from the data.

The output of LDA is the words that represent a topic and probability of a sentence/ text observation being a mixture of the N topics we initially assumed.

We will get a 1xN row vector for each text observation that has probability of it containing topic n (which is one of the N topics)


How to select features for clustering to detect the number of different unique products in a search result?

You cannot expect clustering to understand what a "product" is. So it will not make one cluster per product.

Also, the lines are very blurry: is the same mobile phone in different colors the same product, or not?

An unsupervised method cannot do this. You need to provide training data and use supervised learning.


How to split pieces of dataframe and create new dataframes based on it?

One way of doing this is explained here, for example:

import pandas as pd
import numpy as np

df = pd.DataFrame([('bird', 'Falconiformes', 389.0),
                   ('bird', 'Psittaciformes', 24.0),
                   ('mammal', 'Carnivora', 80.2),
                   ('mammal', 'Primates', np.nan),
                   ('mammal', 'Carnivora', 58)],
                   columns=('class', 'order', 'max_speed'))

Simply do the following, would return groupby separated dataframe, stored in a list:

dfs = [x for _, x in df.groupby('class')]

#dfs
[  class           order  max_speed
 0  bird   Falconiformes      389.0
 1  bird  Psittaciformes       24.0,
     class      order  max_speed
 2  mammal  Carnivora       80.2
 3  mammal   Primates        NaN
 4  mammal  Carnivora       58.0]

Please note here you should lookup via list index not the class names, for example df[0]:

   class    order           max_speed
0   bird    Falconiformes   389.0
1   bird    Psittaciformes  24.0

And df[1] the other class!


How do standardization and normalization impact the coefficients of linear models?

When you have a linear regression (without any scaling, just plain numbers) and you have a model with one explanatory variable $x$ and coefficients $\beta_0=0$ and $\beta_1=1$, then you essentially have a (estimated) function:
$$y = 0 + 1x .$$
This tells you that when $x$ goes up (down) by one unit, $y$ goes up (down) by one unit. In this case it is just a linear function with slope 1.
Now when you scale $x$ (the plain numbers) like:
scale(c(1,2,3,4,5))
           [,1]
[1,] -1.2649111
[2,] -0.6324555
[3,]  0.0000000
[4,]  0.6324555
[5,]  1.2649111

you essentially have different units or a different scale (with mean=0, sd=1).
However, the way OLS works will be the same, it still tells you "if $x$ goes up (down) by one unit, $y$ will change by $\beta_1$ units. So in this case (given a different scale of $x$), $\beta_1$ will be different.
The interpretation here would be "if $x$ changes by one standard deviation...". This is very handy when you have several $x$ with different units. When you standardise all the different units, you make them comparable to some extent. I.e. the $\beta$ coefficients of your regression will be compareable in terms of how strong the variables impact on $y$ is. This is sometimes calles Beta-Coefficients or Standardised Coefficients.
A very similar thing happens when you normalise. In this case you will also change the scale of $x$, so the way how $x$ is measured.
Also see this handout.


how to derive the steps of principal component analysis?

\begin{align}
\frac1N\sum_{i=1}^N Y_{1i}^2 &= \frac1N\sum_{i=1}^N (v_1'z_i)(z_i'v_1)\\
&=v_1'\left(\frac1N \sum_{i=1}^N z_iz_i' \right) v_1 \\
&= v_1'Rv_1
\end{align}

where $R=\frac1N \sum_{i=1}^N z_iz_i'.$


I'm going to assume your using python and scikit-learn mostly because it has a method for providing model metrics.

from sklearn.metrics import classification_report

# I presume that you've already trained a model and it's saved as xgb
# X_test is your testing X data (NOT THE DATA YOU TRAINED ON!!!!)
# Y_test is the corresponding correct values
print('Accuracy score is: ',accuracy_score(Y_test, xgb.predict(X_test)) * 100)
print(classification_report(Y_test, xgb.predict(X_test)))
>>> Accuracy score is:  61.13989637305699
>>>           precision    recall  f1-score   support

           0       0.38      0.98      0.55        47
           1       0.99      0.49      0.66       146

   micro avg       0.61      0.61      0.61       193
   macro avg       0.68      0.74      0.60       193
weighted avg       0.84      0.61      0.63       193

As you can see there is lots of info available.
The accuracy score is the % of correct predictions overall.
And the classification report goes into more depth about how good the model is predicting each class more info here crossvalidated SE explanation.


Predicting next medical condition from past conditions in claims data

I've never worked with medical data, but from general reasoning I'd say that relations between variables in healthcare are pretty complicated. Different models, such as random forests, regression, etc. could capture only part of relations and ignore others. In such circumstances it makes sense to use general statistical exploration and modelling.

For example, the very first thing I would do is finding out correlations between possible precursor conditions and diagnoses. E.g. in what percent of cases chronic kidney disease was preceded by long flu? If it is high, it doesn't always mean causality, but gives pretty good food for thought and helps to better understand relations between different conditions.

Another important step is data visualisation. Does CKD happens in males more often than in females? What about their place of residence? What is distribution of CKD cases by age? It's hard to grasp large dataset as a set of numbers, plotting them out makes it much easier.

When you have an idea of what's going on, perform hypothesis testing to check your assumption. If you reject null hypothesis (basic assumption) in favour of alternative one, congratulations, you've made "something real".

Finally, when you have a good understanding of your data, try to create complete model. It may be something general like PGM (e.g. manually-crafted Bayesian network), or something more specific like linear regression or SVM, or anything. But in any way you will already know how this model corresponds to your data and how you can measure its efficiency.

As a good starting resource for learning statistical approach I would recommend Intro to Statistics course by Sebastian Thrun. While it's pretty basic and doesn't include advanced topics, it describes most important concepts and gives systematic understanding of probability theory and statistics.


What Machine Learning Algorithm could I use to determine some measure in a date?

I would calculate the daily deltas of your sales data and implement a time series forecast model by using an Azure ML R/Python model module that predicts the daily deltas until the end of the month and returns the resulting monthly sum.

If you are not familiar with time series analysis, here is a link to a good description of some basic methods and how to apply them using R.

I think, would predict the daily deltas using an ARIMA model. I found this website containing a description of a time series analysis work flow using R with which you could find a good ARIMA model.

I hope, the above is useful for you.


How to understand the performance of different machine learning models?

First, it would be beneficial if you could mention whether that $R2$ presented in the figure is for the test or training dataset.
Let's assume that it is for the test dataset.
The answer to all of your questions is subjective and depends on the type of data you used for this purpose. But let me briefly answer them.
Because the Random Forest model (RF) is an ensembling method (a combination of weak learners), it is expected to have a very good performance. This method is robust to overfitting. We can see that when the number of features increases, there is no drop in the performance of the RF model; while using a large number of features could lead to a drop in the performance of the model on the test dataset. RF model is not prone to overfitting because of using many weak learners (decision trees here).
Linear Regression and SVM probably perform the same because you might use a linear kernel for SVM. If that is not the case, it means that transforming the data to a new space (SVM transforms the data into new space to easily separate the classes) is not useful in your dataset because of the nature of your data (it means that your data might be linearly separable).
It seems that you did not use regularization for linear regression. If that is true, your model is overfitted especially when you have a large number of features. If you add regularization to your model, the performance of the linear regression model does not decrease as the number of features increases.
Naive Bayes is good when features are independent and when dependencies of features from each other are similar between features. So, if they are not true for your dataset that might be the reason that Naive Bayes does not work well.
Also, Naive Bayes classifiers can quickly learn to use high dimensional features with limited training data because of the independence assumption. So, it is expected that Naive Bayes works better when the number of features >> sample size compared to more sophisticated ML algorithms. But we can see that is not the case for your study. Probably, it is because of the violation of the feature's independence.


receive value error decision tree classifier after one-hot encoding

It looks like Y is a SpareSeries as well as y_train and y_test. So when that is passed to the decision tree fit method, it only interprets those entries with label 1 as existing. According to the pandas documentation:

  We have implemented “sparse” versions of Series and DataFrame. These are not sparse in the typical “mostly 0”. Rather, you can view these objects as being “compressed” where any data matching a specific value (NaN / missing value, though any value can be chosen) is omitted.

I'm not sure why it is a sparse data structure, but you can use the to_dense method to densify it:

Y = df.iloc[:, 23].to_dense()

Edit: Danny below mentions you could just remove Sparse=True from get_dummies.


What techniques to use for image matching

Hashing is the way to go if you want fast -- constant time -- retrieval of nearest neighbors. Here's a recent example using neural networks to learn a binary hash:

Deep Learning of Binary Hash Codes for Fast Image Retrieval (code) (slides)

You want to avoid doing all-pairs computations like correlations.


Building a text classification model from scratch

To train a model on your collected data, you need to label your text first. For example, given a sentence

  Cats can fly.

It need to be labelled as 'Faked'. Otherwise your model wouldn't know how to classifier between 'Faked' and 'Not Faked'.

However, labelling needs huge effort. So if your data doesn't have any label, I suggest you use a trained model to predict or train a model on an existing sentiment classification dataset. You can search on the Google for the appropriate model of sentiment analysis.


Missing population values in census data

I think you need to be wary of using curves to extrapolate beyond the age thresholds - specifically I think you should consider:

mortality increases with age; I would imagine it increases at an increasing rate with age (especially at higher ages). Would you be able to capture this effectively?
there is obviously some sort of overlap between populations in subsequent years (e.g. someone who is included in 1985 data could also be in 1986 data). What effect would this have on the data imputation?

I would suggest a different approach. Actuaries have traditionally produced "life tables" which capture mortality in population cohorts. If you can find a set of tables which is applicable to the period in question, you could use these to calculate population numbers.


Machine learning PhD Interview technical questions

In general, linear algebra and basic concepts in statistics (probability distributions, marginalization) and machine learning (you should be familiar with terns like Maximum-A-Posteriori estimation and Maximum-Likelihood-Estimation, maybe Markov chains and SVMs, neural-networks, etc.).

Best practice would be to find out who is interviewing you and than look at what courses he is leading and what papers he has published in recent years. He is most likely to ask specific question from his field of expertise and interests.


How to load pre-processed .txt data and use it for training convolutional networks?

I did not really feel clear on what you were trying to accomplish. But here is how you load the format you showed into a numpy array:

import numpy as np

# Read file from disk
with open('path/to/file.txt') as f:
    txt = f.read()

# Make a list of values in string format
vals = txt.replace('{','').replace('}','').strip().split()

# Cast values to floats
vals = [np.float32(val) for val in vals]

# Store in numpy array
vals = np.array(vals)


Target encoding before split on skewed data

R2 and encoding before split are both can be misleading. At first, target encoding, I wouldn't recommend to do it before the split - this is basically target leak. There're at least two major downsides of doing target encoding before split:

1) Overfitting - putting knowledge about your target into features during training always tend to overfit.
2) Impossible to do the same in production. Handling unseen categories is the thing which should be supported in production as well.

Running well-done N-fold cross-validation can help you check how are you handling unseen categories "in average".

There's a good example of target encoding I always share if anyone asks me about this technique. Also for highly skewed data, I can suggest using log(y) when you encoding your feature, that may help as well.

Also, I would recommend reading about R2 (from my experience it's very bad metrics for non-linear problems):
 - Why R Squared is Useless?
 - another post about R2

I would rather use MAPE/MAE or some metric which fits your task. Looking at your distribution, I guess it shouldn't be sensible to outliers.


Different results for LogisticRegression on python 2.7 and 3

Assuming that you are using sklearn, if you go to the source in the github
you will see that they do not differ.

But you have variability elsewhere. For example train_test_split of sklearn also takes random seed, so it could be that your data is fundamentally different between these approaches (you could do cross valdiation for example to include all the data in train and test)


Sequence labeling with partially known labels

You can train a regular sequence labeling model (typically CRF) where one of the features is the rule-based predicted label: its value is the actual label when known or a special unknown value otherwise. Given that the model can take into account dependencies between labels (as specified in the parameters) and that the rule-based feature always gives the label except if unknown, the model should be able learn:

in which cases it should "trust" the rule-based feature (when value is not unknown)
to predict the missing labels by exploiting both the features and the known labels.

Note that in some rare cases the model might still predict a wrong label even if the rule-based label is provided. That would happen if the probability of the sequence is maximized in this way, but it's unlikely to happen if the training data is representative of the distribution.


After fitting the model you can use predict_proba( ) from the docs here

This returns a numpy array with the probability of each data example being of a given class. The three highest probabilities will be your best 3 predictions.


If you can afford to do the full join once, do it and learn which columns are useful through feature selection. Then you can only SELECT these columns for subsequent iterations, when the database is updated.

Here's a survey: Feature Selection for Clustering: A Review


In high dimensional spaces like the one-dimensional space, you have to employ a very simple network, maybe just a single neuron and investigate whether your data is linearly separable in that feature space which can not be visualized. If you observe that you do not have a good performance, you can increase the size or the layers of your network step by step. You can generalise what I've just referred to other networks like RNNs.

What I've referred means that your data may be in a way that is separable though it's in 1000 dimensional space and you just need one neuron to classify it.


What information does output of [SEP] token captures in BERT?

You are right, CLS token tries to capture the sentence representation, because during pretraining this token is used to decide if 2 sentences are contiguous or not.

But the author mentioned that CLS token was not supposed to be a sentence representation, and should be used carefully.

The SEP token is a token used simply to separate sentence, in order to make is easier for BERT to know that the input is made of several sentences. Since the SEP token is not used at pretraining time, the SEP token does not represent anything.

About your last question : We don't know.

In pretraining, the model was not train with several sentence, so the model will not know how to handle several sentence.

But if you finetune it, the model may be able to learn new representation and can learn so that the CLS token contain information about all the sentences.

For example, this code finetuned BERT with a new pattern :

[CLS] Sen 1 [SEP] [CLS] Sen 2 [SEP] [CLS] Sen 2 [SEP] ...

And the CLS token is used to represent each sentence. Because the model is finetuned, BERT is learning to represent each sentence in the corresponding CLS token.


Doc2Vec (and words vectors) need significant amount of data to learn useful vector representation.  50k sentences is not sufficient for this.   To overcome this, you can feed word vectors as initial weights in Embedding Layer of network.
For example, code from following question :
How to implement LSTM using Doc2Vec vectors?
model_doc2vec = Sequential()
model_doc2vec.add(Embedding(voacabulary_dim, 100, input_length=longest_document, weights=[training_weights], trainable=False))
model_doc2vec.add(LSTM(units=10, dropout=0.25, recurrent_dropout=0.25, return_sequences=True))
model_doc2vec.add(Flatten())
model_doc2vec.add(Dense(3, activation='softmax'))
model_doc2vec.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Output of "Flatten" layer will be vector representation of a sentence / document.
Article with example code.


How do I get the latitude and longitude information contained csv file?

If what you have are places names and you want to get its coordinates, you can use some library, for example, geopandas. It has the geocoding feature, which converts place names to locations on Earth.

You can also use the Google maps Geocoding API.

There are many more API services which solve this problem, but those two should get you started.

Hope this helps!


What data treatment/transformation should be applied if there are a lot of outliers and features lack normal distribution?

This comments aren't all mine, I have asked on a slack forum,

boxplot is shouting at you:  skewness, and also high dispersion.  Can not ask much more to a boxplot than location, dispersion and skewness..

Also checkout this term heteroscedasticity(completely suits your case)

Try switching the transformation to a log plot or lower..

Also your eda can't be boxplots dependent as prices are involved here..one of the remedy would be doing box-cox transformation

https://www.differencebetween.com/difference-between-dispersion-and-vs-skewness/

http://www.statsmakemecry.com/smmctheblog/confusing-stats-terms-explained-heteroscedasticity-heteroske.html

To have a look at it,
https://datascienceplus.com/how-to-detect-heteroscedasticity-and-rectify-it/


How to convert photo to a vector drawing

This problem is known as "style transfer", and deep learning can achieve some pretty amazing results.

Here's a fantastic GitHub repo with an easy-to-use implementation of neural style transfer (usage instructions).  I think you'll have better luck using a pre-built model like this rather than training your own.

If you want to train your own, this repo might be a good starting point.  It has a training script included, you'll just need a bunch of training images and a style image.


From the formulation of the question, I assume that there are no "examples" of anomalies (i.e. labels) whatsoever. With that assumption, a feasible approach would be to use autoencoders: neural networks that receive as input your data and are trained to output that very same data. The idea is that the training has allowed the net to learn representations of the input data distributions in the form of latent variables.

There is a type of autoencoder called denoising autoencoder, which is trained with corrupted versions of the original data as input and with the uncorrupted original data as output. This delivers a network that can remove noise (i.e. data corruptions) from the inputs.

You may train a denoising autoencoder with the daily data. Then use it on new daily data; this way you have the original daily data and an uncorrupted version of those very same data. You can then compare both to detect significant differences.

The key here is which definition of significant difference you choose. You could compute the euclidean distance and assume that if it surpasses certain arbitrary threshold, you have an anomaly. Another important factor is the kind of corruptions you introduce; they should be as close as possible to reasonable abnormalities.

Another option would be to use Generative Adversarial Networks. The byproduct of the training is a discriminator network that tells apart normal daily data from abnormal data.


Can the Generative Adversarial Network useful for Outlier detection and Outlier explanation in a high dimentional numerical data?

See this answer by Ian Goodfellow (the creator of GANs) to the same question at Quora:

There are definitely some papers about it, such as [1703.05921] Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery. I don’t work on anomaly detection so I haven’t read these papers and don’t know a lot about how they work.
It’s worth mentioning that there are also papers on anomaly detection using inputs that resemble adversarial examples: https://arxiv.org/pdf/1706.02690.pdf
For GANs, one thing to keep in mind is that the discriminator is not a generalized detector of weird things. It is trying to tell whether a sample came from the real data or one specific non-data distribution: the generator. Because of that, it seems like the discriminator would only be useful for anomaly detection if you think you can make your generator resemble the anomalies you expect to need to detect.

Nevertheless, neural networks in general are black boxes (i.e. not very interpretable), so they might detect outliers but you would not be able to tell what characterized them as so.


Working with audio data with different sample rates in Tensorflow

If you have ffmpeg installed, you can just use tf.contrib.ffmpeg.decode_audio. That function takes a samples_per_second argument and does the resampling for you. The following calculates a log magnitude short-time spectrogram of your data.

waveform = tf.squeeze(
    tf.contrib.ffmpeg.decode_audio(
        tf.read_file(tf.placeholder(tf.string, name="filename")),
        file_format=tf.placeholder(tf.string, name="format"),
        samples_per_second=p["sample_rate"],
        channel_count=1))

log_mag_spectrogram = tf.log(tf.abs(tf.contrib.signal.stft(
    waveform, win_length, hop_length,
    n_fft, pad_end=False) + 1e-8))


Changing reference class in imbalanced data drastically affects the error rate

Decision function of GLM itself does not depend on the choice of "reference level". What can probably depend is your threshold, and I guess it is chosen poorly.

For the problem of churn prediction, you probably shouldn't use error rate or confusion matrix at all. You can read here why ROC AUC or other metrics can be preferred to accuracy. Or if you still use error rate, choose your threshold in a wiser way than letting an algorithm maximize F1 metric for a single class.

What you need in the end is to decide for each client whether to treat her as ready-to-churn (it would certainly cost you $c_1$) or leave her alone (but if she churns, you lose $c_2$). If it is the case, you will profit from your anti-churn measures iff the probability of churn is higher than $\frac{c_1}{c_2}$. This is the natural threshold for your classification problem, and you can use the corresponding cost function to measure your success. Or if you don't know exact losses $c_1$ and $c_2$ in advance, use ROC AUC, which averages all the possible thresholds. And yes, ROC AUC is not affected by class balance/imbalance.


In your example what differentiates the clusters is not the raw value but rapid departure from previous points.  I might look into change-point detection. Nonparametic, but still requires some fiddling with tuning parameters.


If you use R the package forecast provides neural networks algorithm as nnetar() function and it has xreg option. You could use your stock data as regressor. Examples and details: http://pkg.robjhyndman.com/forecast/reference/nnetar.html

Or you could use var models because there is relation between stock and sales e.g. if you have low stock your sales would be min(y,stock) where y is the true demand to your products. It also applies in other direction too if you have high y then you will be out of stock fast.

But I'll suggest you to take look forecast package function nnetar() (in R).


If an information is never available for one of the classes, it's not a usable indication. So it seems to me that the login name is simply irrelevant for the task, so it shouldn't be included as a feature.
Mixing different sources of data can be ok in some cases, but only if the different datasets provide consistent features and are generally representative of the distribution you would expect (another important problem). It looks like you have no username nor timestamp in the "bad" one, do you? if so it's impossible to use these features.
No you cannot give priority to any feature. This wouldn't make sense, since the point of ML is to let the algorithm find patterns in the data. If you already know how the decision should be made then you should simply write a regular deterministic program (this is called a heuristic).
Don't forget that you can decide to add any feature that you think could be relevant:

you could have a boolean feature indicating whether a login name is known or unknown, instead of setting the value to 0. But be careful that by definition in your external dataset they are all unknown whereas they are all known in your "good" dataset, that's an clear construction bias.
if you have a timestamp you could think of many other features, for instance number of repeated attempts in the past N seconds, regular time patterns for login, etc.


Your arrays have different shapes on the 0 axis, so you cannot use numpy.stack directly.

You can either use padding or put all arrays in a list. Using padding:

import numpy as np

a0 = np.empty((2,150))
a1 = np.empty((5,150))
a2 = np.empty((10,150))

max_shape = [0,0]
for a in [a0, a1, a2]:
    if max_shape[0] < a.shape[0]:
        max_shape[0] = a.shape[0]
    if max_shape[1] < a.shape[1]:
        max_shape[1] = a.shape[1]
arrays = []
for a in [a0, a1, a2]:
    arrays.append(np.pad(a, pad_width=((0, max_shape[0] - a.shape[0]),
                                       (0, max_shape[1] - a.shape[1])),
                         mode='constant'))
stacked_array = np.stack(arrays)


How to cluster Houses on the basis of similarity of features+location?

Check the ranges of your dimensions and consider scaling if you see a large difference.

I would interpret your described  behaviour due to much larger range if the index that the two other dimensions.
See also the question.


Logistic Regression Multi-level Independent variables

Once a Feature is one-Hot encoded, it is as code as an "N" new feature.
You may discard those which has low Feature-importance


Feature scaling data with missing values

Well -9999 can matter more or less depending on the variability within your data (is -9999 an extreme outlier value, or a moderate value somewhat close to the mean/median>)

Depends a lot on your data.  A value like that can skew it a lot, and can affect modeling quite a bit depending on the technique you use.  Are the NA's biased for a class?  Consider decision trees that classify on optimal splits.  They would detect a bias (e.g. if 90% of your NA's a certain class) and split at -9999 value to manifest that bias.

also check out Pareto scaling, which emphasizes small to medium changes in your data.  Might be along the lines you are looking for.


Start with linear kernel and see if your data is linearly seperable or not. Performing that is simpler than looking for early indications.

Linear kernels are suggested when the number of features is larger than the number of observations in the dataset (otherwise RBF will be a better choice).

However, once you conclude that you have non-linear data, you can try to visualize and map the non-linear separable data into a higher dimensional space and see if it makes the data linearly separable. That is what a kernel does anyway and so you visualizing will give you insights and indications about the type of kernel to be used.

This link might help.


confused about parameter updates and forward/backward pass according to batches and epochs in CNN?

I understand your confusion, and the real cause of all this mismatch between different tutorials is because there is many equivalent ways to train a neural network when dealing with batches, epochs. However, I think it is best to stick with the most common terminology, i.e. that used by the deep learning libraries.

Backpropagation

For question 1, you are correct. We feed the data through the inputs, the data goes through a forward pass and then we obtain an output. With the output we can calculate a loss. Then we will use backpropagation to attribute some fault to each model parameter for the resulting error (loss).  Then we will use gradient descent to update the model parameters accordingly. You can see how this works here.

Epochs and batches

For question 2. First let's define some terms. One epoch ends when all the training data available has been consumed. The second epoch goes through all the data again. In a simple neural network with not much data, you will pass all the training instances through the network successively and get the loss for each output. Then we will get an average of these losses to estimate the total loss for all instances. This results in one backpropagation per epoch.

However, most of the time it is not possible to fit all the data into memory so we must use batches, this means we will only feed-forward some training instances at a time. Then we will calculate the loss resulting from these instances and tune the parameters using backpropagation. Say we have 1000 training instances, then we can use a batch size of 100, we will thus do back-propagation 10 times per epoch.

Pros and cons

The pros of using batches is you can use larger datasets to train your model, however the smaller the batch size the less accurate the loss function estimate.


what are effects of working with categorical dataset

You need to distinguish a few problems here. How well can your data classify some outcome, what features should be used, how to deal with imbalanced classes.

Categorical features are not a problem per se. In the current stage, feature selection might be an issue. Thus, use logit with lasso or ridge to shrink features which are not too helpful (happens automatically). Also dummy/one-hot encoding would be worth a try (jointly with lasso).

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeClassifier.html


How could PageRank be used to rank paragraphs in relevance to keywords?

If I understand, you have a set of paragraphs, and each paragraph is tagged with keywords. This defines a mapping from paragraph to keyword and vice-versa (an index and a reverse index). You can create a graph of paragraphs and keywords, but all of the edges are undirected (or they all have the same direction). Paragraphs only connect with keywords, and keywords only connect with paragraphs.

To establish relationships between paragraphs, you could define a transitive equivalence relation such as "all paragraphs that connect to the same keyword are now connected to each other". This would allow you to rank the paragraphs that contain a given keyword by their number of connections to other paragraphs, but it may not be an appropriate measure of relevance.

For PageRank to be most helpful, I think you need directional relationships (e.g. if "see page 123" was mentioned in a paragraph). To better represent the complexity of your data, creating additional relationships between keywords (e.g. semantic meaning, lexical relationships) and creating additional relationships between paragraphs (e.g. document classification) may help. Natural Language Processing with Python discusses these topics.


Clustering without information about identifier

1) About clustering the products with no product information
Clustering is always possible but not always what you expected
You can cluster the products based on your dataset using the Hamming distance (if values can only be 0 or 1). Two products separated by a small distance are products that are likely to be sold in the same shops. However, there is no assurance that these products will be similar in terms of their characteristics.
Example

expensive ham (A) & cheap ham (B)
expensive cheese (C) & cheap cheese (D)

It is likely that shops located in wealthy neighborhoods (Shop1) will sell expensive ham and expensive cheese. It is also likely that shops in modest neighborhoods (Shop2) will sell more cheap cheese and cheap ham.
  Shop1 Shop2
A   1     0      (Exp. Ham)
B   0     1      (Cheap Ham)
C   1     0      (Exp. Cheese)
D   0     1      (Cheap Cheese)

Based on this :

A and C are very similar (distance = 0)
B and D are very similar (distance = 0)

So the clustering algorithm will (in my example) cluster products, not on their descriptions (ham vs cheese) but based on the prices. (because of shops location in this case)
I have highlighted the price-shop relation, but it is possible that some shops are clothing-only stores and thus products sold in those will likely be clothing.
Conclusion: you will discover clusters of products that are sold in the same stores but it will be difficult to understand why without further information on the shops.
2) About the (Product x Product) distance matrix clustering
The points on the resulting matrix are product duo. AB, BC, DA, etc.
The value of the points is the distance between the products of the duo. So clustering this would result in clusters of duos that have the same similarity. I don't think this makes sense.
Hope this will help.


Machine learning algorithms for time series analysis

It mainly depends on the task you are looking to accomplish with your time series data. Is it a regression or a classification task. In general, Random Forests are known to be easy to use and pretty useful. So I think it's a good way to start. Of course, dimensionality reduction techniques and feature extractors like PCA, kernel PCA, t-SNE and many others are very useful if you want to get better results. Neural networks are also useful, depending on the task you can try both recurrent (LSTM) or convolutional NN.


How to detect phrases from an English sentence.

The name for what you want to do is Named Entity Recognition (NER).  There are several great software packages for this, namely:

Stanford NER
MIT's Information Extractor
spaCy


Spark is intended to be pointed at large distributed data sets, so as you suggest, the most typical use cases will involve connecting to some sort of Cloud system like AWS.

In fact, if the data set you aim to analyze can fit on your local system, you'll usually find that you can analyze it just as simply using pure python.  If you're trying to leverage a series of local VMs, you're going to run out of memory pretty quickly and jobs will either fail or grind to a halt.

With that said, a local instance of spark is very useful for the purpose of development.

One way that I've found that works is if I have a directory in HDFS with many files, I'll pull over a single file, develop locally, then port my spark script to my cloud system for execution.  If you're using AWS, this is really helpful for avoiding big fees while you're developing.


How to obtain and load a good initial data set for object localization?

The ones that are available out of the box -

Open Images Dataset (Object Detection Track)
MS COCO dataset
Indoor Objects Detection
ILSVRC dataset

Create your own

Search for the object of interest on Google Images (like bikes, cars etc.)
Use the Fatkun Chrome browser plugin to download all images at once
Use LabelImg to annotate and generate a custom dataset
Use pre-trained models like SSD MobileNet to re-train your own models

Might not be an exhaustive one, but it will definitely get you started. Cheers!


That sounds like k nearest neighbor clustering because a large portion of the string is similar. This answer to a similar question from 2014 has code you could try in Python.

https://stats.stackexchange.com/a/158090


Based on the information given by you. I'm assuming you have performed multiple linear regression ie multiple features and one response feature to be predicted.

First, apply PCA on all of your features except the response variable you want to predict. In your case the four features you mentioned. Then transform it into a 2 component matrix using PCA. Once you are done with that plot the new Matrix you formed with the response features as a scatter plot.So effectively a 3D scatter plot.

When you generate this scatter plot you will be able to visualize much better on which regression you have to use. You can decide for yourself if it is linear or not. Depending on how many outliers you are comfortable with.


How to split data into 3 parts in Python - training(70%), validation(15%) and test(15%) and each part have similar target rate?

Split to a validation set it's not implemented in sklearn.
But you could do it by tricky way:

1) At first step you split X and y to train and test set.

2) At second step you split your train set from previous step into validation and smaller train set.

X_train, X_test, y_train, y_test
    = train_test_split(X, y, test_size=0.7, random_state=123)

X_train, X_val, y_train, y_val
    = train_test_split(X_train, y_train, test_size=0.5, random_state=123)


Firstly, I hope that the label is either a short summary or words of varying length, not just one word direction. Because moving cars involved in an accident may have multiple directions, or one car could be just parked like the example.
Secondly, given that you are planning to predict varying length label, and given the example text, I am pretty sure that bag of words will not work well. You need the context. And you dont have million sized training set to train a transformer deep neural net. So try to utilize  pre-trained embeddings like USE, GloVe etc. If you are using the embedding, it provides you awesome feature engineering done. Just train some low complexity model like random forest/ xgboost on (label, embeddings). You can also explore pretrained summary generators like bertseq2seq, however the code for this didn't work for me yet.


I would suggest hierarchical clustering. It's unsupervised, and you don't need to predefine number of clusters. How it works (for the bottom up version) is each sentence (or object) is initialized as its own cluster. At each iteration of the algorithm, the two clusters of the smallest intra-cluster distance are joined, all the way until there's a 'root' where everything is one cluster. The result of this is a big dendrogram, and you can cut the dendrogram at whatever point you want to define clusters. Or you can just inspect it. Here's a nice numerical example of the alg in action.

It should detect the same types of clusters you're already finding, and you wouldn't need to redefine a distance metric. And, I anticipate you'll get the same amount of class separation you seem to already be getting; in fact, the nice thing about the dendrogram from hierarchical clustering is that it illustrates class separation nicely. it's 'hclust' in R, not sure for Python.


How can I train a model for localizing objects(classification not required) in Python

The mainstream algorithms used today for finding the bounding boxes of objects in an image includes:

R-CNN (and related Fast R-CNN and Faster R-CNN). This family of the algorithms actual does the localization (finding the bounding boxes) and classification in two separate steps. It finds the bounding box then does the classification. So if you don't need classification, just skip the second step.
YOLO stands for 'You only look once'. The name comes from the fact this algorithm actually does the localisation and classification in one go.

Both algorithms can be implemented in the mainstream python deep learning library like tensor-flow or pytorch etc


Visualization methods in R to examine correlation of labels against response

Below are two functions using my favorite packages:

The first one shows a scatterplot of every column against the target column
The second one shows the correlation of every column with the target column, with confidence intervals (I found how to do that with ggplot here).

Code:
library(ggplot2)
library(reshape2)
library(plyr)

scatterplot <- function(data, targetColumn='Fertility') {
  d<-melt(data,id.vars = targetColumn)
#  ggplot(d, aes_string('value',targetColumn))+geom_point()+facet_grid(variable~.)
  ggplot(d, aes_string('value',targetColumn))+geom_point()+facet_wrap(variable~.)
}

corplotCI <- function(data, targetColumn='Fertility', method='pearson') {
  d<-ldply(colnames(data), function(col) {
    if (col != targetColumn) {
      r <- cor.test(data[,col], data[,targetColumn],method=method)
      data.frame(variable=col,cor=r$estimate, lowerCI=r$conf.int[1],upperCI=r$conf.int[2])
    }
  })
  ggplot(d,aes(cor,variable))+geom_point(size=3)+geom_errorbarh(aes(xmin = lowerCI,xmax = upperCI),height=.5)+coord_cartesian(xlim=c(-1,1))
}

Usage:
scatterplot(swiss)

corplotCI(swiss)


Slightly different results between scipy.stats.spearmanr and manual calculation

I think you have a small error in your manual calculation. You assign rank 4 to 4, 4, and 5. The first two should have rank 3.5 and the last should be rank 5. Your calculation then gives the same answer, 0.7181818181818181


is it possible determine a best neural network classifier by considering only accuracy

There are several issues you should consider before choosing the best classifier.

  Precision, recall and F-score

It is highly recommended that you also check precision, recall and F-score metrics in addition to accuracy. That's because if your dataset have imbalanced labels (for example when 90% of labels are positive and only 10% are negative) considering just accuracy may mislead you. For example in this case if you choose a classifier that always outputs 1, it will have 90% accuracy but it will fail to work well on unseen data. In such cases, precision, recall and F-score comes in handy.

  Validation and test set

In addition to the test set, you should set aside a validation set and tune the hyper-parameters of your model using this set. If you use the test set to tune the hyper-parameters of your model, you can not expect your model to generalize well so it performs poorly on unseen data.

  Overfitting and underfitting

Overfitting happens when your model performs dramatically well on training dataset but poorly on validation and test set. Underfitting happens when your model performs poorly on training dataset. To ensure that you are not underfitting or overfitting, you should check the performance of your model on the training and validation datasets, simultaneously.

So, first you should change your accuracy plots so it shows training and validation loss instead. Then, we consider choosing the best model. If after some epochs, the training score is still low, you are underfitting. If training score is much higher than validation score, you are overfitting. The best case is when training and validation scores are close together after enough epochs.


How do you create an optimized walk list given longitude and latitude coordinates?

As Steve Kallestad has said, this is a TSP problem, and there are wonderful free solvers to find approximate solutions.

It may be too much work for what you are looking for, but you may try use one of those solvers in combination with the Google Maps API, to find real walking distances beetwen your coordinates:
https://developers.google.com/maps/documentation/directions/#DirectionsRequests

(I have never used this API, so I don't know how easy or effective it would be)


Identifying parts of speech for individual words (not texts)

A POS tagged dataset that might help: link


Negative examples for a Yes/No image classification neural network

Have you tried creating a dataset with 2500 images of cars and ~2500 images of random stuff and trained it first?
if so you should try that and see how it works, if it doesnt work you might have to add more images in the "distractor" class.
Provide more information about what you find after you do this experiment and may be we can help further?


What to input into machine learning algorithm for image recogniton?

Usually, neural networks are used to do image classification. The reason is that image data is very complex (highly non-linear) if you want to say so.

Have a look at Keras/Tensorflow. You can even use pre-trained models to quickly achieve good results on most problems. Find an example for Python here: https://github.com/Bixi81/Python-ml/blob/master/keras_pretrained_imagerec_binaryclass.py

There is a very instructive book „Deep learning with Python“, which can give you a quick start. Find the book here: https://github.com/fchollet/deep-learning-with-python-notebooks


It seem that the textbook "Machine Learning - A Probabilistic Perspective" uses input and output in a opposite way, is it?

No, it is not the case. I’m almost sure that it’s a typo and it should be changed to:

“We now consider unsupervised learning, where we are just given input data, without any outputs.”

It can be deduced by looking at the definition of supervised learning from the book:

“In this section, we discuss classification. Here the goal is to learn a mapping from inputs x to outputs y, where y ∈ {1,...,C}, with C being the number of classes.”


4 Class Classification - Machine Learning Model

Yes, binning a continuous variable so that in can take discrete values is reasonable, as long as you are OK with transforming this regression problem into a classification one.

Just note that when dealing with a balanced 4-class classification problem, if an algorithm hadn't learnt anything and was predicting randomly it would achieve a 25% accuracy. Your results indicate that the algorithms are at least learning something. It might just be a very difficult task to solve, in that the features might have little to no correlation with the output variable.

I'd suggest trying out more algorithms with and without GA.


Loss function for classifying when more than one output can be 1 at a time

If your classes arre not mutually exlcusive, then you just have multiple sigmoid outputs (instead of softmax function as seen in example MNIST classifiers). Each output will be a separate probability that the network assigns to membership in that class.

For a matching loss function, in TensorFlow, you could use the built-in tf.nn.sigmoid_cross_entropy_with_logits - note that it works on the logits - the inputs to the sigmoid function - for efficiency. The link explains the maths involved.

You will still want a sigmoid function on the output layer too, for when you read off the predictions, but you apply the loss function above to the input of the sigmoid function. Note this is not a requirement of your problem, you can easily write a loss function that works from the sigmoid outputs, just the TensorFlow built-in has been written differently to get a small speed boost.


Potential speedup by applying PCA once on dataset with m rows vs. IncrementalPCA to x batches of size m/x?

Incremental PCA typically only reduces the memory footprint of the algorithm. So it's used when your full dataset doesn't fit into memory. The time complexity remains the same.

If you want to speed up the computation you can look into randomized matrix decompositions. Sklearn, for example, implements it with the keyword svd_solver='randomized' where you only approximate the singular values you will ultimately keep.


When a sampling unit is drawn from a finite population and is returned
to that population, after its characteristic(s) have been recorded,
before the next unit is drawn, the sampling is said to be “with
replacement”. In the contrary case the sampling is “without
replacement”.

Source: OECD Glossary of Statistical Terms


PCA and K means in the context of machine learning

It is always quite tricky to define Intelligence and Learning since they are very abstract concepts.
in my opinion, both algorithms are machine learning, and I will try to argue why.
Learning is (to me and Wikipedia) the action of improving at performing a task without being programmed explicitly. In our case, both algorithm improve at doing their task the more data we feed into them. And we do not code them explicitly, but instead we give them a goal : minimizing a loss function.
This is quite similar to the way humans learn. When we learn how to read, we get faster and faster and better at imagining what is written the more we read. Yet we do not know exactly how to read faster, we just know the objective function : minimize the time required to understand. And so our brain does the learning from its experiences (which are similar to the datas we feed to ML algorithms).
Another hint that leads to learning to me is the fact that both these algorithms won't give the same results twice if we initialize with random values. To compare to humans again, each person have his way of improving at the task : we all get better, yet each person has its own tricks for the task.
Anyway this is just my opinion on the subject and I do not pretend it does make sense to everyone.
Here is another interesting thread about PCA on Stack if you wanna read some more.
Hope my answer helps you in your philosophical reflection about this complicated topic.


All of the answers here, including the accepted one, are conspicuously confused. I down-voted the accepted answer but downvotes of users who lack reputation in this "community" are not counted. I have a reputation of more than 200,000 (two-hundred-thousand) on math.stackexchange.com and I have a Ph.D. in statistics, but none of that counts here.

The question says $P(B\mid A)$ is the likelihood, but it ought to say instead that $A\mapsto P(B\mid A)$ is the likelihood. I.e. it's this probability as a function of $A,$ not of $B.$

It also says $P(B\mid A)P(A)$ is the conditional probability. That is not right. $P(B\mid A)$ is a conditional probability. $P(B\mid A)P(A)$ is $P(A\ \&\ B).$

Now suppose you have
$$
\Pr(A_1) = \frac 1 {20}, \quad \Pr(A_2) = \frac 2 {20}, \quad \Pr(A_3) = \frac{17}{20}
$$
and $A_1,A_2,A_3$ are mutually exclusive. This is a prior probability distribution.

Further suppose that
$$
\Pr(B\mid A_1) = \frac 9 {10}, \quad \Pr(B\mid A_2)= \frac 2 3, \quad \Pr(B\mid A_3) = \frac 1 2.
$$
Note well:

These three probabilities do not add up to $1.$ What is expressed here is not a probability distribution.
It is a function of $\text{“}A\text{''},$ a variable whose value may be either $A_1,$ $A_2,$ or $A_3.$
The likelihood is $\Pr(B\mid A)$ as a function of $A,$ not of $B.$

Bayes's theorem says: When one multiplies (pointwise) the prior probability distribution by the the likelihood and then normalizes, one gets the posterior probability distribution, i.e. the distribution conditional on the data. The "data" is the observed event $B.$

Thus
\begin{align}
& \left( \tfrac 1 {20}, \tfrac 2 {20}, \tfrac{17}{20} \right) \times\left( \tfrac 9{10}, \tfrac 2 3, \tfrac 1 2 \right) \\[8pt]
= {} & \left( \tfrac 9 {200}, \tfrac 2 {60}, \tfrac{17}{40} \right) \\[8pt]
\propto {} & \left( 27, 20, 255 \right) \\
& \text{(Here I multiplied all three components} \\
& \phantom{(} \text{by 600, which is the l.c.m. of the denominators.)} \\[8pt]
\propto {} & \left( \tfrac{27}{302}, \tfrac{20}{302}, \tfrac{255}{302} \right) \\
& \text{(Here I normalized, i.e. divided by} \\
& \phantom{(} 27+20+255 = 302 \text{ so that the} \\
& \phantom{(} \text{sum of the three components is $1.$)} \\[8pt]
= {} & \left( \Pr(A_1\mid B), \Pr(A_2\mid B), \Pr(A_3\mid B) \right) \\[8pt]
= {} & \textbf{the posterior probability distribution.}
\end{align}

Often one sees something like
$$
X\sim\operatorname{Binomial}(4,p),
$$
so that, for example
$$
\Pr(X=2) = \binom 4 2 p^2 (1-p)^{4-2}.
$$
Then if the data consists of the observation that $X=2,$ then the likelihood function is
$$
L(p) = \binom 4 2 p^2(1-p)^{4-2}.
$$
This is a function of $p,$ not of a variable whose values can be the possible values of $X,$ which in this case are $0,1,2,3,4.$


How to predict the probability of an event?

Please double-check if there's the only data you have got, because all you have is a single predictor date.

If this is indeed your only data source, then you only have a single predictor, and your independent variable is continuous. Now, you should plot date vs amount and fit a single linear regression. Does the fitting look good? Only you can tell because we don't have the full data-set.

If it's not a good fit, look at the plot and ask yourself does this look like a curve? If so, you might want to fit a spline curve or something like that.

You should also check the autocorrelation. This makes sense because your data look like a time series (you'll need to check it yourself). If this is the case, you might want to consider MA and ARCH model.

It's not possible for us to give you accurate advice because we don't know your data.


Dense layers doesn't reduce dimensions of inputs, so if you provide a (12,2,3) input, it expects a (12,2,1) output (for your case).

If you want to make it work, you should flatten your input (or output of first layer) and provide a (12,1) output.


Features from images using opencv in Python

Your numpy arrays are 3-dimensional because they are colored images. The first dimension is the y-dimension, the second dimension is your x-dimension and the third dimension is the different color channels. What order they are in depends on your reader, but I think OpenCV uses BGR (check the documentation here). A lot of the feature extraction methods only work on grayscale arrays which are mapped back to two dimensions, any of the well known image packages can do this for you (OpenCV, scikit-image).


How to reduce the error in linear regression

Answer based on the comment:

  Intention here is to reduce the error by all means and considerations

Choice of model: First let's address the obvious assumption: linear regression is a model which requires the response variable to be expressed as a linear combination of the independent variables. In order to improve performance in general one must make sure that these constraints are satisfied. If not it's often worth looking into other models or finding a way to "linearize" the data.
Data cleaning: depending on the size of the data, linear regression can be very sensitive to outliers. If it makes sense for the problem, outliers can be discarded in order to improve the quality of the model. Of course one shouldn't remove points just because it decreases the error, this has to be done sensibly with respect to the characteristics of the task.
Feature engineering: it's worth analyzing/experimenting which of the independent variables actually help obtaining a good model. For instance redundant variables may decrease performance. It's also possible that variables can be expressed differently or transformed in a way which improves performance.


ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

pos_label is an argument of scikit-learn's precision_score (docs); its purpose is, well, to indicate which label is the positive one and, if not given explicitly (like in your case here), it assumes the default value of 1 (again, check the docs).

Since it seems that the positive label in your case is 'Y', replace the last line with:

print("Precision:",metrics.precision_score(y_test, y_pred, pos_label='Y'))


Scikitlearn grid search random forest using oob as metric?

Check out the make_scorer function in sklearn: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html

You may need to code the OoB component yourself, I haven't ever used that metric for scoring before with Sklearn, so I don't know for sure.

Also, consider using random search rather than grid search, it is more practical when dealing with a large hyper parameter space, since it does not need to search the entire dimension of a parameters that have no impact.


Should highly correlated features be omitted before applying Lasso?

Use scikit-learn package.
In your case you need find sklearn.linear_model.LogisticRegression

and User guide

It's clear enough for understanding. You needn't special actions to win collinearity. But instead of linear method you can use non parametric algorithm like random forest sklearn.ensemble.RandomForestClassifier.

Сompare the results of the logistic regression & random forest on the test data


You can try

Dynamic mode decomposition.
Dynamic Time Warping. Found a nice resource on Towards data science blog.

These two have proven better approaches than PCA for time series clustering.
Happy coding


regarding lasso.score in lasso modeling using scikit-learn

R^2 is a statistical measure of how close the data are to the fitted regression line. It does this by seeing percentage of the variance of dependent varible that's explained by independent variable.
To know more about R^2 score refer this video.
So, basically it is a metric to see how well model fits the data but it is not adequate.
Refer this discussion on why R^2 score is not good for Lasso.