Python sample function TypeError: '<' not supported between instances of 'str' and 'int'

Welcome to the site!

I've checked the issue, it's because you are trying to create a sample using a str variable.(df['A']). df.sample cannot create a sample by assigning weights to categorical variable.

I made a small work around i.e.,

df = pd.DataFrame(dict( A=['a', 'a', 'a', 'b', 'c', 'c', 'c', 'a', 'b', 'b', 'b', 'c', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'], B=range(20) ))

#converting the text into categorical codes which are in numeric
df['cat'] = df.A.astype('category').cat.codes

#creating a sample using newly created column
Output = df.sample(n=100,weights = 'cat',replace=True)

Output would have 100 records and created using 'cat' for weights.


Analytic Hierarchy Process - consistency ratio over 50%

I read some articles and came up to the solution when consistency is very high like 100% I need to reduce it to min 20% by playing with most dismised values.

I recommend online calculator for AHP
https://bpmsg.com/ahp/ahp-calc.php


What are features for state-action pairs in RL?

In the cartpole example, a state-action feature could be

$$\begin{bmatrix}
\text{Cart Position}\\
\text{Cart Velocity}\\
\text{Pole Angle}\\
\text{Pole Tip Velocity}\\
\text{Action}
\end{bmatrix}$$

where Action is either left, right, or do nothing. The reward is not part of the feature vector because reward does not describe the state of the agent; it is not an input. It is a (possibly stochastic) signal received from the environment that the agent is trying to predict/control with the use of feature vectors.


GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds.
From the docs:

class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, return_train_score=False)
cv: int, cross-validation generator or an iterable, optional
Determines the cross-validation splitting strategy. Possible inputs for cv are:
    None, to use the default 5-fold cross validation,

    integer, to specify the number of folds in a (Stratified)KFold,

    CV splitter,

    An iterable yielding (train, test) splits as arrays of indices.

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

cv defaults to 5, so changing it to 2 should provide a significant speedup for you. This will weaken the cross validation significantly.
Alternatively, you may be able to pass in a single test/train split for your value of cv. This would effectively disable cross validation and remove the benefits that it provides.


If you look at the Keras documentation, you will observe that for Sequential model's first layers takes the required input. So for example, your first layer is Dense layer with input dimension as 400. Hence each input should be a numpy array of size 400. You can pass a 2D numpy array with size (x,400). (I assume that x is the number of input examples). Your output has 13 dimensions for each output document and hence your final Dense layer has to be model.add(Dense(13)) instead of model.add(Dense(1)).

Coming to how it knows, it takes the first dimension of X as number of training examples and second dimension as the size of each example. Similarly for output layer.


What is the difference between biclustering and clustering?

Clustering find groups of observations in a p dimensional space (p features).

Biclustering enables you to find clusters in a subset of features.
Then why would you need other features than this subset to perform the clustering?
Because locally you may find dimensions a & b useful to identify a cluster (green cluster in the figure) and somewhere else in the space, dimensions b & c might be useful to identify other clusters (light blue and purple clusters on the figure).

Sources :

https://towardsdatascience.com/subspace-clustering-7b884e8fff73
https://en.wikipedia.org/wiki/Biclustering


Gradient boosting machines can return values outside of the training range. Have a look at this post Can Boosted Trees predict below the minimum value of the training label?
In practice this is unlikely to happen, but it can be the case for your data.
If this is happening probably what it means is that your training data and the one you are evaluating are different.


Identifying if the sentence if it comprise information about education

I think what you are looking is to differentiate between 'be' and 'BE' based on context. Word2Vec is a good place to start but to determine the difference between words based on contexts is 'Sense2Vec.' Word Sense Disambiguation paper is worth to look at. I am not sure if this is what you're looking for because if you have 'be' in every sentence which is most likely, then this won't be an ideal solution. You might have to do some extra data preprocessing like changing 'be' to 'Bachelor of Engineering' based on your sentence.


On design of the training set: conceptual question

One of the basic assumptions governing Machine Learning is that samples from the training set must follow the same underlying distribution as samples from the test set (and so must any other sample you want fed to your model)!

This is why, usually, we randomly partition the same dataset into training and test sets.

This is actually one of the main reasons ML models underperform in some real world applications. You might have trained your model in a specific dataset, but overtime the data slightly changed its characteristics and new data differ from the old ones that were used to train the deployed model. In this case you need to retrain your model on the new data.


Unbalanced discounted reward in reinforcement learning : is it a problem?

Most of the RL problems are sequential decision making processes. That is the action taken at time $t$, can influence the future rewards in a sequential way. From this perspective, the first action lead to a sequence of states/actions which produced positive rewards, which means it deserves more positive feedback than the last action. It only seems natural to prefer the first action in the first state unless we know some other action yields better cumulative rewards. Rewards are gains in short-term, an action might be good in the short term but not in the long run, which is why we use cumulative rewards called returns to estimate the value of an action in a given state.


Smoothing Proportions :: Massive User Database

A simple way would be to consider Laplace Smoothing (http://en.wikipedia.org/wiki/Additive_smoothing ) or something like it.

Basically, instead of calculating your response rate as (Clicks)/(Impressions) you calculate (Clicks + X)/(Impressions + Y), with X and Y chosen, for example, so that X/Y is the global average of clicks/impressions.

When Clicks and Impressions are both high, this smoothed response rate is basically equal to the true response rate (signal dominates the prior). When Clicks and Impressions are both low, the this smoothed response rate will be close to the global average response rate - a good guess when you have little data and don't want to put much weight on it!

The absolute scale of X and Y will determine how many data points you consider "enough data". It's been argued that the right thing to do is set X to 1, and Y appropriately given that.


Training, cross validation and testing accuracy (RMSE and R2) differs when using different shuffles and splits

In terms of evaluation, the best you can do with a very small amount of data is repeating $k$-fold cross-validation many times (i.e. very large $k$), and consider the whole distribution of scores as the performance (in particular take into account the variance across folds).

It's going to be difficult anyway to obtain a reliable measure of performance with such a small dataset. Two options come to mind:

obtain more instances, possibly by using some interpolation method to generate artificial data (but it's not as good as real data).
make the model less complex by reducing the number of features, as this is likely to reduce the variance in the performance.


Since the various hyperparameter are related you cannot be sure that - for instance - a tuned value for min_samples_split from a model where all other hyperparameter are set to default values, will be generally optimal. When you have a situation where you limit the depth of trees (by max_depth assuming you use sklearn), min_samples_split may be a non-binding constraint. It may however be relevant when you set max_depth=None, since as stated in the docs:

If None, then nodes are expanded until all leaves are pure or until
all leaves contain less than min_samples_split samples.

From a practical perspective I would start tuning n_estimators (more tend to be better), max_depth, and max_features (square root of the number of features is a good starting point) with a fixed random_state. Contingent on the reults you may start adding additional constraints.


How do tech-companies employ Random Forest on large data sets?

For the Random Forests algorithm, the time complexity for building a complete un-pruned tree is $O(m.n\log(n))$, where $n$ is the number of records/instances and $m$ is the number of variables. The algorithm is embarrassingly parallel so in many cases companies with available resources will simply use sufficient compute nodes to enable the model to run in a time that they consider reasonable.


Number of batches = total number of samples in dataset/batch size.
If the total samples are not completely divisible by batch size, then it will take the remaining in the next batch.
Eg: 15x66+10=1000 which means it will take 66 batches of size 15 and for the final steps it takes only 10.


Create tensor of a sequence with a defined shape using TensorFlow

As given in tf.constant documentation:

  If value is a list, then the length of the list must be less than or equal to the number of elements implied by the shape argument (if specified).

In your code, the value is the returned array from np.arange(12,153,3) containing 47 integers.

Now, when you specify shape = [2,3], it implies that the length of the list you have provided as value is 2x3 = 6 which is not true because the array generated by np.arange(12,153,3) has the length = 47. In this case, you will get an error saying:

  ValueError: Too many elements provided. Needed at most 6, but received 47

On the other hand, when you specify shape = [8,2,3], it implies that the length of the list you have provided as value is 8x2x3 = 48 which is also not true because the actual length is 47. It does not raise an error because the length of the list can be less than the number implied by the shape argument as it says in the documentation:

  In the case where the list length is less than the number of elements specified by shape, the last element in the list will be used to fill the remaining entries.

So, it is not possible to shape a sequence of 47 elements to [2,3]


Although this answer shows code with Python, the theory is exactly the same.
Two advices when running a linear regression via gradient descent from scratch:

use matrix notation for your features data (i.e. input data) so you can apply it to n-dimensional datasets (not just 1-D as in this case)
standardize your feature matrix data (although in this case you have actually 1-D, but it is a good practice which always works)

Your data plotted with matplotlib:

import numpy as np
from numpy.random import randn, seed

x_data = 20 * randn(1000) + 100
y_data = x_data + (10 * randn(1000) + 50)
n=1000
t=1.4
sigma_R = t*0.001
min_value_t = t-sigma_R
max_value_t = t+sigma_R
y_data = min_value_t + (max_value_t - min_value_t) * np.random.rand(n,1)
x_data=np.array(range(1000))

pyplot.scatter(x_data, y_data)
pyplot.show()

Build your dataset to train the model:

final_df_DICT = {'X': x_data}
import pandas as pd
H = pd.DataFrame(final_df_DICT)
feature_matrix = np.zeros(n*2)
feature_matrix.shape = (n, 2)
feature_matrix[:,0] = 1
feature_matrix[:,1] = H['X']
#standardize features
feature_matrix = (feature_matrix - feature_matrix.mean()) / feature_matrix.std()
target_data = y_data.reshape(len(y_data), )

Look at the feature matrix, having the first column filled with 1's and the second column having the actual x_data:

Model training with your hiperparameters:

w = [0, 0]
L=0.0001
epochs=1000
iteration = 0
cost=[]
while iteration < epochs:
    pred = np.dot(feature_matrix, w)
    residuals = pred-target_data
    #we calculate the gradient for the 2 coeffs with the scalar product
    for i in range(len(w)):
        partial = 2*np.dot(residuals, feature_matrix[:, i])
        w[i] = w[i] - L*partial

    iteration += 1
    computed_cost = np.sum(np.power((pred - target_data), 2)) / n

    cost.append(computed_cost)

print('coef: {}'.format(w))
print('cost: {}'.format(cost[-1]))

Result:

  coef: [-1.80963253e+00 -6.15189807e-06] cost: 6.466287828899486e-07

Let's plot the fitted regression model predictions over the original dataset (we trained on the whole dataset, not taking into account any validation set in this case...):

my_predictions = np.dot(feature_matrix, w)
pyplot.scatter(feature_matrix[:, 1], target_data)
pyplot.scatter(feature_matrix[:, 1], my_predictions, color='r')

pyplot.show()


What is the difference between using numpy array images and using images files in deep learning?

In order to pass an image as an input to a model first need to convert it to a numpy array. Each image actually is represented as an array of values when you load it into python. Even if you don't do it explicitly (i.e. through keras' ImageDataGenerator), it is done behind the scenes.

If your question is: Is it better to use generators than loading the images in a large numpy array?

The answer is: it depends. Is the dataset small enough to fit in your memory?

If not, you are forced to use a generator that loads the images in batches and passes each batch to the model.

If yes, you either can use a generator to save memory for other things (e.g. the model) or you can load the images into a numpy array so that you can save on computation time (i.e. the overhead of loading images again and again).


How to plot categorical data and encode it through colour and shape size?

The easiest solution I found is Altair:

import altair as alt

# assume df contains the DataFrame from the question + a numeric column `Participant`
alt.Chart(df).mark_point().encode(y='Skill', x='Participant', size='Age', color='Gender')

This is what gets rendered out of the box:


Linear regression with white Gaussian noise

η is an error term with distribution (0, sigma). The error or residual is how the true x differ from the estimated function.

Do you mean the theta when you say „parameter vector“.

In this case, the estimated function is:
y_hat = -1 + 0.3x + 0.6x^2.

So if you want to infer the training set, you need to generate x points, evenly scattered around the quadratic function (mean of error = 0) with a variance of 0.1.


How to handle unseen class feature in test dataset

Depends on the model.  I would say most of the time the new level is dropped or mapped to a NA level.  If possible, you should try to do a stratified sample for your training set to ensure that you get every level you care about.


Correlation with missing values. Is least squares an acceptable option?

Yes you "can" use least squares to do that. However, you would probably need some parametric assumptions to do it, which in turn might require some knowledge about the correlation matrix...

To go back to your original problem, I would actually argue against your reference about pairwise deletion. The weird behavior he identifies mostly happens because the estimator is computed with a dataset that is too small. The empirical estimator of Pearson's correlation is NOT guaranteed to be positive-definite. Also, there is some sort of pattern in its missing data: entries for the $X_3$ are missing only when $X_1 \neq X_2$. Let us instead assume that entries are missing purely randomly.

The estimator you compute using $\hat\Sigma$ := cor(X, use="pairwise.complete.obs"), for a data matrix $X$ still makes a lot of sense. The biggest difference with the non-missing data case is that you now get some entries of $\hat\Sigma$ that are computed on less data; but they still estimate the quantity you wish to estimate.

What is less clear is how to use $\hat\Sigma$ in the desired application: the fact that some entries were computed on less data means that some entries might be less reliable than others. However, notice that even in the classical case, some entries are less reliable than others anyway (this depends on the underlying distribution).

To understand the impact of using all available pairwise data to compute $\hat\Sigma$ in the presence of missing data, consider the two following situations:

Situation 1: You have a $n \times d$ data matrix $X$ and compute the empirical correlation with $\hat\Sigma^{(1)}$ := cor(X).

Situation 2: You have a $[nd(d-1)/2] \times d$ data matrix $X$ in which you have $n$ pairwise observations for all $d(d-1)/2$ pairs of variables and NA everywhere else. in this case you compute the empirical correlation with $\hat\Sigma^{(2)}$ := cor(X, use="pairwise.complete.obs").

In both situations, you have the same number of non NA pairs (although more non NA entries in the second case). However, you have, for $i_1,j_1 \neq i_2,j_2$
\begin{align*}
\mathrm{cor}(\hat\Sigma^{(1)}_{i_1,j_1},\hat\Sigma^{(1)}_{i_2,j_2}) \neq 0
\end{align*}
assuming the your variables are indeed related, and
\begin{align*}
\mathrm{cor}(\hat\Sigma^{(2)}_{i_1,j_1},\hat\Sigma^{(2)}_{i_2,j_2}) = 0.
\end{align*}
Somehow, I tend to think the zero correlation between the estimates might be a good thing more that a bad one...

All this to say, pairwise deletion is not necessarily a bad way to proceed.


Library to count number of objects in image?

I think the OP was asking for a specific library for Theano or TensorFlow that provides off-the-shelf functionality for object identification.
I would look first into scikit image, an image processing library for Python in the Scipy/Numpy stack. It has all sorts of edge detectors, blob detectors, shape identifications, color separations... Look at their gallery of examples.
Btw, counting humans in a crowd may not be so trivial to automate...


The cosine distance formula is:

And the formula used by the cosine function of the spatial class of scipy is:

So, the actual cosine similarity metric is: -0.9998.

So, it signifies complete dissimilarity.


Depending on your data, you may be overfitting, however that isn't necessarily the definitive answer.

Gradient boosted trees are a powerful algorithm and for a while performed as state-of-the-art. If your data happen to represent the target value in a systematic way that you haven't uncovered yet, it's likely that with 500 estimating trees the algorithm found a perfect solution. It's not unheard of.

On the other hand, I don't know much about your data. How many samples do you have? 100? 100,000? The former will be much easier to perfectly model. The latter may also be predictable (albeit less likely) if the variance between classes is predictable. The number of features may also play a role, and the significance of each feature.

As suggested in the comments, Cross Validation may help you discover what's going on here. I highly suggest reading the paper I linked above to see an example of rigorous CV. Carefully read what they did to see how you can model your own CV setup.

You might consider checking out the feature importance returned by your classifier. If one feature is significantly important, it might indicate a close correlation between that feature and the target variable (which should indicate that you need to take a close look at that feature).


How to form Hessian matrix in BFGS Quasi-Newton Method

In optimization, you are looking for minima and maxima of a certain function f. Equivalently, under certain derivative conditions, you are looking for the zeros of g, the derivative of f.
The first equation of your link describes one step of the Newton Method - not BFGS. It involves the Hessian of f which is not accessible in practice - it's ineficcient in terms of time and memory storage. Due to this fact, some have developed methods not to calculate the hessian directly.
Thanks to Taylor expansion, you can write

g(x) = g(y) + g'(y)(x-y) + Θ(|x-y|²)

Replacing x by x(k+1) and y by x(k), and g' by the Hessian of f gives you (with the following convention : g(k+1) = g( x(k+1) )

g(k+1) ≈ g(k) + ∇²f(x(k+1)) ( x(k+1) - x(k) )
H(k+1) (x(k+1) - x(k)) ≈ g(k+1) - g(k)
where H(k) aims to approximate ∇²f(x(k)). But instead of calculating it at each step, we'll try to approximate it thanks to the previous H(k-1)

In 1970, Broyden, Fletcher, Goldfard and Shamo proposed a rank 2 update of H(k), ie

H(k+1) = H(k) + Term1 + Term2

where Term1 does not depend on H and Term2 depends on H(k). You can find these terms here.
But this solves only the time issue, not the memory storage. As for that, some have developed Limited-memory BFGS (or L-BFGS) to limit the amount of memory used.


Identifying repeating sequences of data in byte array

I believe the problem that you are referring to, is that of "Motif Discovery in Time Series Data". An appreciable amount of research literature already exists in this domain, so you can look through those. If the data that you handle is not very large, you can find some relatively easy to implement algorithms.

If the data is large , you can look at more recent publications in this domain. As a starting point I would recommend taking a look at how Motif Discovery is done in SAX. SAX takes continuous signals as inputs and discretizes them. These discrete levels are then stored as alphabets. This resulting data looks very much like yours in my opinion. Take a look at what they do in "Mining Motifs in Massive Time Series Databases".


A context window applies to the number of words you will use to determine the context of each word. Like of your statement is "the quick brown fox"  a context window of two would mean your samples are like (the,quick) and ( the, brown). Then you slide one word and your samples become  (quick, the ), (quick, brown) and (quick fox) and so on. I would suggest reading up this word 2vec tutorial to understand the training method and terminology.


Which Technique should we use for predicting an integer output?

If you are using a poisson regression model, this question might be better suited for cross-validated as it is a statistical model.

This appears to be related to the reason why you are getting floats instead of the expected integer value. Poisson regression returns the expected value, E[Y|X] which does not necessarily have to be an integer.

https://stackoverflow.com/questions/49416697/statsmodel-poisson-prediction-return-floats-instead-of-whole-numbers


How can I plot line plots based on an input python dataframe?

Here's a solution:

I've created a sample dataframe with some arbitrary values. Here it is:

import pandas as pd
import numpy as np
from datetime import datetime

test = pd.read_csv('/home/sagar/Desktop/test.csv')
# Convert your date from 'str' to 'datetime' format
test['SUMRY_DT'] = test['SUMRY_DT'].map(lambda x: datetime.strptime(x, '%Y-%m-%d'))
# Set it as your dataframe index
test.set_index('SUMRY_DT', inplace=True)
test
            CMPGN_NM                    COST_SUM
SUMRY_DT
2018-05-17  GSA_SMB_SMB_Generic_BMM     8985
2018-05-18  GSA_SMB_SMB_Generic_BMM     7456
2018-05-19  GSA_SMB_SMB_Generic_BMM     5761
2018-05-20  GSA_SMB_SMB_Generic_BMM     4251
2018-05-21  GSA_SMB_SMB_Generic_BMM     10521
2018-05-22  GSA_SMB_SMB_Generic_BMM     10216
2018-05-23  GSA_SMB_SMB_Generic_BMM     11023
2018-05-24  GSA_SMB_SMB_Spark           11242
2018-05-25  GSA_SMB_SMB_Generic_BMM     8817
2018-05-26  GSA_SMB_SMB_Generic_BMM     6937
2018-05-27  GSA_SMB_SMB_Generic_BMM     4581
2018-05-10  GSA_SMB_SMB_Spark           7089
2018-05-13  GSA_SMB_SMB_Spark           2121
2018-05-11  GSA_SMB_SMB_Spark           234
2018-05-12  GSA_SMB_SMB_Spark           11077

# Plot your data
test.groupby('CMPGN_NM')['COST_SUM'].plot(legend=True)

With the actual data, your chart would resemble the picture you have provided.

Hope this helps.


Can Generative Adversarial Network be run on any embedded / edge device?

1. Mini introduction to Generative Adversarial Networks (GANs)

During the training phase, a GAN model essentially makes two neural networks compete against one another; the generator and the discrimator. As always we start with a training dataset, e.g. of images. Then:

The discrimator network is given images and must deicde whether or not they are real images
The generator learns how to generate new images, with the aim of tricking the discriminator into thinking the image is real

Here is a simple sketch from this article:

Because both networks are fighting over the same metric (the loss of the discrimator network), this idea is also described as a min-max problem. The discriminator is minimising it's own error, while the generator is maximising the same error.

More resources:

To see where GANs all started, have a skim of the seiminal paper by Ian Goodfellow. For a nice video introduction:

A general overview from Numberphile
A more detailed explanation from Ian Goodfellow himself, explaining GAN-related concepts, such as:

mode collapse, where the generator produces copies of the same image
using GAN to generate super-resolution images

2. Inference using GANs

Using the process outlined above, we have trained two neural networks. Performing inference could actually mean using either one of those (or a combination of both, I suppose). In your case, it would seem you only require the generator part:

  I am using DCGAN ( Deep Convolution GAN ) to generate images

This means you would take your trained generator model and use it to generate random samples by providing different random noise states as input (as is shown in the diagram above). Hopefully the learned distribution of your generator during training will be diverse enough to produce different samples. If it ends up always producing the same image, given different random noise samples as input, you are seeing an example of mode collapse (see linked video above).

3. GANs on embedded devices

To end up with a generator that is really of any practical use, you will need to do a lot of training, with many examples. State-of-the-art models use databases of images such as ImageNet, which contains 1 million images over 1000 categories.

Inference:

To perform this kind of training from scratch, I really don't think you will have any luck on an embedded or micro device, such as a Jetson.
To perform inference, however, you might be able to do it. This would simply require loading the pre-trained generator model onto the device (along with the deep learning framework required e.g. Tensorflow). This might work, and I have seen people manage to do this - there were frame rates of around 1 images per second on a conservative convolutional network.

Memory constraints are just a problem as computation itself. On an embedded device, this will likely be a painfully slow process, but could work out if you are able to optimise many parts of the network and training process. You could e.g. compress images to smaller scales, use a final output layer which has fewer neurons, or even go down the path of using less floating point precision in your calculations! See this paper on 8-bit training (as opposed to standard 32-bit). There are also smaller versions of well-known models such as tiny-YOLO.

Training:

Another approach to edge-training (online training) you might take would be to start with transfer learning. You could take a model that was already trained on a task similar to your own, which you will have to search for in open-source projects. This model could then be loaded onto the embedded device and made to work as was done for inference, however we want to train. We do this by keeping the weights from the pre-trained model, freezing the majority of the layers and just allowing the final one or two layers to actually be fine-tuned with your new edge data.


Spliting training data into multiple variables using R

You didn't explained how is your dataset and its variables, so I'll put a generic example.

Let's say your dataset has a variable called SET, whose content is TRN for training and TST for test. To create a dataset containing only training set you have to use the following code:

trn.ds<- ds[ds$SET == 'TRN',]

If you need a test dataset, just adapt the code:

tst.ds<- ds[ds$SET == 'TST']

PS: It wasn't me who downvoted your question, I don't understand why people downvote almost every question here.


If you look at older machine learning algorithms, they rely on the input being a feature and learn a classifier, regressor, etc on top of that. Most of these features were hand crafted, meaning, they were designed by humans. Classical examples of features in computer vision include Harris, SIFT, LBP, etc .

The problem with these is that they were designed by humans based on heuristics. Images can be represented using these features and ML algorithms can be applied on top of that. However, they may not be the most optimal in terms of the objective function, i.e, it may be possible to design better features that can lead to lower objective function values. Instead of hand crafting these image representations, we can learn them. That is known as representation learning. We can have a neural network which takes the image as an input and outputs a vector, which is the feature representation of the image. This is the representation learner. This be followed by another neural network that acts as the classifier, regressor, etc .

So, in the wheel example, you can try to manually describe how a wheel should look like and how it can be represented. Say, it should be circular, be black in colour, have treads, etc. But these are all hand crafted features and may not generalize to all situations. For example, if you look at the wheel from a different angle, it might be oval in shape. Or the lighting may cause it to have lighter and darker patches. These kinds of variations are hard to account for manually. Instead, we can let the representation learning neural network learn them from data by giving it several positive and negative examples of a wheel and training it end to end.

Hope this makes sense. Let me know if I can clarify any point I made.


How to use SMOTE in Stacking in SKLearn?

First question, whether to use SMOTE for the first or second of a stacked classifiers. Generally, SMOTE should be done before any classification since SMOTE gives the minority class an increased likelihood be being successfully learned. The first classifier should be given the most useful features. Another way to approach is looking for empirical evidence. Train models both ways and choose the ordering that performs betters.
Second question, SMOTE is only done on the training dataset. During prediction, only the data that is present is predicted. If you use imblearn's Pipeline, this automatically handled.


Trying to figure out how to set weights for convolutional networks

You are right there are just 10 params in your example.

For determining gradients, you just add up all the deltas from backpropagation in each location - i.e. you run backpropagation 30x30 = 900 times, for each position the 3x3 kernel is used, for every example in your batch (or just for one example if you are running most simple onine stochastic gradient descent), and for each position you add those delta values into a suitably-sized buffer (10 values for weight deltas, or 9 values for previous layer activation deltas). You will end up with one set of summed deltas matching your single 3x3 filter (plus a delta bias term). You then apply the summed version to update the weights of your single filter + bias.

Note this is a general rule you can apply whenever multiple gradient sources from backpropagation can be applied to any parameter - they just add. This occurs in RNNs too, or in any structure where you can set an objective function for non-output neurons.


can a perceptron be used for regression?

The deep feedforward neural networks used for regression are nothing but multilayer perceptron architectures.

Originally, perceptrons were used as binary classifiers i.e to classify binary labels ( 0 or 1 ). But, if no non-linear activation function is applied to the dot product of the features and weights, then it is simply a linear regressor.

If the linear function is $f(x) = x$ and $N$ is the number of features then,

$\Large y = \sum_{i=0}^N x_i w_i$

or using vector notation,

$\Large y = \vec{x}.\vec{w} ...(1)$

For multiple regression, we use the below equation with mean-squared error loss function to optimize the $\beta$ parameters,

$\Large y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 ... \beta_n X_n$

Let ,

$\Large
\vec{W} = [ \beta_1 \ \beta_2 \ \beta_3 \ ... \ \beta_n ] \\
\vec{X} = [ X_1 \ X_2 \ X_3 \ ... \ X_n ]
$

Where $N$ is the number of features, $\vec{X}$ is the feature vector and $\vec{W}$ is the weights vector. Then,

$\Large y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 ... \beta_n X_n = \beta_0 + \vec{W}.\vec{X} ...(2) $

Where, $\beta_0$ is also called as the bias coefficient in Artificial Neural Networks. You can relate equations 1 and 2 and understand the concept.

  Hence, $\Large y = \vec{x}.\vec{w} + bias $ generally represents a hyperplane
  which is used in linear regression.


How Do I Find the Matthews Correlation Coefficient (MCC) in Python?

Taken form the Matthews Correlation Coefficient documentation in sklearn:

  The Matthews correlation coefficient is used in machine learning as a measure of the quality of binary and multiclass classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient.

So it's important to have the actual targets and the prediction values in the range of: -1 and +1. With this in mind you can simply run the following code:

> from sklearn.metrics import matthews_corrcoef
> y_true = [+1, +1, +1, -1]
> y_pred = [+1, -1, +1, +1]
> matthews_corrcoef(y_true, y_pred)

-0.33...

For more details about the formula refer to the Wikipedia page of Matthews Correlation Coefficient


What metric does `Scalar test loss` in AutoKeras mean?

Since you are using StructuredDataRegressor, the default loss function is mean squared error. That information can be found in the __init__ function with the default keyword argument `loss='mean_squared_error'.


Proper/Possible methods for extracting unstructured data from websites

You need to build a couple of classifiers. First, you need a classifier to give you thumbs up that you want to parse the page at all. Call this the "is_relevant" model. Once you've determined that a page is relevant, you should pass it through a separate classifier for each data element you're hoping to capture (or a multiclass classifier capable of recognizing each of those elements and distinguishing them from content you're not interested in).


The Silhouette Coefficient is a good option. The Calinski-Harabaz Index has been shown to be unstable for evaluating noisy datsets, so I would recommend the Silhouette Coefficient over Calinski-Harabaz Index.

The Clustering Validation index based on Nearest Neighbors (CVNN) index has been demonstrated to work well for many different datasets containing high noise, arbitrarily shaped clusters, and skewed distributions.

A few other validation measures you can try using are the SD validity index, the S_Dbw validitiy index, and the Xie-Beni index.

The following is a good manuscript for interal clustering validation metrics manuscript.


Null-invariant measures of association in R

interestMeasure function from arules v1.5-5 let’s you find additional measures and also some invariant ones. Here’s the link to the R documentation: https://www.rdocumentation.org/packages/arules/versions/1.5-5/topics/interestMeasure


How does KNN work if there are duplicates?

Your reasoning is correct - you should consider duplicate points as separate. You can see that this must be the case in several ways:

Introduction of small random noise to the data should not affect the classifier on average. This would not be the case if you removed duplicates.
Suppose that your input space only has two possible values - 1 and 2, and all points "1" belong to the positive class while points "2" - to the negative. If you remove duplicates in the KNN(2) algorithm, you would always end up with both possible input values as the nearest neighbors of any point, and would have to predict a 50% probability for either class, which is certainly not a consistent classification strategy.

The extra question to think about is how to deal with the situation when you have different Y labels assigned to several points with the same X coordinate.

You could mix all classes together and say that the label of each point in the set of duplicates is represented by the distribution of labels in the whole set of points with that coordinate. Alternatively, you could simply sample K random points from the set.

Both strategies should result in a consistent classifier, however in the second case your predictions may not be deterministic. Most practical implementations (including, for example, sklearn.neighbors.KNeighborsClassifier), however, use this simpler, nondeterministic strategy, as it is perhaps slightly more straightforward.


I didn't have enough room in the comments so I'm responding here.

I don't think you'll provide anything useful if you build models to predict any of the columns in your sample.

However, your descriptive analytics should offer a lot of insight, for example, you can say which type of vehicle is more popular for a certain purpose in a certain city and so much more. So while you don't actually predict anything, you're showing the likelihood that something happens for a specific purpose. This itself is useful for business decisions.


In my view, you should better go for user-movie matrix, to create a sparse matrix that can be more meaningful. By doing that, if you already know polarity of choices i.e. positive, negative or neutral, you may also have some intensity score of like or dislike like in case of rating. Now, if you want to just classify them as likes, neutral, and dislike, you may assign 1,0 and -1 respectively. This method can have more meaning when you have many movies and many users. Further this can help you establish user-user, movie-user and movie-movie relationship as well. These relationship can potentially help you understand and explore more about recommendation engines.

     m1 m2 m3 m4
u1 -1   0  -1  1
u2  1   1  -1  0
u3  0   0   1  0
u4  -1  1   1   0

If you want to add rating instead of just polarity, just add numbers scaled 0-1. I personally worked on them to write hybrid recommendation engines which have both collaborative filtering and context based filtering models embedded. I would suggest you to work more on relationships and try to explore more about next steps like possibility of like or dislike bias, association of movie info/ actors-directors influence and similarity in movies etc. I hope this explanation helps.


How much of a background in programming is necessary to become a data scientist?

First of all, the fact that you have known some Java, even ten years ago, already means that you don't "know nothing about programming" (I suggest you update the title of your question to reflect that - change "nothing" to "a little"). I'd like to make several points, which I hope will be useful to you.

In terms of the level of programming proficiency, which is expected (needed) for a data scientist, the following popular definition says it all:

  A data scientist is someone who is better at statistics than any
  software engineer and better at software engineering than any
  statistician.

Another perspective on the role of programming abilities in a data scientist's skill set can be found in a popular visual representation of data science, using Venn diagrams. The original data science Venn diagram was presented by data scientist Drew Conway (see this blog post):

Since its original introduction, the original diagram was modified by various people for various reasons. The two interesting adaptations are for data science in the social sciences domain (http://www.datascienceassn.org/content/fourth-bubble-data-science-venn-diagram-social-sciences), as well as data science Venn diagram V2.0, where data science is represented not as an intersection of knowledge domains, but as their union (http://www.anlytcs.com/2014/01/data-science-venn-diagram-v20.html). Another very interesting and useful visual perspective of data science skill set, also based on Venn diagram, is the following Gartner's diagram, mapping specific skills to business intelligence (BI) or business analytics knowledge domains:

An alternative perspective for a data scientist's skill set and domain knowledge is a taxonomy of data scientists, such as this taxonomy, which classifies data scientists, according to their focus (or the strongest skill set): mathematics, data engineering, machine learning, business, software engineering, visualization, spacial data (GIS) or others.
If you're curious about the meaning of the "Danger Zone" in the original data science Venn diagram, this Quora discussion, containing, among other nice answers, also an answer by the original diagram's author, can be very helpful.
If you're interested in learning about a range of skills and knowledge domains, useful for a data scientist, check this open source curriculum for learning data science: http://datasciencemasters.org, or on GitHub: https://github.com/datasciencemasters/go. Of course, popular and research papers, lectures on YouTube, MOOC courses, online and offline bootcamps as well as a wealth of other resources is only an Internet search away.
Finally, a note on programming languages for data science. I think that it is important to understand that this aspect is really of secondary importance. The focus should be on two words, which the term "data science" consists of: data and science. Focus on data means that it is important to think about data science (or BI, or analytics) tasks in terms of the corresponding domain knowledge as well as to pay attention to data quality and representativeness. Focus on science means adhering to scientific approaches to data collection and analysis, of which reproducibility plays an important role. A programming language for data science is just a tool and, therefore, should be chosen to match the task at hand. Python and R represent very good and the most popular programming languages and environments for a data scientist, however, you should be aware of other options (tool set).


Feature extraction for sentiment analysis

Have a look at these papers-

Semantic Similarity - 1
Semantic Similarity - 2

They came up with a good solution. Have a look maybe you will find something of your use in it. Plus try exploring these too

vaderSentiment 0.5
(VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains.)
Demo- Sentiment Analysis with Python
(This is a demonstration of sentiment analysis using a NLTK 2.0.4 powered text classification process. It can tell you whether it thinks the text you enter below expresses positive sentiment, negative sentiment, or if it's neutral. Using hierarchical classification, neutrality is determined first, and sentiment polarity is determined second, but only if the text is not neutral.)

Hope it helps! :)


Attribute error when loading pickle file in python jupyter notebook

I guess the Config() is a class that you created. It will be needed again when you load a pickle file storing the config. Therefore, you need to import it in the jupyter notebook you use to load your pickle file.

Hope this could be helpful.


Can I forecast with discontinued data using ARIMA?

I ran into a similar problem and this worked for me:
#First I resampled the data to have all the months
train_data = df['sum of received quantity'].resample('MS').mean()
#Then I filled the Nan values
train_data = train_data.fillna(train_data.bfill())


No, they won't have neither the same performance nor the same architecture if you were to try to visualize it. An XGBoost with 100 n_estimators and a learning rate of 0.1 is a 100 trees grown sequentially, and each tree's output is multiplied by 0.1. The latter is a model of 50 trees grown the same way and the outputs are multiplied by 0.2, so your second model ( given the usual size of datasets nowadays ) will probably underfit the data, a low number of trees to learn from the data and a high learning rate which will make it somewhat innaccurate.

Unless you're looking for another kind of similarity, this should explain it enough. I hope. If not, feel free to comment to clarify your question more.


While retraining a pretrained model, getting: ValueError: Input 0 is incompatible with layer flatten_1: expected min_ndim=3, found ndim=2

The reason for this error is that you are trying to flatten an already flat layer. The output of your model is (batch_size, 4), which cannot be flattened further. To simply fix the error remove the flatten layer from your code.

However, when fine-tuning a pretrained model, you should first remove the top layers of that model before adding your own. The reason is that these layers are trained for classification on a task different than yours.

If I were you I'd drop the last two layers of your pretrained model:

# code same as before ...

x = model.layers[-3].output  # Flatten layer output

# don't add flatten again

for fc in fc_layers:
    x = Dense(fc, activation='relu')(x)
    x = Dropout(dropout)(x)

predictions = Dense(num_classes, activation='softmax')(x)

finetune_model = Model(inputs=model.input, outputs=predictions)


Feature selection for data with both continuous and categorical features?

I recommend using the Lasso for feature selection. I think the main advantage is that it is a multivariate method. However, it is more suited for linear models than for tree-based ones.

There are some broad theoretical and practical resources that explain why it is used. I wrote one article comparing it to univariate methods and another one comparing it to feature importance.

You can use the glmnet package in R or use the Lasso() class in scikit-learn.


Python: forecast unevenly spaced time-series?

I reccomend you to use Prophet:

  Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.

In the documentation you can see that is really easy to implement in python.

Also you could convert the problem to a supervised learning one. You can read this blog where they make an introduction of how to face the problem.


Data Science Take Home Challenge Interpretation of Question

Second one but dont stop there.

First of all you specifying if target_metric > x: engaged is just wrong and you have to let data tell you this x, not you choosing it.
Second one seems nice I would start with that, do some extensive analysis and only then start finding x empirically. How? just of the bat you can pose it as minimization problem for different x and find the one that has biggest purity/separation between classes.

Why dont stop there? There is no wright and wrong, if you have some plausible approach in theory but it fails in practice I would definitely
include it and elaborate on it, discuss it etc...


In algorithm 1 of the original GAN article (https://arxiv.org/pdf/1406.2661.pdf), the discriminator is said to be updated by "ascending its stochastic gradient". This is referring to equation 1:
$$
\min_G \max_D V(D, G)=
\mathbb{E}_{x\sim p_{data}(x)}[\log D(x)]
+ \mathbb{E}_{z\sim p_z(z)}[\log(1 - D(G(z)))]
$$

When we want to minimize something, we do grandient descent. When we want to maximize something, we do gradient ascent. In this context, we want to maximize $V(D, G)$ with respect to the discriminator $D$, that is, the $\max_D V(D, G)$ part from equation 1.

I recommend you have a look at NIPS 2016 GAN tutorial video and text. They are very enlightening.


Subsampling the “right” amout of data to train an ML model

Every Machine Learning Problem is different, so there is no standard answer to your question. For the problem you're working on maybe a 70-30 train-test split would result in an optimal model which performs equally well on the test dataset, whereas for another problem may be that ratio just won't do any justice to the model. It's all about experimentation.
Basically, While training your model, you are basically trying to teach the model all relations, dependencies among attributes which can help the model create a clearer decision boundary between data points associated with your response variable.
Less training may not achieve the task as the model may not learn the underlying structure of the data and hence how the attributes are linked to the response variable, too much training data can have adverse affects too as then you may overfit. I would recommend that start from a 50-50 split record the model performance based on the metric you chose and then repeat the exercise for 60-40, 70-30 etc.


Add more columns when you are doing group by in the first parentheses..

First we should understand why it's giving this result..

It's similar to Sql,we are applying an aggregate function on a grouped by value,
That's why it's giving only one value,

If you want to have further information,
Add the columns name in the first parentheses in a $list$

It will be something like this

df.groupby([Col1,Col2...,Coln])(another col).sum().sort_values().tail()

Edit -1

Recently enough, I came across that we can cascade group by together by playing smart and using the .join(second group by(). aggregate ())

Here's the working example (ignore the column names...it's relevant to my dataset)

df.groupby('Year_of_Release')[['Global_Sales']].sum().join( df.groupby('Year_of_Release')[['Name']].count())


Studying social media platforms - key data analysis methods?

The kinds of tools you will use will vary based on the problem you are trying to solve. Social media data is rich and therefore many questions can be asked - and many tools can be used.

However, there is a general pattern you might keep in mind. Typically, you will have to use the platform's API to gather data. You will then have to normalized and store the data in a data warehouse. Finally, you will access and analyze the data with the tools you desire. Keeping the end goal in mind, you will have to strategically choose the best technologies for the job you are doing.

For example, let's say you wanted to study the relationships between users on a social network - a question like "Who are the mutual friends of the most popular individuals in this social network?" In this case, you would gather data using the social media platform's API, normalize it into a CSV, import it into a Neo4j database, and then use Cypher to make queries.


Evaluation of regression models with different evaluations (MSE, variance, VAF etc.)

Variability accounted for (VAF) is not meaningful in regression.
VAF can be driven higher by the distribution of the independent variables which does tell you nothing about how good a regression model is.


Why does cost function on a neural network increase?

It turned out that my activation function is not well chosen. I chose a linear activation function for the output layer where indeed a ReLu should be chosen. The cost function decreases well after I changed the activation function. I guess this solution works case by case though. May it helps.


Representing the architecture of a deep CNN

Will 64 filters be created for each channel (red, green and blue) of
the image?

No, Rather filters also will have 3 channels. It would be (333).
Its work something like this-

Borrowed from- http://datahacker.rs/convolution-rgb-image/
More here-
https://cs231n.github.io/convolutional-networks/.


Look for entity recognition. The entities in your case would be price, number of bedrooms, size etcetera. What I would do is label a number of training examples with:

From character 12 to character 22 indicates the number of bedrooms
From character 39 to character 41 indicates the price

Etcetera.

After this you can train a model to classify entities. Sometimes the number of bedrooms will be in text instead of numbers so then you would need to write something that translates it to numerical values but you could likely just use some dictionary to do this for you.

Entity recognition models are plenty, but I think a character based LSTM approach could work well if your texts are not too long.


perform cluster on a multiple dimensional data in R

Short answer
You can use any number of dimensions for k-means. Therefore, you can use the standard k-means library for R as long as:

your data contains only metric variables and
every column is scaled to the same range.

See this related question.
If your data is not scaled:
Scale it.
If your data contains only categorical variables:
Use k-modes. [R]
If your data contains categorical and metric variables:
Use k-prototypes. [R]

Long answer
How does k-means work?

The algorithm randomly initializes k points in your data. These are your cluster centers.
It assigns every data point to the nearest cluster center. Usually, this is done by calculating the Euclidian distance of the data point to each of the k cluster centers. Now you have k clusters.
It chooses the centroid of each of the k clusters as the new cluster center.
The steps 2 and 3 are repeated until the solution converges or the maximum number of steps is reached.

What this means
The Euclidian distance can be calculated in n-dimensional space. Therefore, you can use any number of dimensions you want.
However, if your data is not scaled, the distance calculations will be different for every feature. Therefore, you need to scale it.
Since Euclidian distance won't make a lot of sense with categorical or with mixed data, you have to use an algorithm that uses Hamming distance instead for these use cases. k-modes uses only Hamming distance. k-prototypes uses Hamming distance for categorical and Euclidian distance for metric variables.


Random forest model in R - predictors and training data types mismatch

Your training set should be true representation of the entire population which is not true in your case. The levels in your train data set's media column has 4 factor levels which is 1 level less than the test data set's media column factor levels. Assuming you are using R, you can fix it with below code

levels(TrainAquifier.media) <- levels(TestAquifier.media)

You can find answer to similar question in stackoverflow here


Why don't we find the analytical function of the cost function?

The power of networks come from hidden layers with non-linear activation functions. Said non-linear activation function make the calculation of an analytical solution impossible (except maybe for some very specific cases that are not really usefull).
If you need more convincing look at logistic regression : a one-neuron network with sigmoid activation which does not have an analytical solution.
As to why there is no closed form solution, you need to understand that the linear regression you gave in exemple is a very simple problem of projecting a 'point' on a 'plane'. Using an activation function on one neuron is equivalent to projecting a point on a complex manifold, which do not yield an analytical solution (except, again for some trivial cases). For neurons in succesions I am not even sure there is a geometrical interpretation.


Creating obligatory combinations of variables for drawing by random forest

You may want to consider gradient boosted trees rather than random forests. They're also an ensemble tree-based method, but since this method doesn't sample dimensions, it won't run in to the problem of not having a useful predictor available to split on at any particular time.

Different implementations of GBDT have different ways of handling missing values, which will make a big difference in your case; I believe R does ternary splits which is likely to work fine.


The concept of a loss function comes from decision theory. Although there are some 'classic' loss functions the point is to be subjective, in the sense of being flexible enough to represent any particular problem conctext. So in that sense, yes, loss functions can be customised.
One of the main ways this has been achieved is via Bayesian regression, as the output of Bayesian regression is a probability distribution which can then the input for a loss function. This blog post shows the mechanics in a simple way (texts for Bayesian statistics and decision are often non-accessible), and also gives some examples of somewhat 'custom' loss functions in 'Examples: Part 3'. -> http://www.statsathome.com/2017/10/12/bayesian-decision-theory-made-ridiculously-simple/


Should I normalise image pixel wise for pretrained VGG16 model

It all depends on how the original pretrained model was trained. If it was trained with normalized data, you should also normalize your data before giving it to the model. Otherwise, the input data distribution will not match what the network was trained with, and then you probably won't obtain good results.
VGG16 pretrained weights expect normalized data as input, so the answer is yes, you should normalize the data (subtracting the mean and dividing by the standard deviation).
If your image domain is similar to ImageNet, you may use the same mean and standard deviation statistics. If your images are from a very different domain, you should compute your own statistics (source).


How to measure the similarity between two images?

Check this handout!

Well, there a few so... lets go:

Given two images $J[x,y]$ and $I[x,y]$ with $(x,y) \in N^{N \times M}$...

A - Used in template matching:

Template Matching is linear and is not invariant to rotation (actually not even robust to it) but it is pretty simple and robust to noise such as the ones in photography taken with low illumination.

You can easily implement these using OpenCV Template Matching. Bellow there are mathematical equations defining some of the similarity measures (adapted for comparing 2 equal sized images) used by cv2.matchTemplate:

1 - Sum Square Difference

$$ S_{sq} = \sum_{(n,m) \in N^{M \times N}}  \bigr(J[n,m] - I[n,m]\bigl)^2$$

This can be normalized as
$$ \frac{S_{sq}}{\sqrt{\sum J[n,m]^2 \times \sum I[n,m]^2}} $$

2 - Cross-Correlation

$$ C_{crr} = \sum_{(n,m) \in N^{M \times N}}  \bigr(J[n,m] \times I[n,m]\bigl)^2$$

This can be normalized as
$$ \frac{C_{crr}}{\sqrt{\sum J[n,m]^2 \times \sum I[n,m]^2}} $$

B - Image descriptors/feature detectors:

Many descriptors were developed for images, their main use is to register images/objects and search for them in other scenes. But, still they offer a lot of information about the image and were used in pupil detection (A joint cascaded framework for simultaneous eye detection and eye state estimation) and even seem it used for lip reading (can't direct you to it since I am not sure it was already published)

They detect points that can be considered as features in images (relevant points) the local texture of these points or even their geometrical position to each other can be used as features.

You can learn more about it in Stanford's Image Processing Classes (check handouts for classes 12,13 and 14, if you want to keep research on Computer vision I recomend you check the whole course and maybe Rich Radke classes on Digital Image Processing and Computer Vision for Visual Effects, there is a lot of information there that can be useful for this hard working computer vision style you're trying to take)

1 - SIFT and SURF:

These are Scale Invariant methods, SURF is a speed-up and open version of SIFT, SIFT is proprietary.

2 - BRIEF, BRISK and FAST:

These are binary descriptors and are really fast (mainly on processors with a pop_count instruction) and can be used in a similar way to SIFT and SURF. Also, I've used BRIEF features as substitutes on template matching for Facial Landmark Detection with high gain on speed and no loss on accuracy for both the IPD and the KIPD classifiers, although I didn't publish any of it yet (and this is just an incremental observation on the future articles so I don't think there is harm in sharing).

3 - Histogram of Oriented Gradients (HoG):

This is rotation invariant and is used for face detection...

C - Convolutional Neural Networks:

I know you don't want to used NN's but I think it is fair to point they are REALLY POWERFULL, training a CNN with Triplet Loss can be really nice for learning a representative feature space for clustering (and classification).

Check Wesley's GitHub for a example of it's power in facial recognition using Triplet Loss to get features and then SVM to classify.

Also, if your problem with Deep Learning is computational cost, you can easily find pre-trained layers with cats and dogs around.

D - Check on previous work:

This cats and dogs fight has been going on for a long time... you can check solutions on Kaggle Competitions (Forum and Kernels), there were 2 on cats and dogs This One and That One

E - Famous Measures:

SSIM Structural similarity Index
L2 Norm (Or Euclidean Distance)
Mahalanobis Distance

F - Check on other kind of features

Cats and dogs can be a easy to identify by their ears and nose... size too but I had cats as big as dogs... so not really that safe to use size.
But you can try segmenting the images into animals and background and then try to do region property analisys...
Also, check on this image similarity metrics toolkit page it is in C but...
Check this paper on image similarity
Take a look on this Stack Overflow question and this Research Gate one
If you have the time, this book here: Feature Extraction & Image Processing for Computer Vision from Mark S. Nixon have much information on this kind of procedure
You can try Fisher Discriminant Analysis and PCA to create a mapping and the evaluate with Mahalanobis Distance or L2 Norm


Suspiciously good accuracy using neural network

As you can see, there is nothing significantly separable, which really confuses me. I am definitely using the test set to run the cross validation which is 40% of the sample data.

Well, your interpretation is wrong: there is nothing  linearly separable on the two first directions.
The other components in PCA may be more relevant. Remember that PCA gives you the direction where the data has most variance, which doesn't necessarily mean it is the direction in which it is most easily separated/classified.
I see you tryed LDA, which follows on that assumption that the most variable direction is not the most significant for classification, but since it is linear we can conclude that your data is not Linearly separable.
Your model seem pretty large, it is considerably more powerful than LDA, LR, etc.
If you are not confident with the results, try using other validation methods such as K-FOLD or Leave-one-out.


How many normalization methods are there and what are they for?

StandardScaler

RobustScaler for outliers-heavy dataset(can handle better outliers than other scalers)

to name a few..


Practical limitations of machine learning

How does one even start describing these practical limitations?

Most authoritative source on the main practical (and theoretical) problems related to machine learning is formal framework describing it. Mathematics.

Right here we have a problem, even tough you look at all of these ML algorithms and you conclude its math, we do not explain the whole process of a ML problem. I cant find a citation but Bengio or someone said that computer science is hard science because you have mathematical underpinning for everything, and ML is more of a soft science where you learn by trying, but only after you tried. (Not always the case ofcourse, thats why everyone is researching ML now to give it some structure)

Take a simple neural network. You know its matrix multiplication, backpropagation bla bla. Great. But what are the topological properties of certain architecture, convergence criteria, what functions can it approximate. Some of these questions are known and/or are being researched. Lets look at this one for example: What functions can we guarantee (in a formal sense) that a NN can approximate and under what conditions?

Well we can express a task as an optimization one. And in order to converge to optimal solution, under certain constraints, we need to satisfy certain assumptions.

Regarding DNN (deep neural networks) and mathematical theory behind it , convergence assurance is given with famous Universal Approximation theorem that states that every smooth function can be estimated given enough parameters.

Caveat just because we can do it in theory does not mean its possible. For example approximating a function that generates random numbers would require infinite recources

But what about non-smooth functions (such as Time-series) one?

Well the TL;DR of the DNNS FOR NON-SMOOTH FUNCTIONS is that for a special set of piecewise smooth functions "convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions"

What is piecewise smooth function? function whose domain can be partitioned locally into finitely many "pieces" relative on which smoothness holds, and continuity holds across the joins of the pieces.

Ok but WHY can a DNN approximate these types of functions?

" The most notable fact is that DNNs can approximate non-smooth functions with a small number of parameters, due to activation functions and multi-layer structures. A combination of two ReLU functions can approximate step functions, and a composition of the step functions in a combination of other parts of the network can easily express smooth functions restricted to pieces. In contrast, even though the other methods have the universal approximation property, they require a larger number of parameters to approximate non-smooth structures"

Conclusion there is a mathematical theory that insures approximations of a set of certain non-smooth functions using DNN. So if we have non-smooth function that satisfies these constraints, we can find an optimal architecture and get optimal convergance rates.

Concluding your question There are best practices that constantly evolve and you can get a check-list that just isnt relevant (Take computer vision problems, check list 2 years ago isnt same as today). BUT what is constant and remains the best authority is formal under pinnings such as mathematics. It can tell you directly the "best practice" of when it would be futile to even try to approximate a function.


Q-learning why do we subtract the Q(s, a) term during update?

The wikipedia formulation does indeed show you a better view of how the update rule for action values is constructed:

$$Q(s_t, a_t) \leftarrow (1-\alpha)\cdot Q(s_t, a_t) + \alpha\left[ r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a')) \right] $$

. . . here you can see that you are taking a weighted average between $Q(s_t, a_t)$ and $ r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a')$. An aside: In both cases you have written $\text{argmax}$, where the actual term is $\text{max}$.

The intuitive reason for the update is because the second term contains a new estimate of the true value of Q from the environment - the immediate reward $r_t$, and the actual state transition that occurred $s_{t+1}$ are observed data that are part of the new estimate. The term  $r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a'))$ is usually called the Temporal Difference Target or just TD Target.

  In the original representation (the first equation), it magically seems that gamma would be enough - can someone clear it up for me?

The $\gamma$ term is the discount rate, and nothing to do with convergence of Q-learning or SARSA. It is a term used to control preference for more immediate rewards (low values) or longer-term rewards (high values), and as such is part of the problem definition. It is not a learning parameter like $\alpha$, the learning rate. And in fact $\gamma = 0$ is possible (rarely used, but meaning that only next reward is important); $\gamma = 1$ is also possible and quite a common choice for episodic problems. Clearly when using $\gamma = 1$, there is no "decay of future rewards" happening.

What is actually happening is that the TD Target is a stochastic estimate for the correct Q action value (in fact with all TD learning such as SARSA and Q-learning, this is a biased estimate, although the bias should reduce as the system approaches convergence). If you rename $Q(s,a)$ as $Q_{so\_far}$ and $r_t + \gamma \cdot \text{max}_{a'}(Q(s_{t+1}, a'))$ as $Q_{latest\_estimate}$, you get the idea of the update:

$$Q_{updated} \leftarrow (1-\alpha)Q_{so\_far} + \alpha Q_{latest\_estimate}$$

which is the same as

$$Q_{updated} \leftarrow Q_{so\_far} + \alpha(Q_{latest\_estimate} - Q_{so\_far})$$

You might ask "Why is the TD Target an estimate of $Q(s_t, a_t)$?"

The answer can be derived from the definition of Q, and is called the Bellman equation - in this case the Bellman equation for action value under the optimal policy $\pi^*$ (anther aside: what you have called the Bellman equation is not, although it is related). I won't give the full derivation here, but in essence the definition of Q is "the expected total reward when starting with state $s_t$ and action $a_t$ then following the optimal policy form then on", or

$$q_{*}(s, a) = \mathbb{E}_{\pi^*}[\sum_{k=0}^{\infty}\gamma^k R_{t+k+1} | S_t = s, A_t = a]$$

and from this definition, you can extract the first reward term and write out $q_*(s_t, a_t)$ in terms of $q_*(s_{t+1}, a_{t+1})$.

$$q_{*}(s, a) = \sum_{r,s'} p(r,s'|s,a)(r + \gamma \text{max}_{a'} q_{*}(s', a'))$$

Where $p(r,s'|s,a)$ is the transition probability of getting reward $r$ and ending up in state $s'$ when starting in state $s$ and taking action $a$.

Note this is an equality, not an update rule. Q-learning and SARSA turn it into an update rule by sampling the right hand side, and using that sample to update the current best estimate of the left-hand term. Note that by taking a sample, you don't need to know $p(r,s'|s,a)$ . . . that is what is meant by "model free" learning, as if you knew $p(r,s'|s,a)$ then you would have a model of the underlying Markov Decision Process (MDP).


From regression neural network to generative one

There's more than one type of generative network. However, I am not aware of a generic approach that can take a trained RNN-based network and essentially run it backwards to sample an input that is expected to produce a given output. So I am suggesting a couple of generative approaches that I have seen working, but that will require that you construct and train a new network.

You can bring in some knowledge about the typical size of network that learns the regression model, but you cannot AFAIK directly re-use the regression model and somehow reverse it.

A caveat: Although I have played briefly with both types of generative network, I have never constructed one conditioned on desired goal like the one you want to work with.

With a purely RNN-based approach, you might do OK by making your network predict the next letter/word in the title - a classifier - whilst taking the title so far (or X characters/words of it) and the rating (normalised) as inputs.

Then, once trained, you can sample from the RNN randomly to generate new strings. This is the technique used by e.g. Karpathy in his now famous blog "The Unreasonable Effectiveness of Recurrent Neural Networks". There are many examples of such sequence sampling generators available to study.

You could also take the output of such a model and see if it matches your regression model from earlier. But I don't think there is much you can do if it does not, except perhaps filter out generated titles if they don't meet expectations - e.g. generate many titles with intended rating of 10, and only display one when your initial model also agrees with a close to 10 rating.

The most relevant Keras example for this I could find is lstm_text_generation.py

A RNN-based generative adversarial network (GAN) might also be able to achieve what you want. However, please note that GANs are notoriously fiddly to train.

A GAN is actually 2 networks. You create a discriminator and a generator and train them in parallel. The generator takes a small completely random vector (e.g. 10 numbers sampled from N(0,1)), plus the rating you want to achieve. Then it generates a text sequence output. The discriminator would take a text input, and a rating, and outputs 1 if it is real, or 0 if it is fake. You present either real training data or output form the generator to the discriminator, and use these to train it. You train the generator based on whether it fools the discriminator.

The tricky part is to maintain balance between the two components - if either becomes too good relative to the other, training will stall.

However, if you can get it to work, you will have a true generative model, which samples from a population space (the noise vector that you input) plus is conditioned on the rating.

The most relevant Keras example for this I could find is mnist_acgan.py which generates an image, not text sequence, but hopefully should give a start.


shapes recognition for shapes formed by dotted points

Use MINST dataset trained model, make predictions.

Look here. find the one with biggest accuracy, and the circle should be class that
represents class of number 0.

EDIT: check this notebook where you can download your own images from google, annotate them andperform transfer learning relatively fast. Here
So you find all geometrical shape that you want and update some bigge model, or train your own from scratch


What is the ideal database that allows fast cosine distance?

If it's only a few thousand entries each with a 1,000 features, you may just be able to keep it in RAM if you are running this on some kind of server. Then when you get a new feature vector, just run cosine similarity routine. An easy way to do this is just use something standard like pandas and scikit-learn.

Alternatively you can keep everything in SQL, load it into something like pandas and use scikit-learn.

I'm actually not sure you'll get much of a speed up, if any, by writing the computation in SQL itself.


Is FPGrowth still considered "state of the art" in frequent pattern mining?

State of the art as in: used in practise or worked on in theory?

APRIORI is used everywhere, except in developing new frequent itemset algorithms. It's easy to implement, and easy to reuse in very different domains. You'll find hundreds of APRIORI implementations of varying quality. And it's easy to get APRIORI wrong, actually.

FPgrowth is much harder to implement, but also much more interesting. So from an academic point of view, everybody tries to improve FPgrowth - getting work based on APRIORI accepted will be very hard by now.

If you have a good implementation, every algorithm has it's good and it's bad situations in my opinion. A good APRIORI implementation will only need to scan the database k times to find all frequent itemsets of length k. In particular if your data fits into main memory this is cheap. What can kill APRIORI is too many frequent 2-itemsets (in particular when you don't use a Trie and similar acceleration techniques etc.). It works best on large data with a low number of frequent itemsets.

Eclat works on columns; but it needs to read each column much more often. There is some work on diffsets to reduce this work. If your data does not fit into main memory, Eclat suffers probably more than Apriori. By going depth first, it will also be able to return a first interesting result much earlier than Apriori, and you can use these results to adjust parameters; so you need less iterations to find good parameters. But by design, it cannot exploit pruning as neatly as Apriori did.

FPGrowth compresses the data set into the tree. This works best when you have lots of duplicate records. You could probably reap of quite some gains for Apriori and Eclat too if you can presort your data and merge duplicates into weighted vectors. FPGrowth does this at an extreme level. The drawback is that the implementation is much harder; and once this tree does not fit into memory anymore it gets a mess to implement.

As for performance results and benchmarks - don't trust them. There are so many things to implement incorrectly. Try 10 different implementations, and you get 10 very different performance results. In particular for APRIORI, I have the impression that most implementations are broken in the sense of missing some of the main contributions of APRIORI... and of those that have these parts right, the quality of optimizations varies a lot.

There are actually even papers on how to implement these algorithms efficiently:

  Efficient Implementations of Apriori and Eclat. Christian BorgeltWorkshop of Frequent Item Set Mining Implementations (FIMI 2003, Melbourne, FL, USA).

You may also want to read these surveys on this domain:

  Goethals, Bart. "Survey on frequent pattern mining." Univ. of Helsinki (2003).

  Ferenc Bodon, A Survey on Frequent Itemset Mining, Technical Report, Budapest University of Technology and Economic, 2006,

  Frequent Item Set MiningChristian BorgeltWiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 2(6):437-456. 2012


I have limited samples for one class, unlimited samples for the other class. Need to balance?

Yes, train with all the data.

To adjust for the imbalanced data size, you want your minority class X to be more important than the majority class. You can either (1) use weighted cross-entropy loss, e.g. tf.nn.weighted_cross_entropy_with_logits, or (2) when adding up the loss from different traning examples, weight them inversely proportional to the size of that label, or use both.

This is a good explanation here.


I have used S3, S4 and R6 (you forgot that one in your overview ;)). I would agree with Hadley Wickham that S3 is sufficient for most tasks. However, this is only required if you start building advanced functions that operate on objects. Say for example you build a model with one function and you want to create a summary and print function for the object returned by your model building function. For general Data Science purposes I would say that it helps to know the systems but none of them are very good examples of real OOP. For that I would recommend to work in Python, Ruby or Java. All have been build with OOP in mind.

In regards to knowing OOP, I think it is vital for someone involved in ML. Not when you are prototyping in R or Python but definitely when you start working on production code. I think this Quora thread gives a good run down on when OOP becomes important in ML.

If your focus is more on statistics in R it may be of less importance.


Best regression model to use for sales prediction

This is a pretty classic ARIMA dataset. ARIMA is implemented in the StatsModels package for Python, the documentation for which is available here.

An ARIMA model with seasonal adjustment may be the simplest reasonably successful forecast for a complex time series such as sales forecasting. It may (probably will) be that you need to combine the method with an additional model layer to detect additional fluctuation beyond the auto-regressive function of your sales trend.

Unfortunately, simple linear regression models tend to fare quite poorly on time series data.


Normalize the output of a dense layer with linear activation

Your way seems to be correct. I would suggest one other way that you might want to try (maybe it won't make your life better, but still): Since you want your data points to basically live on a sphere, you could train the angles in spherical coordinates. Fix the radius to $\sqrt{n}$ and this way your network has to learn one dimension fewer. I'm not sure how this will affect performance though.
One way to implement this in tensorflow could be
def spherical(input):
    layer = []

    for i in range(input.get_shape()[1]+1):
        elem = 1
        for j in range(i):
            elem = elem*tf.sin(input[:,j])
        if i < input.get_shape()[1]:
            elem = elem*tf.cos(input[:,i])
        layer.append(elem)

    return tf.transpose(tf.convert_to_tensor(layer))

This will map any input (a tensor with shape=(ndatapoints, ndims)) to the unit sphere in ndims+1. By multiplication and shifting you could map it to any sphere you want. It is probably not the most elegant way, but it gets the job done. The mapping is not injective, if you would need it to be, you would have to make sure the elements of input stick to the respective intervals.
If you end up trying this, let me know how it went please ;)


Which model may be best for outcome of a surgery?

Question 1: I would propose Decision Trees as a first thing you should look into, as they are easy to implement and the results are straightforward to interpret.
You will end up with a tree-structure, where the nodes hold intervals/values of your variables. At each node, the tree tries to split your data most efficiently into your binary classifications $C_1$ and $C_2$. Thus, in the end, you will be able to extract most important features in your variables at the top of the tree. A typical measure of "efficiency" when selecting attributes is Entropy.

Question 2: I think it is also solved with the above method. For each node of the tree you know the amount of $C_1$ and $C_2$ classifications on the left and on the right. Assume your attribute in the node is "Has diabetes". Your data (and thus the tree) tells you, 100 out of 110 patients "failed" the operation if the person had diabetes. Thus, you can give an estimate about how certain attributed contribute to success of your operations, and conclude it would not make much sense to operate the given person.

Nonetheless, always be careful about how you interpret the results. You should not try to interpret attributes that split small subgroups, since this can easily lead to false assumptions about real-world behaviors.


Vector representation of documents for text classification

Not necessarily. Documents embeddings are constructions from (sub)word embeddings. Nowadays, you should always start from a pre-trained model. Please check tensorflow Hub or Hugging Face. Even without retraining, as long as your text is not too specific, the pre-trained encoder should capture enough relations.


My layman's understanding is that binary classification is usually calculated using the logit transform.

I believe then that the residuals are the difference between the response and the predicted probabilities and the default metric for this application is the Area-Under-Curve (AUC).


Classification Based Collaborative Filtering Model

Actually, your objective here is not clear.
Following your example, I belive you want to obtain likeness/relationship between users and places.
So basically what you would want to do is to create a domain of users and places. Now for different addresses (eg. A and C), you can treat the same user (eg. One) as different users i.e. "One-A" and "One-C". You can now assign a binary value for each "user-address" v/s "place". Then you can use any collaborative filtering method to get the affinity of the users corresponding to a particular address, for a particular ordering place.


R: Comparing dissimilarity between metabolic models with discrete wavelet transformation

Take a look at the TSclust package. Here how you would apply it to your sample data.

require(TSclust)

#read in the data
model_a <- read.csv("~/Desktop/Model A.csv", header = TRUE, stringsAsFactors = FALSE)
model_b <- read.csv("~/Desktop/Model B.csv", header = TRUE, stringsAsFactors = FALSE)

#data must be in rows rather than columns
model_a <- as.data.frame(t(model_a))

model_b <- as.data.frame(t(model_b))

#calculate dissimlarities between metabolites in models 1 and 2
met1_DWT.diss <- as.numeric(diss.DWT(rbind(model_a['Met1', ], model_b['Met1', ])))
met1_DWT.diss
[1] 90.80332

met2_DWT.diss <- as.numeric(diss.DWT(rbind(model_a['Met2', ], model_b['Met2', ])))
met2_DWT.diss
[1] 1.499241


Google's election results data visualization

Possibly because google was predicting based on changes from previous elections, possibly combined with opinion polls, on a precinct-basis. If all the pro-Trump precincts report in first, you'd still see a blue box because of the very low probability of historically Democrat precincts voting Trump.

"Calling" something in a US election is not an official result, its a confident prediction by an agency. Sometimes these are wrong. Search for "Dewey Defeats Truman" for a classic example.


It depends.

The problem you are trying to solve decides which among the curves is the best

For example
If you are trying to solve a problem that is like identifying cancer where the cost of false positive is extremely high. The first blue plot (model 3) is the best. However, you are compromising on recall or number of positive examples captured. It means that not all patients who have cancer will be identified as having cancer by model 3 but who ever the model says has cancer is most likely to have it amongst all the models plotted.

If your objective is just a simple marketing campaign where you would want most responders to be targetted but you are fine contacting people who are not likely to respond, model 2 which has the maximum recall.

When targetting a non responder is also expensive like a direct mail campaign, model 4 is the best because it balances both precision and recall.

Having said this response rate and sample size also need to be taken into account before arriving at which model is good.
Because if accuracy of 94% and 87% are because of just 2 accurate samples (very low response) accuracy alone might not make sense.


I want to run several Association rule mining techniques such as
  Apriori, Eclat and FP Growth. [...] Please recommend me a suitable
  tool/approach to perform this

Christian Borgelt's website offers several tools for frequent pattern mining:

FPgrowth,
Eclat,
Apriori, and some others.

Check out the full descriptions to see how to use each one.

Here are some requirements for Weka's FPgrowth algorithm.


Is over fitting okay if test accuracy is high enough?

If you properly isolate your test set such that it doesn't affect training, you should only look at the test set accuracy. Here are some of my remarks:

Having your model being really good on the train set is not a bad thing in itself. On the contrary, if the test accuracy is identical, you want to pick the model with the better train accuracy.
You want to look at the test accuracy. That is your primary concern. So pick the model that provides the best performance on the test set.

Overfitting is not when your train accuracy is really high (or even 100%). It is when your train accuracy is high and your test accuracy is low.
it is not abnormal that your train accuracy is higher than your test accuracy. After all, your model has an advantage with the train set since it's been given the correct answer already.

At the end of the day, training a machine learning model is like studying for a test. You (the model) use learning resources such as books, past exams, flash cards etc. (train set) to perform well on a test/exam (test set). Knowing your learning resources perfectly doesn't mean you are overfitting. You would be overfitting if this is all you knew and couldn't perform well on the exam at all.


How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings

You should at least try pre-trained embedding vectors. TfidVectorizer is particularly sensitive to out-of-vocabulary words, which are likely to appear if you're trying transfer learning to a new domain. The GloVe embeddings [1] have a dictionary of 400k vectors, unless you're working with documents from a technical domain they should provide some improvement. Also, consider to perform some extra feature engineering, words by themselves do not provide much information. Why not trying adding n-grams & dependencies extraction, entity recognition, etc. as preprocessing steps?

[1] http://nlp.stanford.edu/projects/glove/


Deriving Prediction Intervals for Orthogonal Distance Regression using `scipy.odr`

[I am working on researching a complete answer, this is as far as I've got so far, I will edit as I work out more of the solution.]
Further research has turned up this paper which provides at least part of the answer.
Briefly, ordinary least squares (OLS) does not consider uncertainties on either axis, and weighted least squares (WLS) only takes into account uncertainties in the predictor variable / dependent variable / on the y-axis. The above publication advocates the use of bivariate least squares (BLS) over orthogonal distance regression (ODR), but goes on to derive formulae for prediction intervals which appear to be compatible with ODR.
In this approach, each 'data point' $\left(x_{i}, y_{i}\right)$ is considered to be the result of multiple experiments or observations under ostensibly the same conditions, thus allowing the calculation of the variance for both the predictor and response variable pair.
The predictor ($x_{i}$) and response ($y_{i}$) variables are related by the following equation
$y_{i}=b_{0}+b_{1} x_{i}+e_{i}$
where $b_{0}$ and $b_{1}$ are the estimates of the intercept and slope of the true linear model, and $e_{i}$ is the $i$th residual error. The variance of $e_{i}$ is referred to as the weighting factor and is denoted as $w_{i}$ or $s_{e_{i}}^{2}$:
$w_{i}=s_{e_{i}}^{2}=s_{y_{i}}^{2}+b_{1}^{2} s_{x_{i}}^{2}-2 b_{1} \operatorname{cov}\left(x_{i}, y_{i}\right)$
where $s_{x_{i}}^{2}$ and $s_{y_{i}}^{2}$ are the experimental variances of point $i$, and $\operatorname{cov}\left(x_{i}, y_{i}\right)$ is the covariance between the measurements for each $\left(x_{i}, y_{i}\right)$ data pair.
The variance of the response variable $y_{0}$, being the mean of the $q$ observations performed at $x_{0}$, is given by:
$s_{y_{0}}^{2}=\left(\frac{1}{q}+ X _{0}^{ T }\left( X ^{ T } W ^{-1} X \right)^{-1} X _{0}+s_{x_{0}}^{2} b_{1}^{2}\right) s^{2}$
where $s^{2}=\frac{\sum_{i=1}^{n}\frac{\left(y_{i}-\hat{y}_{i}\right)^{2}}{w_{i}}}{n-2}$ is the estimate of the true experimental variance, $X_{0}$ is a two element column vector $\left|\begin{array}{l}
1 \\
x_{0}
\end{array}\right|$, and $X$ is an $n \times 2$ matrix for which the first column is a column of ones and the second column is formed by the $n$ values of $x$ corresponding to the experimental points; $W$ is an $n \times n$ diagonal matrix whose $i$th diagonal element is the weighting factor $w_{i}$, defined above.
The variance of the predictor mean value at a given observation $y_{0}$ is
$s_{x_{0}}^{2}=\left( Y _{0}^{ T }\left( Y ^{ T } W ^{\prime -1} Y \right)^{-1} Y _{0}+s_{y_{0}}^{2} \frac{1}{b_{1}^{2}}\right) s^{\prime 2}$
where $Y _{0}$ is $\left|\begin{array}{l}
1 \\
y_{0}
\end{array}\right|$, $Y$ is an $n \times 2$ matrix for which the first column is a column of ones and the second column is the values of $y$, $W^{\prime}$ is an $n \times n$ diagonal matrix whose $i$th diagonal element is the weighting factor  $w_{i}^{\prime}=s_{x_{i}}^{2}+\frac{1}{b_{1}^{2}} s_{y_{i}}^{2}-2 \frac{1}{b_{1}} \operatorname{cov}\left(x_{i}, y_{i}\right)$, and $s^{\prime 2}$ the experimental error associated with the predictor variables, given by
$s^{\prime 2}=\frac{\sum_{i=1}^{n} \frac{\left(y_{i}-\hat{y}_{i}\right)^{2}}{w_{i}^{\prime}}}{n-2}$
The variance of the prediction of the predictor variable of a future sample at $y_{0}$, the mean of $q$ observations, is
$s_{x_{0}}^{2}=\left(\frac{1}{q}+ Y _{0}^{ T }\left( Y ^{ T } W ^{-1} Y \right)^{-1} Y _{0}+s_{y_{0}}^{2} \frac{1}{b_{1}^{2}}\right) s^{\prime 2}$
The prediction intervals of the response and predictor variables are then given by
$y_{0} \pm t_{\alpha, n-2} s_{y_{0}}$
$x_{0} \pm t_{\alpha, n-2} s_{x_{0}}$
where $t_{\alpha, n-2}$ is the $t$-value for the required significance level $\alpha$ and $n-2$ degrees of freedom.
TODO: Interface to scipy.odr output.


does tensorflow use opencv to covert image to numpy array

TensorFlow has two ways of processing image data

Built-in Image Ops  ( https://www.tensorflow.org/api_guides/python/image )
tf.keras Pre-processing ( https://keras.io/preprocessing/image/ )

Feature extraction is at pixels / channel level. Higher level features (Such as Edge / Contrast) are learnt typically by CNN layers.

If needed, you can feed Image Data + Features extracted by OpenCV to TensorFlow.


You may want to model your problem using bayesian regression, which may allow you to introduce your prior knowledge in the form of priors (a priori distributions) of the model parameters. They would also allow you to model latent variables that govern the dynamics of the interactions (and impose priors on them as well).

The specific approach may be based on sampling (e.g. Markov Chain Monte Carlo) or optimization (e.g. Variational Bayes).

One of the most popular bayesian frameworks is Stan, which has bindings to R (rstan) and python (pystan). In R there are other alternatives such a BUGS and JAGS. In the python realm other options are PyMC (which is also pretty popular) or Edward.


Estimate battery voltage based on scheduled events and previous behaviour

One of the simplest ways to get to some predictions would be to use a model like ARIMA, which looks at recent previous observations to predict a number of steps ahead. ARIMA stands for Autoregressive Integrated Moving Average:

Autoregressive means that something looks at its own (auto) historical values
Integrated refers to the _differencing, a step tp help remove non-stationarity
Moving Average just means the model also takes the moving average into account (helps keep predictions in reasonable ball park)

Here is a good little explanation of the main points in more detail.

In your case, given there are factors such as weather and time of data to be used, you would probably benefit from using seasonality. For this there is an extended ARIMA, called SARIMA - where S stands for seasonal.
There is an implementation in Python's statsmodels package - or if you want to use R, then maybe take a look at the forecast package by the great Rob Hyndman, or the sarima package.

If you want to use something a little more modern and cutting edge, then you are really talking about Recurrent Neural Networks (RNNs). I am not sure how familiar you are with there? One key term you should understand: LSTM - a type of unit that considers past data, current data and maintains a state of your model at a specific point in time.

Have a look at a walkthrough (example) to see if it makes sense to you.

ARIMA type models are easy to get running and the results are easy to interpret, so you really know what is going on. The performance of things like RNNs should be better, but it is rarely a short walk to success... of course it will depend on your data.


As rightly pointed out by you the rescale=1./255 will convert the pixels in range [0,255] to range [0,1].
This process is also called Normalizing the input. Scaling every images to the same range [0,1] will make images contributes more evenly to the total loss.
Without scaling, the high pixel range images will have a large say to determine how to update weights. For example, black/white cat image could be higher pixel range than pure black cat image, but it just doesn't mean black/white cat image is more important for training.
Also the neural network has a higher chance of converging as it make the coefficients in range of [0,1] as opposed to [0,255] so helps the model process input faster.


I'm not sure it will matter very much whether you have one row per user or if you have one row for per month per user. The important part is that the data you have is accurate for that user for a particular month. You might construct your training data like this:

--------|-----------------|--------------------|
cust. id| time on website | profit over 10,000?|
--------|-----------------|--------------------|
 3         30               0
--------|-----------------|--------------------|
 3         80               1
--------|-----------------|--------------------|
 5         100              1
--------|-----------------|--------------------|
 7         5                0
--------|-----------------|--------------------|

The important thing to notice is that even though customer 3 is in the dataset more than once, he/she has different values for their data features on which to predict, and different from how they were represented in the previous month. This is assuming that you are aggregating the data by month per customer.

This blog predicts customer churn, but you might be able to use the general strategy for your problem.


How to structure the output layer of an MLP that finds the quadrant of an arbitrary point in a rectangle?

You need only two hyperplanes to solve this.
Thus you need two neurons in the hidden layer.
You can use two or four neurons in the output layer.
Both options result in the correct solution (theoretically).
You can use perceptrons. With perceptrons, the output is a boolean vecotor, not a probability.