Are there any unsupervised learning algorithms for time sequenced data?

What you have is a sequence of events according to time so do not hesitate to call it Time Series!
Clustering in time series has 2 different meanings:

Segmentation of time series i.e. you want to segment an individual time series into different time intervals according to internal similarities.
Time series clustering i.e. you have several time series and you want to find different clusters according to similarities between them.

I assume you mean the second one and here is my suggestion:
You have many vehicles and many observations per vehicle i.e you have many vehicles. So you have several matrices (each vehicle is a matrix) and each matrix contains N rows (Nr of observations) and T columns (time points). One suggestion could be applying PCA to each matrix to reduce the dimenssionality and observing data in PC space and see if there is meaningful relations between different observations within a matrix (vehicle). Then you can put each observation for all vehicles on each other and make a matrix and apply PCA to that to see relations of a single observation between different vehicles.
If you do not have negative values Matrix Factorization is strongly recommended for dimension reduction of matrix form data.
Another suggestion could be putin all matrices on top of each other and build a NxMxT tensor where N is the number of vehicles, M is the number of observations and T is the time sequence and apply Tensor Decomposition to see relations globally.
A very nice approach to Time Series Clustering is shown in this paper where the implementation is quiet straight forward.
I hope it helped!
Good Luck :)

EDIT
As you mentioned you mean Time Series Segmentation I add this to the answer.
Time series segmentation is the only clustering problem that has a ground truth for evaluation. Indeed you consider the generating distribution behind the time series and analyze it I strongly recommend this, this, this, this, this and this where your problem is comprehensively studied. Specially the last one and the PhD thesis.
Good Luck!


If you use these 2 packages:

http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

you will get what you want :)


How to numerically estimate MLE estimators in python when gradients are very small far from the optimal solution?

There are several reasons why you are getting erroneous results.
First, you should consider using log likelihood instead of likelihood. There are numerical issues with multiplying many small numbers(imagine if you had millions of samples you had to multiply millions of small numbers for the lhd). Also taking gradients for optimization methods that require gradients is often easier when you are dealing with the log likelihood. In general, it is good to have an objective which is a sum rather than a product of variables when dealing with optimization problems.
Second, fmin is using Nelder-Mead simplex algorithm which has no convergence guarantees according to scipy documentation. This means the convergence is totally random and you should not expect to find parameters close to the originals. To get around this, I would suggest you to use a gradient based method like stochastic gradient descent or BFGS. Since you know the generative model (rvs are Gaussian distributed) you can write the likelihood and log likelihood as:

Where a,b,c and d are your model parameters 5,2,3 and 4 respectively.
Then take the gradient with respect to [a,b,c,d] and feed that into prime input of fmin_bfgs. Note that due to varying variance what could be solved by just linear regression is now a nastier problem.
Finally, you may also want to check Generalized least squares here and here, which talk about your problem and offer several available solutions.
Good luck!


Found input variables with inconsistent numbers of samples: [30, 24]

I had the same problem, here is how I sloved it:
testX is a tf.data.Dataset, I guess.
In that case try the following:
# make the prdictions from your test set:
testPredictRaw = self.model.predict(testX)
testPredict = np.argmax(testPredictRaw, axis=1)

# Then, take all the y values from the prefetch dataset (thus changing the shape):
trueClasses = tf.concat([y for x, y in testY], axis=0)

# Calculate the confusion matrix using sklearn.metrics
cm = metrics.confusion_matrix(trueClasses, testPredict)

I used these answered questions to make it work for my case:

https://stackoverflow.com/questions/53620259/sci-kit-learn-confusion-matrix-found-input-variables-with-inconsistent-numbers
https://stackoverflow.com/questions/64622210/how-to-plot-confusion-matrix-for-prefetched-dataset-in-tensorflow


Understanding notation of Goodfellow's GAN objective function

You are thinking of it backwards :) i.e. don't ask "How do we get these expectation parts?" or "how he got from" sum to expectation, ask where we got the sums!

The expectations are what we actually care about. The sums are just what we use to approximate them.

Generally in ML we define the data space $\mathcal{X}$ (e.g., the set of all possible images) and a probability distribution $\mathcal{P}$ on that space (so that sampling from it gives e.g., the set of natural images). The goal is then to find a function $f$ such that the expected loss $ \mathbb{E}_{x\sim\mathcal{P}}[\mathcal{L}(f(x))] $ is minimal. (This is a weighted average over all possible images, weighted such that only the images we care about are considered).
This is the basic tenet of empirical risk minimization, upon which most ML theory is based (often, implicitly). Hence, most papers that are even remotely formal will state their loss functions in terms of expectations or integrals, not sums (since one cannot usually sum over a continuous space in a meaningful way).

In practice, we cannot access $\mathcal{P}$ (indeed, the whole point of GANs is to implicitly model it!). Instead, we have only a dataset $X\subset\mathcal{X}$ sampled as $X\sim \mathcal{P}$. For some loss function $\mathcal{L}$, we can Monte Carlo estimate the real expected $\mathcal{L}$ via
$$
\mathbb{E}_{x\sim\mathcal{P}}\left[\mathcal{L}(f(x))\right]
= \int \mathcal{L}(f(x)) p_\mathcal{P}(x)\, dx
\approx \frac{1}{|X|}\sum_{x\in X} \mathcal{L}(f(x))
$$
but even this sum is too hard (expensive) to compute, so we take a subsample (minibatch) $M\sim\mathcal{U}(X)$ uniformly from $X$ instead:
$$
\mathbb{E}_{x\sim\mathcal{P}}\left[\mathcal{L}(f(x))\right]
\approx \frac{1}{|M|}\sum_{x\in M} \mathcal{L}(f(x))
$$

So for

  but why is there expectation instead summation in the first place? Is it same to minimize expectation or to minimize function?

The answer is that we cannot compute the expectation, but we can approximate it with the sum or mean over a minibatch instead, so we use that. Ultimately we define our losses via expectations, and derive the sums to compute from those.

Now for the GAN specific stuff.

  What is the meaning of $V(D,G)$?

The point of GANs is learn two functions: $G$, which outputs $x=G(z)$ for $z\sim Q$, and $D$, which outputs a high value for real data ($x\sim P_R$) and low value for fake data ($x\sim P_G$, which we sample via $z\sim Q$ then $x=G(z)$).

What is a good loss function for $D$? Well, we want high values for real data, so let's try to maximize $\log D(x)$ where $x\sim P_R$ is any real data point. So we want the maximum expected value over all of the true dataset,
i.e. $\mathbb{E}_{x\sim P_R}\log D(x)$

Next we want low values for fake data, so let's try to maximize the expected $\log(1-D(x_g))$ over all generated fake data $x_g\sim P_G$. But we can equivalently say $x_g=G(z)$, for some $z\sim Q$. So we can instead maximize $\mathbb{E}_{x_g\sim P_G}[\log(1-D(x_g))]=\mathbb{E}_{z\sim Q}[\log(1-D(G(z)))]$.

Putting the two together, we see that we want $D$ to maximize both these terms, so we'd like
$$ \max_D \mathbb{E}_{x\sim P_R}[\log D(x)] + \mathbb{E}_{z\sim Q}[\log(1-D(G(z)))] =: \max_D V(G,D)$$
to get a good $D$.

What about $G$? Well, we want $G$ to make $D$ do poorly. But which $D$? The best one. In other words, we want $G$ to make the best $D$ (for that $G$) score as poorly as possible, i.e.,
$$
\min_G \max_D V(G,D)
$$
is our objective. That's where $V(G,D)$ comes from.

Now, how to get the sums from the expectations in $V(G,D)$? It's just a Monte Carlo sample approximation (i.e., a sum or mean over a minibatch) of the expectation, as before. :)


How do you see the element of a csv table with many columns (>30) which the names of its columns is more than 10 character in pandas?

I dont know which compiler you are using but mine is jupyter.

You can make wider notebook by:

from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))

By doing that you can display more columns with df.head(). If you cant display still ("..." between columns), you can use iloc. For you data I used df.iloc[:5,:15] and df.iloc[:5,15:].

Also which is a better solution you can change pandas display options by:

pd.set_option('display.max_columns', 500)

Here is the screenshot of my notebook:


Determine how each feature contribute to XGBoost Classification

If you have just then parameters and two of them are important. You can plot the trees and see the threshold for each of the parameters.

from xgboost import XGBClassifier
from xgboost import plot_tree
import matplotlib.pyplot as plt

# fit the model
model = XGBClassifier().fit(X, y)
# plot single tree
plot_tree(model)
plt.show()

The above code just plots the first tree. You can plot, for example, 4th boosted tree in sequences using the following line of code

plot_tree(model, num_trees=3)

Note that in each tree, you might have a different threshold for each parameter in an ensembling method.


CNN model contains several images that are null

In principle, it is to be expected that several feature maps result in zero output all over the image. What the convolutional layers are doing, essentially, is to move a stencil across the image and give a high value where this stencil shape is found in the image, and zero everywhere else. Each of the feature maps, i.e. each of the channels in the convolutional layers, corresponds to a stencil, the lower layers encode simple shapes, the higher layers more complex ones.

If a channel outputs zero all over, it simply means that this feature was not detected in the image.

As an example, consider an MNIST classifier. One feature map may have learned to look for straight horizontal lines. When the model is presented with a hand-written "7", the straight horizontal lines will be detected in the top part of the image and maybe also in the middle. So the feature map will give high values in the top part and maybe in the middle as well, while it will be zero everywhere else.

If the model is presented with a hand-written "0", there is usually no horizontal lines and therefore the feature map will be zero everywhere.

So if a feature map is zero everywhere, it usually means that the particular feature is not found in the current input, but it might be present in other images.

If, however, a feature map is all zeros for all input images, i.e. the channel in the convolutional layer is zero everywhere for all 60k MNIST examples, then there might something wrong with the network and you might want to apply batch normalization or dropout.


Is this a correct way improving a statistical model?

No offense, but let me guess, you work for a large company where there is no person with a full understanding of the process overseeing this? I say this because it sounds as if the testers want you to manually and intentionally over-fit your models. There is no such thing as a perfectly accurate model.

Yes, I would say that there is bias in your model building. New model input should be a random selection not by finding and labeling similar instances to ones that were misclassified.

Your production model will degrade over time (the distribution in the population has changed). You should have a sense of a threshold, an error rate at which is no longer exceptable and you will need to retrain. To retrain, you could add a random selection of new labeled data to your existing data, then train, validate, select model, deploy, and monitor.

You may "fix" an error report with a new retraining, but it is an unrealistic expectation that this will also cover all previous error reports. Your model needs to generalize. See here:

https://en.wikipedia.org/wiki/Test_set

Look at the image at the bottom. In essence you are being asked to create the orange model were you want to create the green model, which generalizes better to unseen data.


How to Present All Categories in All Samples

The first thing we must accept that the sampling is probably doing the right job.
What I mean is that if only 10% is being sampled then some unique value which is less than 5 can be easily missed.
 Ideally, you should club these values into some generic value i.e. OTHER_COL_1
But, if you want to get away with this natural result, you should apply some tweaking.
We may do the following -

Get the sample as you are doing now
Match the unique element of each column to the unique from the main data
Iterate on each col and missed unique value
Let's assume UNIQUE_4 is missed for COL_2
Sample all the records for UNIQUE_4 from COL_2 of main data and
Pick one random data out of it


Why increasing the number of units or layers does not increase the accuracy and decrease the loss?

The title of your question and the question itself are somehow different but I'll try to answer the question, the meaning of decreasing loss without changes in accuracy.

The reason is simply due to using probabilities. For instance, for classification task if you have an output $0.7$ for input and the last layer is a softmax, then you classify the input as that class which has $0.7$. Imagine you train more and that output changes to something like $0.95$. Consequently, the accuracy does not change because you already classify it as what it really is but the loss lessens.

To answer the question which is in the body of your post, there can be numerous reasons that I'll try to refer to them.

One of the possibilities is that your data of different classes have overlap in the current feature space. This may lead to high Bayes error. For instance, suppose you have two same inputs and the label of them are contradictory. In this situation, your performance cannot be improved. To check it whether you've got this problem or not, take a look at the histogram of your data.

Another problem can be the weakness of LSTM networks which cannot memorise numerous things. LSTM models are very good at things like considering the gender of a subject or the plural or singular form of subjects but in cases where they should consider many things simultaneously, they have difficulties.

Another reason can be the incorrect way of using dropout. At first, do not use it and let your network overfits the training data to find a good model. After fitting your data, try to use dropout.

You can also test Stacked LSTMs which are powerful models.


How can both generator and discriminator losses decrease?

In the subsection of the figure, they compute the average loss across 100 iterations, which is why the loss is monotonically decreasing because on average the loss does decrease with the training.
You are correct in inferring that if this was reported on an iteration to iteration basis, the loss would be a zig zag curve, which is less nice to look at than a smooth curve.
As in the comments, the loss can monotonically decrease to 0, if it is the Wasserstein distance. The Wasserstein distance is computed as $\sup_{f \in L_{k}} E_{x \sim P_X}[f] - E_{x \sim P_{\theta}} [f]$ where the first term is the expectation of the function, $f$ (discriminator) over the batch and the second term is the generated data from the generator computed on the discriminator. Note that by this formulation, it returns the $k$-Wasserstein distance.


Python and GridSearchCV how to eliminate input contains NaN error when using cross validation and decision tree classifier?

You should impute missing values, try with:

from sklearn.impute import SimpleImputer
imp = SimpleImputer(missing_values=np.nan, strategy='mean')
imp.fit(x_train)
x_train = imp.transform(x_train)
x_test = imp.transform(x_test)

Notice that I am fiting just in the train data, so you are not leaking information to the test.


Error encoding categorical features using sklearn pipelines

In this line:

("impute_stage", Imputer(missing_values=np.nan, strategy="median"))

Because your input type is string, you shouldn't fill the null value to median (we cannot average string value).

From the document, you can fill null value with a string constant like:

Imputer(missing_values=None, strategy="constant", fill_value="NULL")

to represent null value in your string field.


$h_\theta$ implies that you're trying to model the relation between $h$ and $x$ with an straight line coming from the origin $(0,0)$. The parementer $\theta_1$ is the slope of this line.

Evaluating $J(\theta_1=0) = J(0)$, implies that $h_\theta(x_i)=0$ whatever the value of $x_i$ is. Since $m=3$, and the labels are $y_1=1, y_2=2, y_3=3$

$J(0) = \frac{1}{6}[(0 -1)^2 + (0 - 2)^2 + (0-3)^2] = \frac{14}{6}$

The value of $\theta_1$ for which $J(\theta_1) = 0$ is obviously 1


Any useful tips on transfer learning for a text classification task

Generally the task of recognizing one type of text against "anything else" is a quite difficult problem, since there is so much diversity in text that there cannot be any good representative sample of "anything else".
Typically this problem is treated as a one-class classification problem: the idea is for the learning algorithm to capture what represents the positive class only, considering anything else as negative. To my knowledge this is used mostly for author identification and related stylometry tasks. The PAN workshop series offer a good deal of state of the arts methods and datasets around these tasks.
It is also possible to frame the problem as binary classification, but then one must be very creative with the negative instances in the training set. Probably the main problem with your current approach is this: your negative instances are only "randomly selected among all other topics of the site". This means that the classifier knows only texts from the site on which it is trained, so it has no idea what to do with any new text which doesn't look like anything seen in the training data. A method which has been used to increase the diversity of the negative instances is to automatically generate google queries with a few random words which appear in one of the positive instances, then download whatever text Google retrieves as negative instance.
Another issue with binary classification is the distribution of positive/negative instances: if you train a model with 50/50 positive/negative, the model expects that by default there is 50% chance for each. This can cause a huge bias when applied to a test set which contains mostly negative instances, especially if these don't look like the negative instances seen during training.
Finally be careful about the distinction semantic topic vs. writing style, because the features for these two are usually very different: in the former case the stop words are usually removed, the content words (nouns, verbs, adjectives) are important (hence one uses things like TFIDF). In the latter it's the opposite: stop words and punctuation should be kept (because they are good indicators of writing style) whereas content words are removed because they tend to bias the model the topic instead of the style. In stylometry features based on characters n-grams have been shown to perform well... even though it's not very clear why it works!


Because the data is time series while only Dense layers are used in the model, the problem is caused by model initialization. A model with a 'bad' initialization will constantly predict zero, as you will see by running the script below.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Model,Sequential
from tensorflow.keras.layers import Dense

import numpy as np
import tensorflow as tf

# fix keras random state
# https://stackoverflow.com/a/52897216/8366805
seed_val = 94
np.random.seed(seed_val)
tf.set_random_seed(seed_val)
# Configure a new global `tensorflow` session
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)

# main
def series_to_supervised(data,n_in,n_out):

    df = pd.DataFrame(data)
    cols = list()
    for i in range(n_in,0,-1): cols.append(df.shift(i))
    for i in range(0, n_out): cols.append(df.shift(-i))
    agg = pd.concat(cols,axis=1)
    agg.dropna(inplace=True)

    return agg.values

n_in,n_out = 14,1
data = np.load('data.npy')
scaler = MinMaxScaler(feature_range=(0, 1))
scaler = scaler.fit(data)
scaled_data=scaler.transform(data)
DATA = series_to_supervised(scaled_data[:-10], n_in, n_out)
X, Y = DATA[:, :-n_out], DATA[:, n_in:]
X_train,X_test,y_train,y_test = train_test_split(X,Y,test_size=0.1,random_state=49)

model = Sequential()
n_nodes = 10
model.add(Dense(4*n_nodes,activation='relu',input_dim=n_in,name='dense_0'))
model.add(Dense(2*n_nodes,activation='relu',name='dense_1'))
model.add(Dense(n_nodes,activation='relu',name='dense_2'))
model.add(Dense(n_out,activation='relu'))
model.compile(loss='mse',optimizer='adam',metrics=['mse'])
# fit
history = model.fit(X_train,y_train,validation_data=(X_test,y_test),epochs=20,)

pred = model.predict(X,)
print('model prediction, mean %.3f, std %.3f' % (np.mean(pred),np.std(pred)))

for ind in range(3):
    intermediate_layer_model = Model(inputs=model.input,outputs=model.get_layer('dense_%i' % ind).output)
    pred = intermediate_layer_model.predict(X)
    print('layer %i, mean %.3f, std %.3f, min %.3f, max %.3f' % (ind,np.mean(pred),np.std(pred),np.min(pred),np.max(pred)))

In this script, I saved the array data in OP's post, to data.npy, which can be found in this repo, for simplicity. Besides, I fixed the random seeds of keras and train_test_split, therefore, you will reproduce the scenario in which the trained model constantly predicts zero.
In fact, as you mentioned in your post, similar scenarios are not rare (and try a shallower model does not help), I think the problem is Dense is simply not capable of dealing with time series, you need LSTM instead. Try the code below, in which I replaced Dense with LSTM+Dropout, besides, I changed the activation function of the output layer to tanh.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Model,Sequential
from tensorflow.keras.layers import Dense,LSTM,Dropout

from matplotlib import pyplot as plt
import numpy as np
import tensorflow as tf
from tensorflow import keras

# main
def series_to_supervised(data,n_in,n_out):

    df = pd.DataFrame(data)
    cols = list()
    for i in range(n_in,0,-1): cols.append(df.shift(i))
    for i in range(0, n_out): cols.append(df.shift(-i))
    agg = pd.concat(cols,axis=1)
    agg.dropna(inplace=True)

    return agg.values

n_in,n_out = 14,1
data = np.load('data.npy')
scaler = MinMaxScaler(feature_range=(0, 1))
scaler = scaler.fit(data)
scaled_data=scaler.transform(data)
DATA = series_to_supervised(scaled_data[:-10], n_in, n_out)
X, Y = DATA[:, :-n_out], DATA[:, n_in:]
X_train,X_test,y_train,y_test = train_test_split(X,Y,test_size=0.1,random_state=49)
X_train = X_train[:,None,:]
X_test = X_test[:,None,:]

wrong_ind = 0
for ind in range(100):
    print('working on %i' % ind)
    keras.backend.clear_session()
    model = Sequential()

    model.add(LSTM(4,name='lstm_0'))
    model.add(Dropout(0.2,name='dropout_0'))
    model.add(Dense(n_out,activation='tanh'))
    model.compile(loss='mse',optimizer='adam',metrics=['mse'])
    # fit
    n_epoch = 5 if ind < 99 else 200
    history = model.fit(X_train,y_train,validation_data=(X_test,y_test),epochs=n_epoch,verbose=0)
    val = history.history['val_loss']
    if np.abs(val[0] - val[-1]) < 1e-4:
        print(ind,val)
        wrong_ind += 1

print(wrong_ind)

fig,ax = plt.subplots(nrows=1,ncols=2,figsize=(12,8))
ax[0].plot(history.history['val_loss'],'r')
ax[0].plot(history.history['loss'],'b')
ax[1].plot(model.predict(X[:,None,:]),'r')
ax[1].plot(Y,'b')
plt.show()

the output is


Yes, you can mix any different sort of inputs when the scales of the features are similar, which is achieved by normalising the feature vectors.

I assume you mean too many features when you say 'too much input'

If you mean the size (number of training examples) of input data, size of input data is not directly related to overfitting. Overfitting depends on model complexity. It happens when model tries to fit to the noise of the input data and hence becomes too specific that it can't generalize well to new training data.

Any model that is "sufficiently" complex (for eg. one that contains many hidden layers, large number of neurons in each layer, whose weights are not regularized) can easily converge to give very little loss on training data (unless it converges to a different sub-optimal local minima), but will give poor accuracy on test data. But in general, on the contrary, lack of enough data often  leads to overfitting because the model tries to learn based on very few specimens which are less diverse. It's like showing a child a samples of balls containing only white and orange table tennis balls, and asking him/her to identify a blue colored ball.

Too many features can lead to overfitting because it can increase model complexity. There is greater chance of redundancy in features and of features that are not at all related to prediction.

For eg. if you're predicting quality of a tennis ball, the feature chosen as colour of the ball is irrelevant, but the network will learn from training examples and there is a chance that since people like yellow colored balls to play with, they play more often with them and those balls don't last long.


Tensorflow hub module taking long time for embedding single sentence

I would recommend using the model like this:
import tensorflow_hub as hub
module_url = "https://tfhub.dev/google/universal-sentence-encoder/4"
model = hub.load(module_url)
print ("module %s loaded" % module_url)
def embed(input):
    return model(input)

And then:
embed(["This is a nice string!"])

Which would return:
<tf.Tensor: shape=(1, 512), dtype=float32, numpy= array([[ 8.84598400e-03, ... , 1.89793427e-02]], dtype=float32)>

That is the 512 dimension vector, but you can choose the one you need between all the encoders here: https://tfhub.dev/
Hope it helps.


How to combine two CART decision trees learned in same type of data?

You mention two decision trees. Traversing a decision tree is very cheap, so running a feature instance through multiple trees is very fast, you could just take all the decision trees from the data centers and average the outcome, maybe weigh it by the (crossvalidated) strength of the models. Random Forests are powerful models that also combine decision trees in this way, except that they are done on random subsets of the features (and in some cases also random subsets of the data).


Extracting Features Using TensorFlow CNN

CNN feature extraction in TensorFlow is now made easier using the tensorflow/models repository on Github. There are pre-trained VGG, ResNet, Inception and MobileNet models available here.

I have used the following wrapper for convenient feature extraction in TensorFlow. You can just provide the tool with a list of images. It will pre-process the images and extract the features by feed-forwarding through the network. You can specify the layer names that you want to extract and save them to HDF5 or pickle afterwards.

https://github.com/tomrunia/TF_FeatureExtraction


R and Python, memory differences yielding performance differences

It depends on the usage of various packages (Numpy/Scipy etc), which are written in C and be incredibly fast also Python can be complied using JIT.  Here is an excellent comparison between R and Python : https://learnanalyticshere.wordpress.com/2015/05/14/clash-of-the-titans-r-vs-python/


R plot(surv(), newdata=...) draws same lines many times - why?

Note that in the linked presentation, on the slide titled "Plotting the effects", the treat object has only 2 rows.

In your case, because frame has 57k rows, fitObjGrouped has predictions for each row of newdata.  You can verify this with fitObjGrouped$n.  To fix the problem, try:

frame <- data.frame(group = unique(dataset$group))


One hot encoding with too many features (~ 10,000)

I think you are on the right track. What you are looking for is what is known as fixed effects in econometrics/statistics. Your have a model in which features X determine outcome y (so y = bX + u). On top of that you have unobserved heterogeneity (each individual employee) in your model. For each employee, the relation y <-> X may be a little different (fixed effect per employee), so that your model becomes y = b1X + b2Z + u, where Z denotes employees.

See here for an overview and here for an R implementation.

You can solve this by adding one dummy/indicator/one-hot per employee (known as least squares dummy variable fixed effects). So technically speaking, each employee gets his/her own intercept in the model. Thus the Z above is a massive matrix of one-hot encoded employee labels.

There are solutions in Stata (areg) and R (felm) which "absorb" the massive dummy variable set, so that it does not bother you when looking at the regression coefficients. If you are not interested in statistical inference (but only in making predictions), you will not look at the coefficients anyway.

I don't know the datails of your data (i.e. observations per employee), but from what you say, adding one dummy (aka one-hot feature) per employee would be the right way to go to control for unobserved heterogeneity of the employees.


Does anyone use Julia programming language?

I personally have used Julia for a good number of professional projects, and while, as Dirk mentioned, this is purely conjecture, I can give some insights on where Julia really stands out. The question of whether or not these reasons will prove enough to have Julia succeed as a language is anyone's guess.

Distributed Systems: Julia is the easiest language I've ever dealt with in terms of building distributed systems. This is becoming more and more relevant in computing, and will potentially become a deciding factor, but the question of whether or not Julia'a relative ease decides this is up for debate
JIT Performance: Julia's JIT compiler is extremely fast, and while there is a lot of debate as to how accurate these benchmark numbers are, the Julia Website shows a series of relevant benchmarks
Community: This is an area where Julia just isn't quite there. The community that is there is generally supportive, but not quite as knowledgable as the R or python communities, which is a definite minus.
Extensibility: This is another place where Julia is currently lacking, there is a large disconnect between the implies code patterns that Julia steers you toward and what it can actually support. The type system is currently overly bulky and difficult to use effectively.

Again, can't say what this means for the future, but these are just a couple of relevant points when it comes to evaluating Julia in my opinion.


No statistical significance but observable trends

There is high variance within each group. Even though there is a mean difference between the groups, there is a high amount of spread within just treatment A or just treatment B.

From a statistical point of view, the difference between the groups could be due to chance because of the large spread relative to the small mean difference.

Due to the amount of randomness observed - if we did ran the experiment again, treatment B mean could be higher than treatment A mean.


over-fitting with good enough test accuracy

There's probably a bit of interpretation involved in the question, here is my take on it.

For the sake of clarity let me start with my definition of the concept: overfitting is when a model takes into account patterns which are present in the training data by chance, i.e. the model assumes that these patterns are characteristic of the distribution even though they're not.

  Is it true to argue that no effective learning has taken place here?

No it would be wrong to say that: the fact that a model overfits does not mean that no effective learning happened at all. In fact it's often the case that a model successfully acquires the patterns that it's supposed to capture (let's call this effective learning), but also acquires patterns that it shouldn't (overfitting). With any complex data it would even be very rare that no overfitting happens at all, and in fact very often it's hard to say exactly where is the line between effective learning and overfitting.

However people usually talk about overfitting when it's actually in excess, that is when the model generalizes too much on the "chance patterns" and not enough on the patterns which actually characterize the distribution. A performance which is significantly lower on the test set than the training set is a typical sign of such "excess overfit". So in this sense (excessive) overfitting is when the "chance patterns" cause the model to be suboptimal. But even a suboptimal model may have learned some relevant patterns.

  If so, can one conclude from this that a better model with better accuracy in test exists that does not overfit?

Not really: as said above overfitting can make the model perform poorly, so naturally it's usually the case that a non-overfit model performs better than an overfit one. However there's no guarantee that a better performance can be reached just by getting rid of the overfitting: as an extreme example, if the features are mostly random and/or not correlated with the response variables then it's very likely that model will overfit, but the performance would be terrible without overfitting anyway.

In the case described by OP I would say that it's always worth trying to avoid (excess) overfitting: first of course because it can only improve performance, but also because at a more general level it means that the model is not very reliable. If later the model is applied "in production" to a large set of instances which happen not to have this particular "chance pattern" which was in the training data, the model is going to go very wrong and it will be too late to detect it.


Unusually High BLEU score on a NMT model

According to recent publications, it is not impossible to get BLEU scores as high as yours for English→Irish. Nevertheless, without any other knowledge, they certainly seem too high.

From the command line arguments, there does not seem to be any evident problem.

The most probable explanation is, as you already pointed out, a data leakage between validation/test and training. Note that, while you removed exact duplicates, you may be getting partial matches that go unnoticed. You may want to look into different similarity metrics. The most straightforward is the Jaccard Similarity.


Standard Scaler drops accuracy significantly in Scala Spark

As @n1k31t4 said in the comments, it's the .setWithMean(false) call (should be true).

Moreover, you don't really need the StandardScaler at all, since LogisticRegression will perform standardization by default (see the standardization param).


Comparing Parameter Importance Across Models

This may not be the answer you are hoping for (I can't leave a comment because my reputation is too low) but I believe feature importance in Sci-Kit Learn is derived from the average depth at which each feature appears. I don't know of any built-in functions in Sci-Kit Learn but perhaps you could scale the feature importance using the depth of the tree and the number of features?

This article may help with developing a comparative measure of feature importance: https://medium.com/the-artificial-impostor/feature-importance-measures-for-tree-models-part-i-47f187c1a2c3.


Do Data Miners realise Machine Learning cannot establish Causality?

I think you're a bit confused about the role machine learning plays in the order of things, and I find your impression of the ML community to be a bit bizarre.

To begin, ML experts are often not the people who use ML algorithms to solve science problems - those people are usually called "data scientists" these days, and in principle they are supposed to have some sort of background in science, statistics, and/or mathematics.  And yes, (good) data scientists spend a lot of time worrying about causal inference - they have to because there are big companies with lots of money riding on their predictions.  Contrary to your impression ML experts are usually aware of these issues too, but their job is to build and analyze modeling tools rather than actually apply them to real data.  Of course there are lots of ML and data science people which straddle the line between the two fields, but my point is that your question is a bit like asking why mathematicians don't worry about experimental error.

That said, many (most?) ML algorithms are more oriented towards classification problems which are not quite as vulnerable to your concerns.  You still have to worry about sticky statistical problems like overfitting and building good training / validation datasets, but the focus is more on understanding the structure of the data that you have than predicting the future.


Multiple output for multi step ahead prediction using LSTM with keras

model = Sequential()
model.add(LSTM(input_dim=n_features, output_dim=neurons))
model.add(Dropout(.2))
model.add(Flatten())
model.add(Dense(ahead))
model.compile(loss='mae', optimizer='adam')
model.fit(X, y, epochs=50)

The differences include:

I'm flattening the output from LSTM, cause it's going to give a 3D tensor and a Dense layer for this scenario I believe is going to expect a 2D tensor.
Made the change the change suggested in another answer.
Removed the activation layer, since default activation of Keras dense layer is linear.


Which function are you using at Loss? Using the right one is important when dealing with imbalanced datasets. 7% is imbalanced, but not that bad.

Have you tried any eXplainable Artificial Intelligence (XAI) method? Normally I use Shap.
It is really good to see which feature contributes in which direction. You can see an example here.

You can check the github here
You can check the paper here


In scikit-learn's linear regression, the parameters that minimise the squared error loss aren't estimated using gradient descent, they are computed exactly.

The minimisation problem for linear least squares is

$$
\hat{\beta} = \underset{\beta}\arg \min || \mathbf{y} - \beta \mathbf{X} ||^2
$$

which has a unique solution (assuming the columns of $\mathbf{X}$ are linearly independent):

$$
\hat{\beta} = (\mathbf{X}^\text{T}\mathbf{X})^{-1}\mathbf{X}^\text{T}\mathbf{y}
$$

For classifiers that are fitted with an iterative optimisation process like gradient descent, e.g., MLPClassifier, there is a parameter called max_iter which sets the maximum number of epochs. If tol is set to 0, the optimisation will run for max_iter epochs.


Does an RL agent still learn if its actions are "blocked"?

By overriding the Agent's action, Agent can theoretically take this action over and over and do nothing else, there is nothing to stop the agent from doing this, or teach the agent that this is not the intended action here.

What I usually do in these situations is, I punish the agent if the action is not desirable. So if agent take the action to go to the left, but there is a wall on the left side, I would not change the state of the environment (So the agent won't move), but I would also send a minus value (punishment) as a reward. This way after some training, the agent would learn that this action is not desirable.

The same can be applied to you Stock example. So if the agent tries to sell more stock that it actually has, you just don't sell the stock but also punish it with a big minus reward. This way it makes it easier for the agent to actually understand the environment.

Hope this helps.


How to Setup Multivariate Time Series Dataset for Classification

I have good experiences with Keras LSTM. Think it should also work with the time constant features. Here is a helpful tutorial. https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/


Why does np.linalg.eig produce an opposite-signed eigenvector?

Any scalar multiple of an eigenvector is also an eigenvector.  LAPACK (which np.linalg.eig uses under the hood) chooses to return unit-length eigenvectors (good for SVD!), but this still leaves two choices, and there doesn't seem to be a convention for which one to return; it's up to the underlying algorithm (which in turn may depend on the input data).

https://stackoverflow.com/questions/17998228/sign-of-eigenvectors-change-depending-on-specification-of-the-symmetric-argument


what is the reason behind the bad outputs gained by RNN, LSTM when using GloVe pretrained model in text classification?

I have already dealt with RNN and LSTM codes and finally I found tow solutons to achieve better result for accuracy measure which are as follows:
at first I change the activation function, optimizers and learning rate but it is worth noting that activation function has the most impact and the second trick which I have used is to removing stop words, now the code works fine.
I hope it is useful.


comparing data from first quarter in two years

I am not sure what output you are looking for.

There are many ways to do it. You can do it by creating a measure in Power BI or in PowerPivot or just using Powerquery - below is an example.

PowerPivot and PowerQuery can both put the output into a pivot table in Excel if that's something you prefer.

Pick up city    Q1       Q2
X              Amount1   Amount 2
Y              ...       ...


Is there some model-based variation of the Q-Learning algorithm which learns on a 3D SxAxS' table instead of a 2D SxA table?

The concept of state-action values $Q$ is to denote how good is to be in a particular state and perform a particular action in terms of expected future reward. From what I understand from your question, you are interested in the problem of model uncertainty (uncertainty on the dynamics of the system). In other words, our artificial agent interacts within an unknown environment (transition dynamics $T(s,a,s')$ and reward dynamics $R(s,a)$ or $R(s,a,s')$ are unknown).

The framework you should take a look at is Bayesian Model-based RL. I outline an approach so you can have an idea:

Modelling Transitions

First assume that we have uncertainty on the transitions of the environment $T(s,a,s')$.
To tackle this we will assume that our agent maintains a distribution over possible transitions. Without getting into the theoretical math, I will illustrate this by using a simple Dirchlet-Multinomial model:

The states are sampled from a Multinomial likelihood $s'\sim Mult(p_{ss'}^{a})$ and we assume a prior over the transitions $p_{ss'}\sim Dir(\alpha)$, where $\alpha$ is set to $1/|\cal{S}|$, where is $\cal{S}$ is the state space.
The posterior over transitions will be also a Dirichlet because of the conjugacy of the likelihood and prior distributions. To update such a posterior you need to perform simple algebraic calculations and maintaining the counts of each transition.

The Algorithm

The agent does two processes:

(Simulation or Planning phase) Samples an MDP from the posterior over transitions (in the first iteration you sample from the prior distribution). Having a 'fixed table of tranistions' solves the sampled MDP (e.g with Value iteration) and selects the optimum action.
(Real-world interaction phase) In the real world, you observe the reward of your action and the new state of the environment. You update the posterior counts.

Eventually, and if you have chosen an appropriate distribution for your domain the agent will adapt to the unknown environment. Of course richer priors with non conjugacy will lead to MCMC sampling methods. I refer you to this paper to get an overview of the problem: Model-based Bayesian Exploration and the quite advanced: Bayes-Adaptive MDPs for further research and exploration.


How can I classify specific types of words in a document given I have the full text of the document and the labels

The task you describe corresponds exactly to Named Entity Recognition (NER). This is a standard task for which there are many available libraries. NER is usually done with sequence labelling models such as Conditional Random Fields. Sequence labelling implies that the data is provided as a sequence which looks like this:
During    <features ...>  O
the                       O
process                   O
of                        O
protein                   O
synthesis                 O
,                         O
X_token1                  B_classX
X_token2                  I_classX
X_token3                  I_classX
was                       O
used                      O
.                         O

Here I'm using the common BIO format (Begin, Inside, Outside an entity) but there are variants. The model is trained with data annotated in this way, where there can be additional features (very often POS tag and others). Then when fresh text is provided (with the features) the model predicts the BIO tag for every token.
There has been a lot of research and resources produced for extracting specific entities in the specific context of biomedical data, so you might be interested in exploring these specific resources as well.

Medline, PMC are huge collections of biomedical abstracts/papers
There are many tools for extracting biomedical annotations based on Medline/PMC data: PubTator, cTakes, SciSpacy,etc.


Using GRU with FeedForward layers in Python

The error is caused by return_states = True. You set it to True only if you feed the output of a recurrent layer to another. The "states" are the hidden states of recurrent cells, that could be fed to Dense() layers only though Flatten().

Moreover, I suggest you to delete the Dropout() layer. Don't put it after an Embedding(), it contains information (i.e. the learned representation of words/chars) that is not safe to be distorted by dropout. This is just a suggestion of course.

EDIT:

You need return_sequences = True only when a recurrent layer outputs to another recurrent layer. If the following layer is Dense(), then you can drop return_sequences = True and also Flatten().


How exactly does DropOut work with convolutional layers?

As you mentioned, the mask matrix is sampled and multiplied with the activations in the feature map at layer $l$ to produce dropped out modified activations which are then convolved with the filter at the next layer $W^{(l+1)}$. (3)

For more details, I think section 3 in this paper might help you out: Max-pooling & Convolutional dropout. Specifically 3.2.

When you test you use all nodes of the network but with the filter’s weights scaled by the retaining probability, as explained in the paper.

Please feel free to refine or correct my answer.

Hope this helps at least a little.


Digitize graphs. Pull out data from points in a scatter plot

Digitizing Graph. I know a tool that can semi-automatically extract data, (scattered points and curves) from graph images within a few clicks. But it is not in Python.
It is PlotDigitizer.com. I have used the tool and based on my experience it is better than some others I have used earlier.
Here is a simple guide to use it:

Upload the image graph
Calibrate both axes graph - you have to do it manually
Pick the color of points or curves you want to extract
Select auto-extracting algorithms, e.g. points, curves

Done. You have extracted the data points.
You can find a more detailed guide on its website: plotdigitizer.com.


Cyclic dependency between feature and predictor class

You have two problems:

Technical problem: As 10xAI said in a comment, if the target also belongs to the features then the model should very easily predict every instance correctly. So you should obtain perfect performance on the test set. The ML model doesn't care about "cyclic dependency", it uses any good indicator it receives as feature. Since you mention a reduction in accuracy, it means that there is an error somewhere in the process.
Semantic problem: which problem are you trying to solve with this ML setup? If the goal is to distinguish software vs. non-software and you have as input a category "software" then it's pointless to train a ML model. You can obtain the same result much more efficiently with a simple condition:

for each instance:
  if instance.category == "software":
     instance.answer="software"
  else:
     instance.answer="non-software"


Unable to print in Jupyter Notebook using Pandas

Instead of calling answer_two, call answer_two(). you are referring to the function object now. You have to call the function.


is there a way to customize my loss function to increase recall in one class only?

What you want to do to boost the performance of one of your classes is to add class weights to your loss function. This will essentially tell your network to pay more attention to some classes (note that this might deteriorate performance on the rest of the classes).

How to compute the weights.

To see what weight you'll give to each class you can either put it yourself or you can let scikit-learn help you. The problem is that your data is imbalanced. This means that you have much more samples in class 2 than you have in classes 0, 1 and 3. Through the use of class weights you can make it so that your model pays more attention to the under-represented classes, so that they appear balanced. Scikit-learn has a function that calculates what weights you should give each sample.

from sklearn.utils.class_weight import compute_class_weight
weight_array = compute_class_weight('balanced', np.unique(y) y)  # y is the array containing your labels

weight_dict = dict(zip(np.unique(y), weight_array))  # dictionary containing your weights

Keras

In keras you can also add class weights in the fit method (which trains the model).

This is easier as you just need to pass a dictionary with the weight of each class (i.e. what we calculated previously)

model.fit(X_train, y_train, class_weight=weight_dict, ...)

TensorFlow

In TensorFlow you can do this simply by using this softmax_cross_entropy loss instead of the one you are currently using. This supports class weights (weights attribute).

This requires you to pass a Tensor with a shape of (batch_size,) where its elements are the weight you want to pass to each sample. What you need to do is to see what samples batch_y has and give them the respective weight!

batch_weights = [weight_dict(x) for x in batch_y]  # assuming that your batch_y is a numpy array that is not one-hot encoded

# if it is one-hot encoded:
batch_weights = [weight_dict(x) for x in np.argmax(batch_y, axis=1)]  # assuming batch y is a numpy array

# if batch_y is not a numpy array you need to get it through
batch_y_arr = sess.run(batch_y)

Then pass the batch_weights to the weights attribute of the loss (through a placeholder using the feed_dict - like you do with batch_x and batch_y)


Need help understanding Hard SVM quadratic program equation

You want to maximize the margin, so that all points of each class are as far as possible into their halfspace. Mathematically this implies your equation: $$y_i\left(\langle\mathbf{w},\mathbf{x_i}\rangle+b\right) > 1   \;\;\;\;\;  \forall_i$$
Let's think about what this actually means.
If you divide both sides of the inequality by $||\mathbf{w}||$ you get:
$$y_i\left(\langle\frac{\mathbf{w}}{||\mathbf{w}||},\mathbf{x_i}\rangle+\frac{\mathbf{b}}{||\mathbf{w}||}\right) > 1/||\mathbf{w}||   \;\;\;\;\;  \forall_i$$
The expression:
$$\langle\frac{\mathbf{w}}{||\mathbf{w}||},\mathbf{x_i}\rangle+\frac{\mathbf{b}}{||\mathbf{w}||}$$
basically expresses the distance of each point from your hyperplane, which is parameterized by the normal-vector $\frac{\mathbf{w}}{||\mathbf{w}||}$ (hyperplane orientation) and $\frac{\mathbf{b}}{||\mathbf{w}||}$ (hyperpplane bias), since you are projecting $\mathbf{X_i}$ on a unit-length vector.
Now if you maximize the righthand side of the equation (minimizing $\mathbf{w}$), it will necessarily maximize the lefthand side as well, due to the inequality direction, but, assuming that your representation is normalized as described above.
Hopes this helps shed some light here.


Are there algorithms for clustering objects with pairwise distances, without computing all pairwise distances?

Well, one may argue that DBSCAN is based on all pairwise distances, but it uses data indexing to avoid computing all of them using geometric bounds.

And there are other examples if you browse through literature.

For example, the classic CLARA method is an approximation to PAM that avoids computing all pairwise distances.

And there are many more such techniques.


The formula would make sense if $y_n$ is a row vector representing the one-hot encoded label of the classes, and the multiplication is with the single column matrix $log(f(B A x_n))$ representing the log likelihood over all the clases given by the softmax function $f$.

As for $x_n$, it of course must be a vector as well, representing the $N$-grams in the $n$-th document.


How to add Error Bar to Matplotlib line plot?

Why not compute the x values explicitly? In the dataframe above,
x = [c for c in res.columns() if isinstance(int, c)]

Then, the code can be as follows.
badges = ["A", "B", "C"]
fig, ax = plt.subplots()
for badge in badges:
    means = res.iloc[:,2:].mean()
    std = res.iloc[:, 2:].std()
    means.plot(kind='line', label = badge)

    ax.plot(means)
    ax.fill_between(x, means.sub(std), means.add(std), color='b', alpha=.1)

Note: In the code above, you use badges only in the plot. Maybe computing means and std should be done outside the cycle? I am not sure because your sample data frame above does not have badges.


Find best (most efficient) combination of Ingredients

This is an optimization problem and optimization is all what Machine Learning does in classic sense (mostly not unsupervised approaches).
Your problem can be designed to be solved by Genetic Algorithms. For this you need to make a sequences of ingrediants which is called chromosome (genetic representation of solution) and an objective function which is to be minimized/maximized according to the chromosomes (it's usually called fitness function and is to be maximized then).
You have genetic operators to be applied randomly on your chromosomes:

Reproduction: 2 parent chromosomes result in two children.
Mutation: One single bit (one of ingrediants) is randomly chosen to
be switched with another one (like in a genetically normal family, a
baby is born with a genetic phenomenon).
Natural Selection: Uncle Darwin believed that nature "keeps the best
and  discards the rest". But who is the best? Well, you already have a fitness function!

Continue till you reach the optimum and select the best chromosome.
Of course it was the intuitive explanation of this beautiful idea. See more here and if you needed implementation details please drop a comment here.
Good Luck.
Update: More Details
I have an optimization problem which is too complex or large to be solved analytically so I need a heuristic method to approximate the optimal solution. For instance each subset of $28$ ingredients outputs a result $y$ where $y \in [0,100]$ (it's much bigger than your sample problem just to use optimization properly).

Initialization: Randomly produce $n$ combinations of material as initial population. Choose probabilities $P_{crossover}=0.25$, $P_{mutation}=0.01$ and $n_{selection}=250$ (values are just examples)
Evaluation of Fitness: Evaluate the output for each of them

| input         | output
| ------------- |:-------------:|
| ABCDEFGH      | 23.5          |
| BDCGTRFJ      | 45.6          |
| ZHTRFGOIUPOIZ | 18.3          |
| ...           | ...           |

Choose top $n_{selection}$ and ignore the rest. If you found an optimal solution (in your case you said output=3 is desired) stop, if not:
According to $P_{crossover}$ choose 25% of survived samples and reproduce children by combining them through a breaking point (image from here).

For instance if $ABCDEFGH$ and $BCDEFGHJ$ are survived, then choose a breaking point and exchange their parts (the image is 2-point crossover. You may apply 1-point crossover. It's all about the variations of the same concept). Technically the crossover point should be also determined probabilistically but I just skipped it. Break them from wherever you want! Here we choose $3^{rd}$ gene for this so $ABCDEFGH$ and $BCDEFGHJ$ will become $ABDEFGHJ$ and $BCCDEFGH$. Two children out of two parents. Apply it on all chosen survived samples.

According to $P_{mutation}$ choose 1% of survived samples and randomly(again, technically with a predefined probability) change one material in it (technically the samples are usually binary so you flip between $0$ and $1$. Here as you have nominal variables you need to change the variable). Let's say in survived sample $FVBGTZHU$ I randomly choose $V$ and exchange it with $P$ so I get $FPBGTZHU$.
Now the new generation is born (keep it as the same size of the original population for better structure). Go back to 2 and continue till you find your answer.

For different variations of GA (Elitism, Island Model, etc.) see the literature for Evolutionary Algorithms e.g. this one.
Hope it helps!


Can machine learning algorithms predict sports scores or plays?

There are a lot of good questions about Football (and sports, in general) that would be awesome to throw to an algorithm and see what comes out. The tricky part is to know what to throw to the algorithm.

A team with a good RB could just pass on 3rd-and-short just because the opponents would probably expect run, for instance. So, in order to actually produce some worthy results, I'd break the problem in smaller pieces and analyse them statistically while throwing them to the machines.

There are a few (good) websites that try to do the same, you should check'em out and use whatever they found to help you out:

Football Outsiders
Advanced Football Analytics

And if you truly want to explore Sports Data Analysis, you should definitely check the Sloan Sports Conference videos. There's a lot of them spread on Youtube.


There are different ways to address the task that you describe:

If the goal is simply to predict the author among a set of predefined authors, then this is not a clustering task (unsupervised) but a classification task (supervised). This implies that you would split the data between training and test set (or use cross-validation), train a model using the training set then apply it the test set, and finally evaluate the performance on the test set by comparing the predicted author with the true author for every text. Typically the performance is aggregated as f1-score for every author, then macro/micro f1-score.
If the goal is some kind of experiment to see whether a clustering algorithm can correctly identify the groups by author (spoiler: it certainly can't, at least not well), then you're doing unsupervised learning with an external evaluation. I'm not aware of any standard evaluation method for this context (there might be). I assume that you would need a matching method between the clusters ids and the authors (e.g. assign each author to the cluster which has the most instances by this author), then you can apply the regular classification evaluation.

Two additional points:

The general problem of distinguishing texts by their author is called stylometry. There are many variants, such as open or closed classification, author verification, author profiling, etc. It's an active area of NLP research (which means there's a lot of literature about it), and state of the art methods are quite different from regular classification/clustering. It doesn't meant that regular methods don't work at all, but usually not as well as specialized methods.
Features: TF-IDF words vectors are used for tasks where the "topic" of the text matters, for instance to classify articles about sports, politics, etc. There are better options for the features when the task is about writing style:

no focus on rare words (as IDF does), instead rare words should usually be discarded (because they cause massive overfitting)
words are not necessarily the right unit to consider. Character trigrams/4-grams have been proved to work better.
tokenizing can be tricky, it's often considered safer not to tokenize at all so that the author's punctuation habits are preserved as features.


Ensemble models - neural network input both original data and predictions of other models?

I suggest you test it by cross-validation to prevent overfitting. However, I guess because of a strong relationship between this variable and output your work will not be better. Instead, try to use an ensemble.

I read something about deep learning which in that way they use several models as a sequence and use results of previous models as inputs of the next models. Maybe if you continue this work it will be in some way a deep learning.


Text annotating process, quality vs quantity?

Both metods are against the nature of machine/statistical learning. Lack of data will not let you generalize well in any case. It will have different effect though. Generally speaking:

You will be able to classify more test samples, but sometimes will do
it wrong due to errors in data.
You will have precise classification of some test samples, but will have no idea how to classify the rest of them.

So, it's up to your's classification score/quality estimation function which case is better for you.


Definition of Type 1 and Type 2 Errors in Classification

What you said is true. Both articles look contradicting and arrive at opposite Type 1 and Type 2 errors.

I have had the same confusion several times.

Let me try to tell you what i concluded, though i am not 100% sure that my conclusion is right.

Definition of Type 1 error depends on False Positive.
Definition of Type 2 error depends on False Negative.

Above two statements are always correct.

Then why there is confusion?

Confusion arises by the definition of Null Hypothesis and Alternate Hypothesis functions.

In statistics generally Null Hypothesis is the one that is mostly true and everybody accepts and doesn't contain much information and Alternate Hypothesis is the one that unsettles status quo and if true is some big news.

So, in your banking example you can consider

Null Hypothesis = Bank is Normal (Solvent)
Alternate Hypothesis = Bank is Bankrupt

So, if this is the definition, then bankruptcy literature is correct.

But, in general in Machine Learning/Data Science we don't define two hypothesis, we just say our problem definition as predicting if a bank is bankrupt or not. In this scenario, our

True Positive is Bank is Bankrupt and we predicted it as Bankrupt.
False Positive is Bank is Solvent and we predicted it as Bankrupt.
False Negative is Bank is Bankrupt and we predicted it as Solvent.

So, again by the above statements, definition by confusion matrix is also right.

Then, why contradicting errors??
Answer is just the difference in the way we define hypothesis.

In banking literature, they defined true as solvent bank and false as bankrupt bank.
In confusion matrix, we assumed true as bankrupt bank and false as solvent bank.

Have a look at the following for more detailed explanations.

https://www.khanacademy.org/math/ap-statistics/tests-significance-ap/error-probabilities-power/v/introduction-to-type-i-and-type-ii-errors

https://www.abtasty.com/blog/type-1-and-type-2-errors/


What exactly does the model generation mean in this diagram?

A model is a simplified representation of a complex real world object/phenomena. As a simple example, I see a mountain and because I cannot study the exact shape of the mountain I represent it by a model that it is a triangle. Representing the mountain by a triangle is a model. Someone else comes and makes a better model by saying it is a cone. Every model captures some features of the real world phenomena. A model is better if it is able to capture the true features of the object/phenomena.

From a machine learning perspective, this corresponds to exploring the dataset and then thinking about the dataset and creating a model of the dataset. For instance, if I have the famous Titanic dataset, I would be thinking how different features affect the probability of survival. One "model" will be to think that different attributes independently affect the chances of a person's survival. I would be wanting to use a naive bayes classifier to learn the parameters of this model and see how it performs. A different model would be to assume that different attributes probably follow a hierarchy. Maybe, first comes gender. Then among those of the same gender, the deciding factor is probably class and so on. To learn the parameters of this model I would want to train the data with a decision tree classifier or a random forest classifier. This is precisely the process of model generation. It involves thinking about the data by combining your insights from the exploratory data analysis and the domain knowledge you have. You then make certain assumptions. And then test your model.


Effect of a few wrongly scaled feature values on logistic regression model

In my opinion, the model will still be valid because even if the range of the predictor variable is high in the case of Logistic Regression, the algorithm itself will improvise itself by reducing the value of the "coefficient" corresponding to this variable.
You can read more about this here and here
Dhanyawad.


How would you improve and optimize a manual encoding like this?

The easiest and most efficient way to do it, is to use the One Hot Encoding ScikitLearn

This is an example:

>>> from sklearn.preprocessing import OneHotEncoder
>>> enc = OneHotEncoder()
>>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]])
OneHotEncoder(categorical_features='all', dtype=<... 'numpy.float64'>,
       handle_unknown='error', n_values='auto', sparse=True)
>>> enc.n_values_
array([2, 3, 4])
>>> enc.feature_indices_
array([0, 2, 5, 9])
>>> enc.transform([[0, 1, 1]]).toarray()
array([[ 1.,  0.,  0.,  1.,  0.,  0.,  1.,  0.,  0.]])


Inverse Relationship Between Precision and Recall

If we decrease the false negative (select more positives), recall always increases, but precision may increase or decrease. Generally, for models better than random, precision and recall have an inverse relationship (@pythinker's answer), but for models worse than random, they have a direct relationship (@kbrose's example).

It is worth noting that we can artificially build a sample that causes a model which is better-than-random on true distribution to perform worse-than-random, so we are assuming that the sample resembles the true distribution.

Recall

We have
$$TP = P - FN$$
therefore, recall would be
$$r = \frac{P-FN}{P} = 1- \frac{FN}{P}$$
which always increases by decrease in $FN$.

Precision

For precision, the relation is not as straightforward. Lets start with two examples.

First case: decrease in precision, by decrease in false negative:

label   model prediction
1       0.8
0       0.2
0       0.2
1       0.2

For threshold $0.5$ (false negative = $\{(1, 0.2)\}$),

$$p = \frac{1}{1+0}=1$$

For threshold $0.0$ (false negative = $\{\}$),

$$p = \frac{2}{2+2}=0.5$$

Second case: increase in precision, by decrease in false negative (the same as @kbrose example):

label   model prediction
0       1.0
1       0.4
0       0.1

For threshold $0.5$ (false negative = $\{(1, 0.4)\}$),

$$p = \frac{0}{0+1}=0$$

For threshold $0.0$ (false negative = $\{\}$),

$$p = \frac{1}{1+2}=0.33$$

It is worth noting that ROC curve for this case is

Analysis of precision based on ROC curve

When we lower the threshold, false negative decreases, and true positive [rate] increases, which is equivalent to moving to the right in ROC plot. I did a simulation for better-than-random, random, and worse-than-random models, and plotted ROC, recall, and precision:

As you can see, by moving to the right, for better-than-random model, precision decreases, for random model, precision has substantial fluctuations, and for worse-than-random model precision increases. And there are slight fluctuations in all three cases.  Therefore,

  By increase in recall, if model is better than random, precision generally decreases. If mode is worse than random, precision generally increases.

Here is the code for simulation:

import numpy as np
from sklearn.metrics import roc_curve
from matplotlib import pyplot

np.random.seed(123)
count = 2000
P = int(count * 0.5)
N = count - P
# first half zero, second half one
y_true = np.concatenate((np.zeros((N, 1)), np.ones((P, 1))))

title = 'Better-than-random model'
# title = 'Random model'
# title = 'Worse-than-random model'
if title == 'Better-than-random model':
    # GOOD: model output increases from 0 to 1 with noise
    y_score = np.array([p + np.random.randint(-1000, 1000)/3000
                        for p in np.arange(0, 1, 1.0 / count)]).reshape((-1, 1))
elif title == 'Random model':
    # RANDOM: model output is purely random
    y_score = np.array([np.random.randint(-1000, 1000)/3000
                        for p in np.arange(0, 1, 1.0 / count)]).reshape((-1, 1))
elif title == 'Worse-than-random model':
    # SUB RANDOM: model output decreases from 0 to -1 (worse than random)
    y_score = np.array([-p + np.random.randint(-1000, 1000)/1000
                        for p in np.arange(0, 1, 1.0 / count)]).reshape((-1, 1))

# calculate ROC (fpr, tpr) points
fpr, tpr, thresholds = roc_curve(y_true, y_score)
# calculate recall, precision, and accuracy for corresponding thresholds
# recall = TP / P
recall = np.array([np.sum(y_true[y_score > t])/P
                   for t in thresholds]).reshape((-1, 1))
# precision = TP / (TP + FP)
precision = np.array([np.sum(y_true[y_score > t])/np.count_nonzero(y_score > t)
                      for t in thresholds]).reshape((-1, 1))
# accuracy = (TP + TN) / (P + N)
accuracy = np.array([(np.sum(y_true[y_score > t]) + np.sum(1 - y_true[y_score < t]))
                     /len(y_score)
                      for t in thresholds]).reshape((-1, 1))

# Sort performance measures from min tpr to max tpr
index = np.argsort(tpr)
tpr_sorted = tpr[index]
recall_sorted = recall[index]
precision_sorted = precision[index]
accuracy_sorted = accuracy[index]

# visualize
fig, ax = pyplot.subplots(3, 1)
fig.suptitle(title, fontsize=12)

line = np.arange(0, len(thresholds))/len(thresholds)
ax[0].plot(fpr, tpr, label='ROC', color='purple')
ax[0].plot(line, line, '--', label='random', color='black')
ax[0].set_xlabel('fpr')
ax[0].legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax[1].plot(line, recall, label='recall', color='blue')
ax[1].plot(line, precision, label='precision', color='red')
ax[1].plot(line, accuracy, label='accuracy', color='black')
ax[1].set_xlabel('1 - threshold')
ax[1].legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax[2].plot(tpr_sorted, recall_sorted, label='recall', color='blue')
ax[2].plot(tpr_sorted, precision_sorted, label='precision', color='red')
ax[2].plot(tpr_sorted, accuracy_sorted, label='accuracy', color='black')
ax[2].set_xlabel('tpr (1 - fnr)')
ax[2].legend(loc='center left', bbox_to_anchor=(1, 0.5))

fig.tight_layout()
fig.subplots_adjust(top=0.88)
pyplot.show()


How to set parameters to search in scikit-learn GridSearchCV

There is a typo in pipe, no_estimators should be n_estimators.

To address your problem, if you run the following piece of code:

for param in rnd_search.get_params().keys():
    print(param)

This will show you how the parameters are passed to different parts of the pipeline, the parameters of interest are:

bag__base_estimator__svc__kernel
bag__base_estimator__svc__decision_function_shape

So you were almost there, you were just missing base_estimator__ in the svc pipeline parameters. All you need to do is change the svc parameters like so:

params = {
    'bag__bootstrap_features' : [True, False],
    'bag__base_estimator__svc__kernel': ['linear', 'rbf'],
    'bag__base_estimator__svc__decision_function_shape': ['ovo', 'ovr']
}


Keras, DNN ending with sigmoid - model.predict produces values < 0.5. This indicates...?

I think the dropout is a bit high and if it's a binary classification, then why in the end a single node?

Make sure your target variable is having proper shape in case of softmax...(one hot/ to_categorical())

def build_model():
    model = models.Sequential()
    model.add(layers.Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.001),
                           input_shape=(train_data.shape[1],)))
    model.add(layers.Dropout(0.3))
    model.add(layers.Dense(32, kernel_regularizer=regularizers.l2(0.001), activation='relu'))
    model.add(layers.Dropout(0.3))
    model.add(layers.Dense(**num_classes**, activation='softmax'))
    model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

    return model

To improve it further, you need to use some techniques, such as cross-validation, batch normalization, and increase the epochs(maybe).


what's this approach to spatiotemporal data named as?

I would call that a multi-headed convolutional model.


This library called Metrics provides most of metrics for Machine Learning including MAP for Recommendation systems. If you only interested in metrics for recommendation systems, perhaps you can see this library.


Why is the curve for alternative hypothesis shifted compared to that of null hypothesis?

The idea is that the population has the mean of the null hypotesis, say mean height of Sweden is 181cm. Then you do a sample of the population and find that the sample height is 185. Is this enough for you to discard the null and accept the alternative?

This is what the curves are trying to illustrate. The reason for the shifted curve for the alternative is that the mean is located at a different point on the x-axis (185 in my example).

So, given that h0 is true (mean 181), what are the probability to draw a sample mean of 185? That would be the right tail of the null curve straight under the alternatives mean and all to the right of it.


R - newdata has X rows but variables have X rows

Cut down your problem to a minimal reproducible example and supply data or generate it. For your problem I can reproduce with a data frame with two columns (we don't care if its 99 or 2 columns):

training = data.frame(response=runif(858), balloons=runif(858))
testing = data.frame(response=runif(284), balloons=runif(284))

Now you are fitting the model thus, and getting this error:

lm1 <-lm(training$response ~ training$balloons, data=training)
lm.predict <-predict(lm1, newdata=testing)
## Warning message:
## 'newdata' had 284 rows but variables found have 858 rows

Your problem is because you are specifying vectors in the model (ie columns of training using the $ column notation) and also specifying a data frame in the data argument (which is ignored). You get the same error if you leave it out:

> lm1 <-lm(training$response ~ training$balloons)
> lm.predict <-predict(lm1, newdata=testing)
Warning message:
'newdata' had 284 rows but variables found have 858 rows

If your observations are coming from a data frame then you don't need to specify the data as vectors, you give a formula which is evaluated in the context of the data frame:

> lm1 <-lm(response ~ balloons, data=training)

and then:

> lm.predict <-predict(lm1, newdata=testing)

which is 284 predictions as expected.


Using time series data from a sensor for ML

You have time series data which is used to measure the acceleration. You which to identify when the machine is in its nominal state (OFF) and anomalous state (ON). This problem would be best solved using anomaly detection algorithms. But, there are so many ways that you can approach this problem.

Preparing you data

All of the methods will rely on the feature extraction method you select. Assuming we continue to use the 3 sample time window as you suggested. In this algorithm you will calculate a statistic for this nominal state $y = 0$. I would suggest the mean as I assume you are already doing, take the average of the three sample resultant accelerations. You will then be left with a large number of values in a training set $S$ defined as

$S = \{s_0, s_1, ..., s_n \}$

where $s$ is the mean of the tree samples in a window. $s$ is defined as

$s_i = \frac{1}{3} \sum_{k=i-2}^{i} x_k$

where $x$ is your sample observations and $i\geq2$.

Then collect more data if it is possible with the machine active such that $y = 1$.

Now you can choose if you want to train your algorithm on a one-class dataset (pure anomlay detection). A biased dataset (anomaly detection) or a well-balanced dataset. The balance of the dataset is the ratio between the two classes in your dataset. A perfect dataset for a 2-class classifier would be 1:1. 50% of the data belonging to each class. You seem to have a biased dataset, assuming you don't want to waste a lot of electricity.

Do note that there is nothing stopping you from keeping the neighboring samples split as an instance in your dataset. For example:

$x_i$ $x_{i-1}$ $x_{i-2}$ | $y_i$

This would make a 3-dimensional input space for a specific output which is defined for the currently taken sample.

A Biased Dataset

Easy Solution

The easiest way that i would suggest. Assume you are using a single statistic to define what is happening throughout the 3 sample window. From the collected data get the maximum $s$ of your nominal points ($y=0$) and the minimum $s$ of your anomalous points ($y=1$). Then take the halfway mark between these two and use that as your threshold.

If a new test sample $\hat{s}$ is larger than the threshold then assign $y=1$.

You can extend this by calculating the mean $s$ for all of your nominal samples $y=0$. Then calculate the mean for your anomalous samples $y=1$. If a new sample falls closer to the mean of the anomalous samples then classify it as $y=1$.

But I want to get fancy!

There are a number of other techniques you can use to do this exact task.

k-Nearest Neighbors
Neural Networks
Linear Regression
SVM

Simply put, almost every machine learning algorithm is well suited for this purpose. It just depends on how much data is available to you and it's distribution.

I really want to use SVM

If this is the case keep the three samples completely separated. Your training matrix will have 3 columns as discussed above. And then you will have your outputs $y$. Using SVM in python is very easy: http://scikit-learn.org/stable/modules/svm.html.

from sklearn import svm

X = [[0, 0, 0], [1, 1, 1], ..., [1, 0, 1]]
y = [0, 1, ..., 1]
clf = svm.SVC()
clf.fit(X, y)

This trains your model. Then you will want to predict the outcome for a new sample.

clf.predict([[2., 2., 1]])


Outlier detection by unsupervised algorithm: Fraud Detection

First of all, start with a subset until you know what you are doing.
There is no use in waiting for hours for a result that doesn't work, or to run out of memory, or to optimize, just to find out it does not work.

Secondly, makes sure your preprocessing is very very well done. Bad preprocessing will hurt your algorithms.

From my experience, one-class SVM does not work well. It assumes all your training data is 'normal' class (no outliers), and this a representative sample of all normal data. Secondly,.you need to tune kernel parameters, but how.can you do so without labeled anomalies?

Instead, I'd try knn outlier detection, LOF and LoOP.

But for these you need to make sure your distance is a very good measure of similarity. If you don't preprocess well, distance does not work, and then nearest-neighbor methods don't work either.


Outlier Detection using K-Means using one column

If you have a one dim data, why do you need to use K-means?

In such a case, to detect the outlier I would recommend creating a simple histogram and then based on its shape you can visually find the outliers. To get a proper outlier threshold you can use np.quantile() function.


Is it always possible for validation accuracy to be as high as training accuracy?

Yes, it is possible to have a situation in which validation accuracy cannot be as high as training accuracy. Any situation in which noise (as opposed to generalizable properties of the feature set) in the training set is more predictive of the target variable within the training set would produce this.

Consider a situation in which a random property of the training sample considered is perfectly predictive, but this is found not to be true of all other examples outside the sample. The predictive power in the training set would be perfect, but outside the training set, less than perfect. This phenomenon is broadly referred to as "overfitting".

For example, let's consider a case where you have a set of fruit data and you're trying to establish whether a given fruit is an orange or a tangerine. You have 4 features- the circumference of the fruit around the stem-medial axis, the height of the fruit across the stem, a numerical value of the hue of the fruit skin, and the first letter of the amateur baseball team whose home park is closest to the field the fruit was grown in. Let's imagine that by some baffling coincidence, the baseball team letter in the training set was perfectly predictive of whether the fruit would be a tangerine or orange.

We can imagine that this would not hold true across the country or the world, which would produce a situation where the training set accuracy would be perfect, but the validation set accuracy would not be able to approach that using the same methods.


Random forest confusion matrix encountered invalid values

Most probably, sum across one of the rows is coming out as zero in this code
matrix = matrix.astype('float') / matrix.sum(axis=1)[:, np.newaxis]

It is throwing a runtime warning and only that particular cell will be np.inf. Rest all division will be fine. That's why the plot is showing some data.
 You may see the same in this sample code
import numpy as np
matrix = np.array([[1, 2, 3, 0, -6],[6, 7, 8, -4, 10]]) #Sum zero for 1st row

matrix.astype('float') / matrix.sum(axis=1)[:, np.newaxis]

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in true_divide after removing the cwd from sys.path.
array([[        inf,         inf,         inf,         nan,        -inf],
       [ 0.22222222,  0.25925926,  0.2962963 , -0.14814815,  0.37037037]])

As a solution, you should add a constant in the denominator.1 will be best
import numpy as np
matrix = np.array([[1, 2, 3, 0, -6],[6, 7, 8, -4, 10]]) #Sum zero for 1st row

matrix.astype('float') / (matrix.sum(axis=1)[:, np.newaxis] + 1)


What are some significance tests to rank features(multiple) before training the data

One of the easiest ways to do this is with a correlation matrix. Variables that are correlated with the value you are predicting will probably be good predictors. Here is a good answer on how to do that in Python:

https://stackoverflow.com/questions/29432629/plot-correlation-matrix-using-pandas

However - be careful. If your variables are correlated with the response variable (the one you are predicting) AND they are correlated with each other it may be best to use just one of those variables instead of both.

The "best case" scenario is that you are predicting "y" which is correlated with "x" and "z" but "x" and "z" are not correlated at all with each other.

Because you are testing for anomalies (a binary response?) you can transform your predictor variables to be between 0 and 1 and then test the correlation between your transformed predictors and with your predictor variable.

If you want a significance test for which variables will be good predictors you will have to build some models. I know that in R the summary(...) function returns the statistical significance of regression model coefficients - there is probably a similar functionality in Python.


How to create dictionaries out of pandas dataframes?

I have one solution to your problem. please be careful with the names of the columns for every step. You may have different ones:

Concat all transposed data frames

z=pd.concat([m_year.T,i_year.T,e_year.T])

Then melt them:

z=z.reset_index().melt(id_vars='index')
print(z.head())

  index      country value
0  1995  Afghanistan   nan
1  1996  Afghanistan   nan
2  1997  Afghanistan   nan
3  1995  Afghanistan   nan
4  1996  Afghanistan   nan

then do this:

z=pd.DataFrame(z.groupby(['country','index'])['value'].apply(lambda x: [i for i in x])).reset_index()

print(z.head)

       country  index            value
0  Afghanistan   1995  [nan, nan, nan]
1  Afghanistan   1996  [nan, nan, nan]
2  Afghanistan   1997  [nan, nan, nan]
3  Afghanistan   1998       [nan, nan]
4  Afghanistan   1999       [nan, nan]

Fianly, a for loop:

final_dict={}
for i in z.country.unique():

final_dict[i]=z[z['country']==i].drop('country',axis=1).set_index('index').T.to_dict()

it will return this:

{'Afghanistan': {1995: {'value': [nan, nan, nan]},
  1996: {'value': [nan, nan, nan]},
  1997: {'value': [nan, nan, nan]},
  1998: {'value': [nan, nan]},
  1999: {'value': [nan, nan]}},
 'Albania': {1995: {'value': [nan, nan, nan]},
  1996: {'value': [nan, nan, nan]},
  1997: {'value': [nan, nan, nan]},
  1998: {'value': [nan, nan]},
  1999: {'value': [nan, nan]}},
 'Angola': {1995: {'value': [5538749000.0, 0.0612, 0.8565]},
  1996: {'value': [7526447000.0, 0.0626, 0.8369]},
  1997: {'value': [7648377000.0, 0.0641, 0.8173]},
  1998: {'value': [0.0655, 0.7976]},
  1999: {'value': [0.067, 0.7777]}}}

And then with list comprehetion we can transform it into your final dictionary :

final={i:{i:j['value'] for i,j in final_dict[i].items()} for i in final_dict.keys()}

final

{'Afghanistan': {1995: [nan, nan, nan],
  1996: [nan, nan, nan],
  1997: [nan, nan, nan],
  1998: [nan, nan],
  1999: [nan, nan]},
 'Albania': {1995: [nan, nan, nan],
  1996: [nan, nan, nan],
  1997: [nan, nan, nan],
  1998: [nan, nan],
  1999: [nan, nan]},
 'Angola': {1995: [5538749000.0, 0.0612, 0.8565],
  1996: [7526447000.0, 0.0626, 0.8369],
  1997: [7648377000.0, 0.0641, 0.8173],
  1998: [0.0655, 0.7976],
  1999: [0.067, 0.7777]}}


There are many options. One option could be to find clusters at the first time point and define a change "metric" at the second time point. A possible change metric is if the data points belong to the same cluster at the second time point.
You mention a variation of k-means. Since k-means is an iterative algorithm, this cluster re-assignment is what happens between each training step. The difference with this method is you are allowing the data points to move between training steps and measuring how many cluster assignments change.


Pandas dataframe with multiple hierarchical indices

I got it working with below simple line of code

df.set_index('fruit').stack().groupby(level=[0,1]).value_counts().unstack(level=[1,2]).fillna(0)


Tools to preprocess a big data for dashboards?

We have dashboards that show information about some processes, which have billions of rows in the database. It's not queried directly though, but instead from pre-aggregated data.

We have automated scripts running in database that populate the aggregated data tables specifically for dashboards. In most extreme cases the raw data is so large and comes in at such a high pace that there's a two-tier aggregation in place. First tier will de-normalize the data and 2nd tier will do the actual sum/count type of aggregation.

So you don't need two tools since you can do aggregation purely in database.

edit (answering question from comment): Our dashboards are in Tableau. Data is all in PostgreSQL databases. Automation is done with unix cronjob executing a database function. The function in turn queries which tasks it has to run and runs them. There is a few moving pieces to the whole system, but the architecture isn't too complicated. Write down your ideas and go talk to some architects / engineers - they will know the best way to approach this.


How to efficiently iterate a supervised model over the Cartesian product of very large number of records?

This problem is called record linkage and there are methods to avoid iterating the whole cartesian product. The main method I know was called "blocking" and consists in doing a first "rough" pass to create groups of matching candidates (the "blocks"). For example you can create groups which contain at least X n-grams in common. This can be done through one linear iteration through all the entities, storing them in every applicable bin based on their n-grams (an entity can be stored in several bins). I assume that some kind of clustering could also be used to generate the groups of similar entities. Then you end up with multiple groups of smaller size, and you run the cartesian product comparison on every group individually. This can greatly reduce the complexity.

note: I was working on this maybe 10 years ago so there might be more recent approaches.


Type of images used to train a neural network

You should read this post from the same author. It is about how to build a learning dataset with Google's search, some javascript and some python.

The idea there is to collect as many pictures of the object as possible (object = banana) and then let the deep learning network to compare and learn to differentiate them from bulk random images from some free database.

I suppose background or partial image variations are all ok. You cannot know which part of the object is visible when the whole bunch is evaluated.

Data needs to be fit to same size, at least Adobe has some tool for that.


How

Adding a dropout layer to a neural network has a few functions. First of all, here is some simple code that implements it:

In [1]: import numpy as np

In [2]: weights = np.random.randint(0, 100, (8, 8))    # example weights

In [3]: weights
Out[3]:
array([[10, 48,  9, 87, 23, 18, 60, 83],
       [29, 48, 88, 43, 34, 56, 52, 82],
       [35,  0, 70,  3,  8, 88,  6, 15],
       [39, 16, 59, 91, 33, 13, 53, 73],
       [20, 56, 70, 35, 16, 12, 80,  6],
       [21, 17, 23, 21, 72, 93, 58, 56],
       [26, 86, 38, 90, 91, 87, 65,  0],
       [67,  6, 10, 94, 19, 25, 49, 61]])

In [4]: dropout_mask = np.random.randint(0, 2, (8, 8))    # either 0 or 1

In [5]: dropout_mask
Out[5]:
array([[1, 1, 0, 0, 0, 1, 0, 1],
       [1, 0, 1, 1, 0, 0, 0, 0],
       [1, 0, 0, 1, 0, 1, 1, 0],
       [0, 0, 0, 0, 1, 0, 1, 0],
       [0, 1, 0, 0, 0, 0, 1, 0],
       [0, 1, 1, 1, 0, 0, 0, 1],
       [0, 1, 1, 0, 1, 1, 1, 1],
       [1, 0, 0, 1, 0, 1, 1, 1]])

        # we make a copy here just to use helpful variable names
In [6]: masked_weights = weights.copy()

        # where the mask is "on" (==1), we set those weights to 0
In [7]: masked_weights[dropout_mask == 1] = 0

In [8]: masked_weights
Out[8]:
array([[ 0,  0,  9, 87, 23,  0, 60,  0],
       [ 0, 48,  0,  0, 34, 56, 52, 82],
       [ 0,  0, 70,  0,  8,  0,  0, 15],
       [39, 16, 59, 91,  0, 13,  0, 73],
       [20,  0, 70, 35, 16, 12,  0,  6],
       [21,  0,  0,  0, 72, 93, 58,  0],
       [26,  0,  0, 90,  0,  0,  0,  0],
       [ 0,  6, 10,  0, 19,  0,  0,  0]])

So we have selected roughly half of weights and turn them off for a pass through the network.

EDIT:

Droput randomly drops neurons on each pass in training, as shown above, but during test time (a.k.a. inference), the dropout layers are deactivated by default. This means that all neurons are available and are used. There is still, however, the issue of scaling - so while all neurons are active during inference, their outputs are scaled to reflect the layers overall output such that the expected overall sum remains unchanged. This can be performed a few different ways - see comments on this answer for related links.

There are of course nice implementations of dropout in all deep learning frameworks (PyTorch, Tensorflow, CNTK, and so on). As is discussed and shown here.

Intuition

One intuitive explanation, is that is ensures each neuron increases its robustness and manages to contribute to the network in its own right. What I mean by this, is that dropout prevent chain or clusters of neurons being strongly dependent on one another. It helps focus the neurons to be able to deliver some useful information in as many different scenarios as possible, e.g. when it only receives signal from a subset of the neurons in the preceding layer.

It can be helpful to think of dropout in terms of regularisation. We add regularisation terms to models in order to penalise them again certain behaviour. An example would be in linear regression via the Bayesian Information Criterion, whereby we increase a model's error if it uses more and more covariates. We also choose the form of error/regularisation when computing values in backpropagation e.g. we select the sum of squared errors (the $L_2$ loss), or just the raw error (the $L_1$ loss). Dropout, on the other hand, applies a change the network itself! We are constricting it by simply removing access to a random selection of neurons during one single pass through the network.

Maths

There actually is not a lot of maths involved, with the original paper explaining the concept in no more than four lines:

The magic really happens in that second line, with the element-wise multiplication of a Bernoulli distributed random variable $\textbf{r}$ and the vector outputs of the preceding layer, $\textbf{y}$. This performs exactly the same operation as IN [7] in my code example above.

I believe the reason is that it was inspired by the intuitive idea, easy to implement and finally proven to work exceptionally well in practice. The above equations can then be applied to various problems - I would suggest reading the paper for some examples.

For further understanding of dropout (and everything around neural network concepts!), I heartily recommend reading Michael Nielsen's book. Exaplantions of dropout with excellently helpful diagrams can be found in Chapter 3.


(SOLVED) Power BI, Page level filter not working with many to one relation

Posting this as an answer.

Turns out the relations were set to Cross filter direction: Single on all relations (and the one from Month table.Month to A.Month was not set to active).

Setting Cross filter direction: Both fixed the problem.


How do I know how to construct the layers of my CNN

There isn't a formula behind it - layer sizes are selected with a lot of trial and error (often automated by cross-validation). But saying "I've used an architecture from a network from a similar problem, or at least from the same domain" is a legitimate answer.


Clustering on categories is not something sklearn can do by default. And assigning sequential values to categories like that certainly won't help - clustering tends to work based on distance, by assigning 0, 1, 2 to Yes, No, Don't Know like that, you are suggesting Yes is 'closer' to No than it is to Don't Know.

I highly recommend having a look at k-modes, a clustering algorithm for categorical data. Essentially it optimises according to how common the set of categories are within the cluster (modes, rather than the means of values). E.g. may find green people carriers (5/6/7 pax) in a cluster, and red sports cars (2/4 pax).

There is a Python library here, which also has links to some papers describing the algorithm. There is also k-prototypes, which clusters with combined numerical and catagorical data.


How can I get prediction for only one instance in Keras?

You can do:

q = model.predict( np.array( [single_x_test,] )  )


Will unnecessary features harm the tree based model?

This is not a direct answer to your question, but more like an experiment. I created a simple script in Python where I ran multiple times the Iris dataset with the regular columns and also with 4 extra columns with random numbers. Then I store the difference of the accuracy between the two models and plot the distribution of it. If you try it yourself as well, you will see that there are cases that the "clean" dataset has better accuracy, even if the majority of the results are exactly the same.

from sklearn.datasets import load_iris
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
import numpy as np
import warnings
import seaborn as sns
import random as random

warnings.filterwarnings("ignore")

iris = load_iris()

df = pd.DataFrame(data= np.c_[iris['data'], iris['target']], columns= iris['feature_names'] + ['target'])

df['rand1'] = np.random.randint(0, 2, df.shape[0])
df['rand2'] = np.random.randint(0, 2, df.shape[0])
df['rand3'] = np.random.randint(0, 2, df.shape[0])
df['rand4'] = np.random.randint(0, 2, df.shape[0])

all_inputs = df[iris['feature_names']].values
all_inputs_with_random = df[iris['feature_names']+['rand1', 'rand2', 'rand3','rand4']].values
all_classes = df['target'].values

dif = []

for i in range(100):
    a = random.randint(0,1000)

    (train_inputs, test_inputs, train_classes, test_classes) = train_test_split(all_inputs, all_classes, train_size=0.7, random_state = a)

    dtc1 = DecisionTreeClassifier()
    dtc1.fit(train_inputs, train_classes)

    a1 = dtc1.score(test_inputs, test_classes)

    (train_inputs, test_inputs, train_classes, test_classes) = train_test_split(all_inputs_with_random, all_classes, train_size=0.7, random_state = a)

    dtc2 = DecisionTreeClassifier()
    dtc2.fit(train_inputs, train_classes)

    a2 = dtc2.score(test_inputs, test_classes)

    dif.append(a1-a2)

sns.distplot(dif)