Weighted Linear Combination of Classifiers

I don't know how to fix your automatic differentiation, but I can show you what I did (and I have seen others do too) when I wanted to achieve the same thing. You can fit a linear meta-classifier on the outputs of your classifiers that you want to ensemble. Here is the implementation from my scikit toolbox:

'''
-------------------------------------------------------
    Stack Classifier - extrakit-learn

    Author: Simon Larsson <larssonsimon0@gmail.com>

    License: MIT
-------------------------------------------------------
'''

from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.utils.validation import check_X_y, check_array, check_is_fitted
import numpy as np

class StackClassifier(BaseEstimator, ClassifierMixin):
    ''' Stack Classifier

    Ensemble classifier that uses one meta classifiers and several sub-classifiers.
    The sub-classifiers give their output to to the meta classifier which will use
    them as input features.

    Parameters
    ----------
    clfs : Classifiers who's output will assist the meta_clf, list classifier

    meta_clf : Ensemble classifier that makes the final output, classifier

    drop_first : Drop first class probability to avoid multi-collinearity, bool

    keep_features : If original input features should be used by meta_clf, bool

    refit : If sub-classifiers should be refit, bool
    '''

    def __init__(self, clfs, meta_clf, drop_first=True, keep_features=False, refit=True):
        self.clfs = clfs
        self.meta_clf = meta_clf
        self.drop_first = drop_first
        self.keep_features = keep_features
        self.refit = refit

    def fit(self, X, y):
        ''' Fitting of the classifier

        Parameters
        ----------
        X : array-like, shape (n_samples, n_features)
            The training input samples.

        y : array-like, shape (n_samples,)
            The target values. An array of int.

        Returns
        -------
        self : object
            Returns self.
        '''

        X, y = check_X_y(X, y, accept_sparse=True)

        # Refit of classifier ensemble
        if self.refit:
            for clf in self.clfs:
                clf.fit(X, y)

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        # Fit meta classifer, Stack the ensemble
        self.meta_clf.fit(X_meta, y)

        # set attributes
        self.n_features_ = X.shape[1]
        self.n_meta_features_ = X_meta.shape[1]
        self.n_clfs_ = len(self.clfs)

        return self

    def predict_proba(self, X):
        ''' Probability prediction

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape (n_samples, n_features)
            The prediction input samples.

        Returns
        -------
        y : ndarray, shape (n_samples,)
            Returns an array of probabilities, floats.
        '''

        X = check_array(X, accept_sparse=True)
        check_is_fitted(self, 'n_features_')

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        return self.meta_clf.predict_proba(X_meta)

    def predict(self, X):
        ''' Classification

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape (n_samples, n_features)
            The prediction input samples.

        Returns
        -------
        y : ndarray, shape (n_samples,)
            Returns an array of classifications, bools.
        '''

        X = check_array(X, accept_sparse=True)
        check_is_fitted(self, 'n_features_')

        # Build new tier-2 features
        X_meta = build_meta_X(self.clfs, X, self.keep_features)

        return self.meta_clf.predict(X_meta)

def build_meta_X(clfs, X=None, drop_first=True, keep_features=False):
    ''' Build features that includes outputs of the sub-classifiers

    Parameters
    ----------
    clfs : Classifiers that who's output will assist the meta_clf, list classifier

    X : {array-like, sparse matrix}, shape (n_samples, n_features)
        The prediction input samples.

    drop_first : Drop first proba to avoid multi-collinearity, bool

    keep_features : If original input features should be used by meta_clf, bool

    Returns
    -------
    X_meta : {array-like, sparse matrix}, shape (n_samples, n_features + n_clfs*classes)
                 The prediction input samples for the meta clf.
    '''

    if keep_features:
        X_meta = X
    else:
        X_meta = None

    for clf in clfs:

        if X_meta is None:
            if drop_first:
                X_meta = clf.predict_proba(X)
            else:
                X_meta = clf.predict_proba(X)[:, 1:]
        else:
            if drop_first:
                y_ = clf.predict_proba(X)
            else:
                y_ = clf.predict_proba(X)[:, 1:]
            X_meta = np.hstack([X_meta, y_])

    return X_meta

This would allow you to use any meta-classifier, but with linear models like ridge/lasso/logistic regression it will acts as learned linear weights of your ensemble classifiers. Like this:

from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from xklearn.models import StackClassifier

X, y = make_classification(n_classes=2, n_features=4, n_samples=1000)

meta_clf = LogisticRegression(solver='lbfgs')
ensemble = [DecisionTreeClassifier(max_depth=1),
            DecisionTreeClassifier(max_depth=5),
            DecisionTreeClassifier(max_depth=10)]

stack_clf = StackClassifier(clfs=ensemble, meta_clf=meta_clf)
stack_clf.fit(X, y)

print('Weights:', stack_clf.meta_clf.coef_[0],' Bias: ', stack_clf.meta_clf.intercept_)

output:

Weights: [0.50017775 2.2626092  6.30510687]  Bias:  [-4.82988374]


Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you.

  Why do so many functions used in data science have derivatives of the
  form f(x)*(1-f(x))?

If you consider the following differential equation

$y' = y \cdot (1-y)$

you will find the general solution in the form

$y(x) = \frac{e^x}{c + e^x} = \frac{1}{1 + c e^{-x}}$

So, in some sense, there are not that many functions with this property: they are "very like" sigmoid function.


custom loss in keras, problem with batch size

One approach could be:

class CustomLoss():
        def __init__(self, steps_per_epoch):
            super().__init__()
            self.steps_per_epoch = steps_per_epoch
            self.step = 0

        def calc_custom_loss(self, y_true, y_pred):
            y_true1 = get_y_true1(self.step)
            y_true2 = get_y_true2(self.step)

            self.step += 1
            self.step %= self.steps_per_epoch

This way you only need to provide get_y_true1() and get_y_true2() functions which receive step index. That index is used to generate appropriate batch.

To use it in code:

# Assuming there are 100 batches in one epoch ...
custom_loss = CustomLoss(steps_per_epoch=100)
model.compile(loss=custom_loss.calc_custom_loss, ...)

Be carefull if you also provide validation data during training as it might change your self.step value ... If you want to use validation data, you will need to handle the step appropriately.


Shouldn't L2 regularization be normalized for the number of nodes in a layer?

However, I don't see anybody doing this so I suspect I'm missing something. What is it?

You are concentrating on how this extra loss term affects the absolute value of the loss function. This is not really relevant. Many forms of regularisation are not expressed as changes to loss function at all - framing L2 regularisation as part of the loss function is convenient because it allows us to re-use the existing weight adjustment logic. However, functionally it is almost identical to a weight decay term applied per batch independently of loss function, e.g. multiply all weights by 0.99 every batch.

If you use L2 regularisation loss, then you will almost certainly want to track a separate metric such as mean square error (i.e. your original loss function before considering regularisation), when comparing test results between different levels of regularisation.

That said, there is no reason that you cannot have a different L2 weight param per layer, and many frameworks will support this. Sometimes this could result in better generalisation. It is not done much in practice because it adds yet more dimensions to search when optimising hyper-parameters to a problem.

There is no reason expect better performance by aiming for roughly equal loss values from regularisation per layer. However, if that is your goal, then a naive scaling by inverse layer size $\frac{1}{N_{in} \times N_{out}}$ would not do it. That is because the weights in a trained network will already tend to scale due to the training targets so that the mean squared value is proportional to $\frac{1}{N_{in} + N_{out}}$. So your actual scaling factor to make the L2 regularisation loss per layer roughly the same would be $\frac{N_{in} + N_{out}}{N_{in} \times N_{out}}$. In code this might look like:

L2 = lambda * (
  sum((w1) ^2)*(n_in+n_1)/(n_in*n_1) + sum((w2) ^2)*(n_1+n_2)/(n_1*n_2)
)

(and of course matching changes to loss gradients). Although I will stress again that I do not expect this change to result in better generalisation.


Calculating similarity where order matters

Not sure if this exists already, but maybe you could build a new similarity metric (let's call it $js$) with a few requirements:

If lists $A$ and $B$ are exactly the same, then $js(A, B) = 1.0$
If lists $A$ and $B$ are completely different (no items in common), then $js(A, B) = 0.0$
If all elements match but are shuffled, then $js(A, B) = 0.5$
If $|A| \neq |B|$, then the similarity metric should consider the items of the longest list (e.g. if there are few extra ingredients in $A$, but $B$ is identical otherwise, it will still have some penalization)

Then you can build a method that basically does the following (e.g. in python):

def js(A, B):
    cummulative_score = 0
    longest_list = B
    shortest_list = A
    if len(A) > len(B):
        longest_list = A
        shortest_list = B

    for index, element in enumerate(longest_list):
        if index < len(shortest_list):
            if element == shortest_list[index]:
                cummulative_score += 1
            if element != shortest_list[index]:
                if element in shortest_list:
                    cummulative_score += 0.5
    return cummulative_score / len(longest_list)

Then you would have something like this:

In [10]: print(l)
['Apples', 'Enriched Unbleached Flour', 'Palm Oil', 'High Fructose Corn Syrup', 'Salt', 'Eggs', 'Spices']

In [11]: print(l1)
['Apples', 'Enriched Unbleached Flour', 'vegetable oil', 'sugar', 'Salt', 'Eggs', 'Spices']

In [12]: print(l2)
['Spices', 'Eggs', 'Salt', 'High Fructose Corn Syrup', 'Palm Oil', 'Enriched Unbleached Flour', 'Apples']

In [13]: print(l3)
['Enriched Unbleached Flour', 'Palm Oil', 'High Fructose Corn Syrup', 'Salt', 'Eggs', 'Spices', 'Apples']

In [14]: print(l4)
['Apples', 'Enriched Unbleached Flour', 'Palm Oil', 'High Fructose Corn Syrup', 'Salt', 'Eggs', 'Spices', 'Ice Cream']

In [15]: js(l, l1)
Out[15]: 0.7142857142857143

In [16]: js(l, l2)
Out[16]: 0.5714285714285714

In [17]: js(l, l3)
Out[17]: 0.5

In [18]: js(l, l4)
Out[18]: 0.875

In [19]: js(l, ['Nutella'])
Out[19]: 0.0


asymmetric cost function for deep neural network binary classifier

Rather than 0 use 0.1 or some other small non-zero value. Use the true y as an indicator function, multiply your class probability by your reward component, and sum terms for all possibilities to get a usable loss, like this:

$Loss(y, y_p) = (1-y)(1)(y_p) + (y)(-0.7)(y_p) + (0.1)(1-y_p)$

Where $y_p =P(\hat{y}=1|X)$


Which direction of the normal vector that defines a Perceptron hyperplane is default?

This question is for general hyperplane.

The normal direction is the direction that is perpendicular (also known as normal) to the plane.

Take a piece of paper (treat it as a plane), hold a pencil perpendicular to the paper. The pencil is parallel to the normal direction.

There is no such thing as the default direction as you see, a piece of paper that is oriented in a different manner would have a different normal direction.


ML Algorithm for anomaly detection in paired time-series

Before think which algorithm, think what features. You may add/ignore features that imply your problem fluently.

This will be around quantifying anomalies (new variable?). Try to add a residual variable? Or simply feature different between red and green variables?


A genetic algorithm is an algorithm based on biological evolution, on how nature evolved. It does exactly that, evolve the algorithm so that "it" finds the best solution to the problem at hand. You can use a genetic algorithm to find a solution to a problem which you don't know the answer, you know the answer but want to know a different one, or you are just lazy. The common steps for a genetic algorithm are:

Generate a random population of elements
Evaluate fitness of each element (how good are they against the
solution)
Take the best elements for the next generation
Generate child elements using the above elements
Mutate (randomly) each child. This can also be done before step 4 with parents.
Repeat from step 2 for n number of generations until the solution is found, the maximum number of generations have been reached or the fitness is not changing anymore (local minima).

Two basic problems with GAs:

To get trapped in a local minima (or local maximum depending on the point of view). This means that you might find a good answer (or not even a good one) but it will never reach the best answer because fitness values are not changing or not getting better. This is fought using Mutation in step 5 to keep diversity in the population so it doesn't get stuck.
They are generally slower than other approaches.

If you can, take a look at this book. It is not free, but worth a look.
Alternatively, take a look at this online book for free, it is a great source and he has his own youtube channel. It's more basic than the other book I recommend, but will help you get started with GAs.

To answer the other part of your question, the GA should be used to find a model, but will not act as a model. For example, if you have a neural network you can train it using the backpropagation method, but you could also train it using a GA. The GA will not use anything of the backpropagation mathematics, it could be used to generate weights in its neurons, and evaluate the answer (last layer). Weights will evolve to get closer to the solution. In this scenario, the model will still be the NN, but you used a different algorithm to find the best NN.

Hope this helps.


Are there any differences between Recurrent Neural Networks and Residual Neural Networks?

Definitely they are different. Very deep nets have exploding/vanishing gradient problem. The authors of ResNet paper had seen that by stacking many layers of convolution and dense layers, the learning did not increase although they used ReLU activation and batch normalization. They used a concept named skip connection which helped the nets to learn whether the input to a typical layer should be preserved or it should be transformed by that layer. Using this concept allowed them to increase the number of layers without hesitating whether they would have vanishing/exploding gradients. The concept of residual nets was originally this. The paper uses this concept for spatial data but recently I've seen people debating using them in temporal cases too __time series data.

Recurrent nets are used in temporal domains. Tasks like sequence classification are examples of their usage. In this domain the net should know the information of previous seen data. Well known examples of these nets are LSTMS. Early recurrent nets had vanishing/exploding gradient problem too. but after years LSTMs get popular amongst deep-learning practitioners. They defined a concept named gates which could learn when to forget and when to keep the previous data.


SSE in classification is proportional to the Brier score:

$$
SSE = \sum_n \sum_k (\hat y_{nk} - y_{nk})^2 \\
Brier = \frac{SSE}{N}
$$

For observations indexed by $n$ and classes indexed by $k$.

In your case, you obtained a Brier score of 0.18.

The Brier score can be decomposed into

$$
Brier = Reliability - Resolution + Uncertainty
$$


This is known as "3D Bin Packing Problem" in literature.

https://en.wikipedia.org/wiki/Bin_packing_problem

Since this is NP-Hard; Some of the approaches  are :

Heuristics : https://www.researchgate.net/publication/226249396_A_New_Heuristic_Algorithm_for_the_3D_Bin_Packing_Problem
Deep Reinforcement Learning : https://arxiv.org/abs/1708.05930
Ensemble : https://medium.com/@alitech_2017/alibabas-ai-solution-for-the-3d-bin-packing-problem-3ce66d730ecc


Reflective padding as pure keras verision

Here a version for reflective padding as pure K function, which should (but not tested) work with every backend:

def reflection_padding(inp, paddings):
    paddings = [(x, x) if isinstance(x, int) else x for x in paddings]
    ishape = inp.shape.dims
    ndims = inp.shape.ndims
    if len(ishape) != len(paddings):
        raise ValueError("Padding dims != input dims")
    last = inp
    _all_slice = slice(None, None, None)

    def _get_slices(ndims, axis, slice_):
        ret = [_all_slice for _ in range(ndims)]
        ret[axis] = slice_
        return tuple(ret)

    for axis, pads in ((i, x) for i, x in enumerate(paddings) if x[0]+x[1] != 0):
        pad_data = []
        if pads[0]:
            pre = last[_get_slices(ndims, axis, slice(pads[0], 0, -1))]
            pad_data.append(pre)
        pad_data.append(last)
        if pads[1]:
            post = last[_get_slices(ndims, axis, slice(-2, -pads[1]-2, -1))]
            pad_data.append(post)
        last = K.concatenate(pad_data, axis)
        ishape = last.shape.dims
    return last

# USAGE: reflection_padding(image_batch, [0, [2,2], [2,2], 0])

I am going to accept my own answer. If somebody has a better answer i'll gladly switch that over to theirs


Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook

Spark itself does not use GPUs at all, so this is not surprising.

The operations it performs are at best L3 BLAS ops of moderate size, and most are small L1 operations, so generally a GPU isn't a win. It does use BLAS to accelerate those ops in hardware if a BLAS library like MKL or OpenBLAS is present.


Data prepration for logistic regression : Value either "not available" or a "year"

First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.

Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:

bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $\rightarrow$ (1, 0, 0)
[2000, 2010] $\rightarrow$ (0, 1, 0)
[2010, 2020] $\rightarrow$ (0, 0, 1)

This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.

I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:

renovation     (has renovation, renovation age)
-1             (0, -1)
2010 in 2019   (1, 9)

Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.


The VC dimension is an estimate for the capability of a binary classifier. If you can find a set of $n$ points, so that it can be shattered by the classifier (i.e. classify all possible $2^n$ labelings correctly) and you cannot find any set of $n+1$ points that can be shattered (i.e. for any set of $n+1$ points there is at least one labeling order so that the classifier can not seperate all points correctly), then the VC dimension is $n$.

In your case, first consider two points $x_1$ and $x_2$, such that $x_1 < x_2$. Then there are are $2^2=4$ possible labelings

$x_1:1$, $x_2:1$
$x_1:0$, $x_2:0$
$x_1:1$, $x_2:0$
$x_1:0$, $x_2:1$

All labelings can be achieved through the classifier $h$ by setting the parameters $a<b \in R$ such that

$a<x_1<x_2<b$
$x_1<x_2<a<b$
$a<x_1<b<x_2$
$x_1<a<x_2<b$

respectively. (Actually, $x_1 < x_2$ can be assumed w.l.o.g. but it is enough to find one set that can be shattered.)

Now, consider three arbitrary(!) points $x_1$, $x_2$, $x_3$ and w.l.o.g. assume $x_1<x_2<x_3$, then you can't achieve the labeling (1,0,1). As in case 3 above, the labels $x_1$:1 and $x_2$:0 imply $a<x_1<b<x_2$. Which implies $x_3$ > b and therefore the label of $x_3$ has to be 0. Thus, the classifier cannot shatter any set of three points and therefore the VC dimension is 2.

-

Maybe it becomes clearer with a more useful classifier. Let's consider hyperplanes (i.e. lines in 2D).

It is easy to find a set of three points that can be classified correctly no matter how they are labeled:

For all $2^3=8$ possible labelings we can find a hyperplane that separates them perfectly.

However, we cannot find any set of 4 points so that we could classify all $2^4=16$ possible labelings correctly. Instead of a formal proof, I try to present a visual argument:

Assume for now, that the 4 points form a figure with 4 sides. Then it is impossible to find a hyperplane that can separate the points correctly if we label the opposite corners with the same label:

If they don't form a figure with 4 sides, there are two "boundary cases": The "outer" points must either form a triangle or all form a straight line. In the case of the triangle, it is easy to see that the labeling where the "inner" point (or the point between two corners) is labeled different from the others can't be achieved:

In the case of a line segment, the same idea applies. If the end points are labeled differently than one of the other points, they cannot be separated by a hyperplane.

Since we covered all possible formations of 4 points in 2D, we can conclude that there are no 4 points that can be shattered. Hence, the VC dimension must be 3.


How does neural network solve XOR problem

Notice that the first table (orange line) is performing an OR operation and the second table (blue line) is performing an AND operation. XOR can be defined as (x OR y) AND NOT (x AND y) or $(x \lor y) \land \lnot (x \land y)$, so in other words: orange should fire and blue shouldn't fire. If we now look at this figure from your tutorial:

we can see that the weight of blue in the second layer is negative and small enough (more negative) such that the output can never fire if blue fires, i.e. the output can't fire if both inputs are firing.

$0 + 0 \ngtr 1 : \emptyset $ shouldn't fire

$-2 + 0 \ngtr 1 : (x \land y) = T$ shouldn't fire

$0 + 1.1 \gt 1 : (x \lor y) \land \lnot (x \land y) = T$ should fire

$-2 + 1.1 \ngtr 1 : (x \land y) \land (x \lor y) = T$ shouldn't fire


A Question: is it possible for me to create an AI that learns to play games which I did not create?

As far as I know you might archieve this with Reinforcement Learning. But there are some limitations in this approach: You have to train your algorithm a long time for even learning to press a button. The second limit is that you have to define the possible actions, that the agent might choose. And remember: Every single thing your system has to learn will affect your training time.

You can dive into this topic by searching for the openai gym, which will give you a good overview about what is done right now in order to build AIs which play games. A good book about this topic is Reinforcement Learning - An Introduction by Barto and Sutton.


Is RL applied to animal dispersion a valid approach?

Admittedly I am only a beginner in RL, so I haven't seen much more than MDP's. I believe that this isn't a Markov problem since the states aren't independent (past actions having an impact on the outcome of the current action).

If there are enough hidden variables, then this could be a real problem for you. A policy maps states to actions - and searching for an optimal policy will always map the same action to the same state. If this is some kind of automated scarecrow system, then the animals are likely to become habituated to the "best" action.

Two ways around that:

If the habituation is slow enough, you might get away with treating the environment as having the simple state that you suggest and have an agent which constantly learns and adapts to changes in what will trigger animals to leave. This would be an environment with non-stationary dynamics (the same action in the same state will, over time, drift in terms of its expected reward and next state). For a RL agent, this could just be a matter of sticking with a relatively high exploration rate and learning rate.
If the habituation is fast or adaptive, then you have to include some memory of recent actions used in the state representation. This will make the state space much larger, but is unavoidable. You could try to keep that memory inside the agent - using methods that work with Partially Observable MDPs (POMDPs) - it doesn't really make the problem any easier, but you may prefer that representation.

  The reward is the difference of detections from $s_{-1}$ to $s$.

You need to re-think the reward system if habituation is fast/adaptive, as it will fail to differentiate between good and bad policies. All policies will end up with some distribution of detections, and all policies will have a mean total reward of 0 as a result. Your reward scheme will average zero in the long term for random behaviour, and also will average zero if the number of creatures remains at 200 the whole time, or is zero the whole time. Presumably having zero creatures would be ideal, whilst having 200 would be terrible - you need to be able to differentiate between those two scenarios.

You want to minimise the number of creatures in the area consistently, so a simple reward scheme is just negative of the number of visible animals, per time step. Maybe scale this - e.g. divide by 100, or by some assumed background rate where you would get -1 reward per time step if the number of animals is the same on average as if the agent took no action. You could measure this average using the camera over a few days when the agent is not present or simply not active. It doesn't need to be super accurate, teh scaling is just for convenience - you could think of an agent that gets a mean reward of -0.2 per time step is five times better than having no agent at all, whilst an agent that scores -1.5 per time step might be attracting creatures for all you know!

  That being said, I'm wondering if there's a specific RL algorithm for this setup or if RL even is the right way to go ?

The problem does seem like a good match to RL. There are other ways of searching for good policy functions, such as genetic algorithms, that might also apply. However, you will still need to equip the agent with either continuous learning so it can re-find the best action as it changes, or with a memory of recent actions, depending on the speed of habituation. You may even need both as smart animals like birds or mammals can adapt in long and short term in different ways.


Deploying models on bigdata platforms like Hadoop and Spark

1 Since dataset is huge I can make use any disributed platform for faster computaion and model creation right?

Yes, that is what distributed platforms are for.

  2 Once the model is ready, then there is no need of these disributed
  platform right ? or are they needed for feature extractions ?

Yes for data extraction, feature generation for scoring and scoring your dataset (the actual reason you trained at model for!), you would still need the platform as that data is also likely to be huge.


Which other algorithms fit residuals like XGBoost?

Training of XGBoost is based on a boosting model, which is a general ensemble method creating a strong model from a number of weak models. This process is performed by building a model from the training dataset, then, creating a second model that attempts to correct the errors from the first model. Models are added until the training set is predicted perfectly or a maximum number of models are added.
All machine learning models based on boosting almost follow the above procedure. For example, AdaBoost  is one of the boosting algorithms developed for binary classification. Methods like LightGBM and Catboost use this algorithm.
Unlike the boosting algorithm, in the bagging algorithm, the models are independent and each model is directly fitted to a subset of the original training data.


Why we need to have the test set remains consistent across multiple runs?

You will judge the performance of the trained model based on certain performance metrics.
If you keep updating your test set, then you will not know whether one run is better than the other.
For e.g. you are trying to predict the value of house. And you have only one data point in your test set and you are using mean absolute error (MAE) as your performance criteria.
Scenario 1: the actual value of house is 100K, your prediction with first model is 101K. MAE = 1000
Scenario 2: the the actual value of house is 25K, your prediction with second model is 26K. MAE = 1000
In reality, prediction is better with model1 as compared to model2. But because you changed your test set, you will not be able to make that conclusion. Had you kept your test set same in scenario2, your prediction would have been much higher making MAE much higher, clearly indicating that model1 is better than model2.
Hope this helps.


keep_dims is deprecated, use keepdims instead

It's not an error, it's simply notifying you that the code is written using past versions of Tensorlfow and some of the arguments of special methods were going to be deprecated in the future releases of the library. It's ok to be used, but you can also check what version is used and make a virtual environment and install the specified version of the library you want.

You can run your cell twice in order not to see the warning again.


How to generate a random sample and distribute values based in an probability distribution?

Create some random data
df <- data.frame(
  cat_cols = c(rep("A", 200), rep("B",150)),
  cont_vals = c(rnorm(200, 20, 5), rnorm(150,25,10)))
# Set desired binwidth and number of non-missing obs
bw = 2
n_obs = sum(!is.na(df$cont_vals))

Now plot it
library(ggplot2)
ggplot(df, aes(cont_vals))  +
  geom_histogram(aes(y = ..density..), binwidth = bw, colour = "black") +
  stat_function(fun = dnorm, args = list(mean = mean(df$cont_vals), sd = sd(df$cont_vals)))


What it Would be easier Building a Deep Net From Scratch or Using an existing Architecture?

It depends on your task and the amount of data you have. If you have so much data but you can not find similar tasks to have appropriate architecture you should stack convolution and dense layers yourself. But if you have appropriate amount of data and there exist good architectures then you have to decide what you want and how is your situation. Suppose that you want to have recognition task, there are so many architectures that are applied to ImageNet data-set. You can use transfer learning but there is a point here. Suppose that you want to fine tune GoogleNet. This is a very large network and is capable for recognizing about a thousand distinct classes. If you have a recognition task with 5 classes and you have an agent that should be online, this is not logical to have such a big network. You may have similar performance by stacking a few layers and get better time complexity. If you don't have so much data, freezing the layers and applying transfer learning to the last layer maybe a typical solution.


Machine learning worker performance features for optimum allocation of tasks to workers

Your problem is going a lot of directions.

If you have known worker attributes, like doctor, engineer, or accountant, you could have mirror document classification, and use machine learning to classify.

If you are going to measure performance, you will need to get a sample from each user. For example, take the top $\frac{1}{4}$ performance and have a target vector for the user.  Map a document to a user's strength.  If that user is over-queued, pick the next highest $\frac{1}{4}$ and so on.  You could use cosine similarity to give you a score. This can get tricky, as you may want to normalize on productivity.  You may get a document that a high productivity user is better at, but a low productivity user is OK at. By way of analogy, an american football player may be the best receiver and half back, but you might have a good backup receiver and no good backup halfback, so you let your backup play receiver.

Static parameters would be different, and I am not sure you should use them if you have a performance measure. My guess would be to queue before you have performance for a worker.  But even at 4 documents, you can select a top $\frac{1}{4}$.

You could also create clusters and measure a user's performance versus the cluster. This is the easiest approach to queuing, as you just take the next document out of the cluster that is a user's best cluster.  You only need a few cross-cluster workers to level queues. The downside is, if you have 20 clusters, it would take a while to get user performance versus cluster, so you would need to use static attributes until you get a valid sampling.


You can check quora for your QnA dataset.

As far as long answers are concerned it may not be because of BERT as  SQuADhas very crisp and short answers for most of its questions.

BERT already does understand long term dependencies in your data(one of reasons for using deeply bidirectional tranformer).


Why Mini batch gradient descent is faster than gradient descent?

It is slower in terms of time necessary to compute one full epoch. BUT it is faster in terms of convergence i.e. how many epochs are necessary to finish training which is what you care about at the end of the day. It is because you take many gradient steps to the optimum in one epoch when using batch/stochastic GD while in GD you only take one step per epoch. Why don't we use batch size equal 1 every time then? Because then we can't calculate things in parallel and computation resourses are not used efficiently. It turns out in every problem there is a batch size sweet spot which maximises training speed by balancing how parallelized your data is and number of gradient updates per epoch.
mprouveur answer is very good; I'll just add that we deal with this problem by simply calculating average or sum loss over all batches' losses. We don't really sacrifice any accuracy i.e. your model is not worse off because of SGD - it's just that you need to add up results from all batches before you can say anything about the results.


For tensorflow 1.x we need to execute

tf.enable_eager_execution()

to see results before creating a tf session.

For tf 2.x eager execution is enabled by default.


Comparing two images and showing the difference in a new image?

A simple and time efficent solution is the computation of the absoulte values of the difference image. This can be done as follows:

import cv2
img_a = cv2.imload([first_image_filename])
img_b = cv2.imload([second_image_filename])
img_diff = cv2.absdiff(img_a, img_b)
cv2.imshow('abs difference', img_diff)
cv2.waitKey(-1)


When is it convenient dropping duplicates when performing Time Series Prediction?

According to me, time-series ML Model are bit different than other routine ML  models. As time-series model is based upon the sequence of previous values, dropping a Date may impact your solution.

Specially in this case where sale is definitely dependent upon the particular day.
For eg consider store 38 is located in Area where on "2017-03-15" is holiday. So sale of store 38 will be comparatively high in comparison to other store and so on.

Rest you can try two different models one with Date and another without Date and compare the results as well.


I think the documentation is kind of self-explanatory here. Fit takes in array of size n_samples in which each element is the class of the datum or if the data point belongs to multiple classes, the input would be obviously of size n_samples x n_classes. That is what you gave in as input in your example. Each point can belong to any of the three classes. That is why you have [0, 1, 2] as number of classes. So as mentioned in the documentation if you try

>> lb.transform([0, 1, 2, 0])
[[1 0 0]
 [0 1 0]
 [0 0 1]
 [1 0 0]]

and if you try a class that is non-existent after fit like

>> lb.transform([0, 1, 2, 1000])
[[1 0 0]
 [0 1 0]
 [0 0 1]
 [0 0 0]]

No class named 1000 exists, so multi-targeted conversion for 1000 class case is plainly [0, 0, 0]. Hope this helps.


Can I use doc2vec for classification big documents (500-2000 words,
  20000 total documents, classification for three classes)?

Yes. This amount of documents is considered small to medium. It is worth noting that the original Doc2Vec paper experimented with 75K documents.

  Is it a problem that the documents are large enough and contain many common words?

No, it is OK. Consider the fact that both common and distinctive words are increasing with the size of documents, and distinctive words are what that matters. That is, larger texts are easier to distinguish.

  Can I train my data together with Wikipedia articles (using unique tag
  for every article) for a more accurate calculation word-embeddings, or
  it can't give positive effect?

It depends and is worth the try. If Wikipedia documents are close in format and content to your documents, they could definitely help. In the Doc2Vec paper, authors used 25K labeled documents (IMDb reviews with positive and negative labels) and 50K unlabeled ones for training.

Also, this is a nice tutorial on using doc2vec. It reports 1-2 hours for 100K documents.


Does it violate the assumptions of linear regression to perform it on time series data?

You are correct when you say that there might be autocorrelation in the error terms when some observation in time $t$ is contingent on $t-1$. Find a rigorous discussion of the problem here.
Usually a solution to cope with this problem is to include additional variables to "control for" autocorrelation. This usually means adding lagged $y$-variables (which are called autoregressive variables or $AR(t)$). So you could simply add a lagged variable $y_{t-1}$ to your model, which reads like "todays $y$ is (among other things) explained by yesterdays $y$".
Say you have a model which is contingent on time (omitting $i$ for convenience):
$$y_t = \beta X_t + u_t.$$
You can add an autoregressive term like:
$$y_t = \beta X_t + \gamma y_{t-1} + u_t.$$
For a detailled discussion including testing procedures, please refer to the link above.


how can i collect data set from social networks like instagram?

I am not aware about Instagram, or if there is a ready-to-use dataset for your purposes, but in general in order to get data from (almost) any online source you should a method called "web scraping" (This might help you while you google).

It might also be useful to you, if you decide what kind of tools you are going to use for your analysis.

I have tried something simple, but it was for Twitter though

Here is a rather simple piece of code that gets twitter 500 tweets concerning Barbie using R

require(twitteR)

setwd("C:/Users/slin/desktop/informatie")

ck <- "4bKJU1Z4B60SzbPPIva7OknnQ"
cs <- "cCxAPYDN3d0APvlurEBoXMNCKl0DKgrneV0w6KQCQenNpmM7fa"
at <- "715872051559575552-qMYaugTPHJeRg9fh70ABNWCjdl0Lp6T"
as <- "zxn6lOyCw2dcPyllkiHJq4zk0wHwGQ4tuKa9IpR96TNvO"

setup_twitter_oauth(ck, cs, access_token = at, access_secret = as)

t_stream <- searchTwitter('barbie', resultType="recent", n=500)

df <- do.call("rbind", lapply(t_stream, as.data.frame))

The ck, cs, at and as can be obtained as it is explained here in step 2

Also in that site you can see a guide of how to web scrap data from Twitter using Python.

You might also find useful this (Python), this (Python) and this (R)


Training neural nets: is it important that the data is randomly sorted?

Yes it is important that the data is randomly ordered. You want to remove patterns of how you descent along the cost function because such patterns can increase the likelihood of overfitting. It's easiest to explain on A,A,A, ... B,B,B ... pattern. Here the network is given an opportunity to adapt to peculiarities of the first training example before moving further. And the goal of training is not to adapt to peculiarities but to find features that generalize well. So it seems that a better idea is to expose the network to a different example in each iteration.

Most research works that I've seen simply randomize the order of examples in each epoch. But it seems sensible to think that a useful pattern might in fact exist. Perhaps one that focuses on maximizing the variety of features that the network is exposed to with each subsequent iteration.


How to interprete percentile information from the describe function in Pandas?

Percentiles indicate the percentage of scores that fall below a particular value. They tell you where a score stands relative to other scores.
For example:
a person height 215 cm is at the 91st percentile, which indicates that his hight is higher than 91 percent of other scores.
Percentiles are a great tool to use when you need to know the position of a value/score respect to a population/data distribution you're considering. Where does a value fall within a distribution of values? While the concept behind percentiles is straight forward, there are different mathematical methods for calculating them.
In your example 50% correspond to the median of the ordered values distribution.
In this case the median is calculated between two values: 1 and 2 so the median is calculated (in this case 'cause the number of values is even so the median as to be calculated between the fifth and sixth ordered values ) as the mean between them 1.5.


How can I perform multi-label classification if many labels are missing?

This is called Automatic Document/Text Categorization (a lecture note on this). There're some approaches you can take, such as ATC with semi-supervised learning or building one-vs-all classifier and labeling a new test sample with a new label when all classifiers reject it. This was discussed here as well.


How to set class weights for imbalanced classes in Tensorflow?

There's a function that does it automatically: tf.contrib.losses.softmax_cross_entropy(logits, onehot_labels, weight=weight)


Back Propagation Vs Learning rate in Neuralnet Optimisation

Using backpropagation is nothing else than performing (stochastic) gradient descent.
It computes the gradient, but it is  not the "optimal" weight. The gradient is used to update the current weight (according to the gradient descent algorithm).
The gradient descent algorithm needs a step size (which is called learning rate in the context of machine learning).
The step size defines how "strong" the current weights are updated by the current gradient.


Is there any APIs for crawling abstract of paper?

Look it up on:

Google Scholar link
Citeseer link

If you get a single exact title match then you have probably found the right article, and can fill in the rest of the info from there.  Both give you download links and bibtex-style output.  What you would likely want to do though to get perfect metadata is download and parse the pdf (if any) and look for DOI-style identifier.

Please be nice and rate-limit your requests if you do this.


Dataset
Here is a data set I created.
There are $143$ phrasal verbs. However, since some have multiple meanings, there are $181$ phrasal verbs pairs.
Examples
phrasal_verb | meaning
------------------------
black out    | faint
calm down    | become more relaxed
get off      | leave
get off      | remove
hold on      | wait
hold on      | grip tightly

Creation
I couldn't find any ready made, but I created the dataset by scraping this website.
I loaded the data set and for 'phrasal verb' I found the meaning in the 'Meaning' column by applying the following rules:
def tokenize_meaning(sentence: str) -> list:
    """
    Splits the string in the meaning column into a list of possible phrasal verbs.
    :param sentence: str from the meaning column
    :return: list of strings
    """
    meanings = re.split('\d\.\s+', sentence)
    meanings = flatten([re.split(', ', meaning) for meaning in meanings])
    meanings = [meaning for meaning in meanings if meaning[:3] == 'To ']
    meanings = flatten([re.split(' or ', meaning) for meaning in meanings])
    meanings = [meaning for meaning in meanings if not meaning == '']
    meanings = [re.sub('[\(|\/].*', '', meaning) for meaning in meanings]
    meanings = [re.sub('[.,?!]', '', meaning) for meaning in meanings]
    meanings = [re.sub('To ', '', meaning) for meaning in meanings]
    meanings = [meaning.strip() for meaning in meanings]

    return meanings


How to convert a SQLContext Dataframe to RDD of vectors in Python?

The Dataframe Python API exposes the RDD of a Dataframe by calling the following :

df.rdd # you can save it, perform transformations of course, etc.

df.rdd returns the content as an pyspark.RDD of Row.

You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what your vector represents with the information given.

Note 1: dfis the variable define our Dataframe.

Note 2: this function is available since Spark 1.3


Can The linearly non-separable data be learned using polynomial features with logistic regression?

Yes in theory the polynomial extension to logistic regression can approximate any arbitrary classification boundary. That is because a polynomial can approximate any function (at least of the types useful to classification problems), and this is proven by the Stone-Weierstrass theorem.

Whether this approximation is practical for all boundary shapes is another matter. You may be better looking for other basis functions (e.g. Fourier series, or radial distance from example points), or other approaches entirely (e.g. SVM) when you suspect a complex boundary shape in feature space. The problem with using high order polynomials is that the number of polynomial features you need to use grows exponentially with degree of the polynomial and number of original features.

You could make a polynomial to classify XOR. $5 - 10 xy$ might be a start if you use $-1$ and $1$ as the binary inputs, this maps input $(x,y)$ to output as follows:

$$(-1,-1): -5 \qquad (-1,1): 5 \qquad (1,-1): 5 \qquad(1, 1): -5$$

Passing that into the logistic function should give you values close enough to 0 and 1.

Similar to your two circular areas is a simple figure-of-eight curve:

$$a(x^2 - y^2 - bx^4 + c)$$

where $a, b$ and $c$ are constants. You can get two disjoint closed areas defined in your classifier - on opposite sides of the $y$ axis, by choosing $a, b$ and $c$ appropriately. For example try $a=1,b=0.05,c=-1$ to get a function that clearly separates into two peaks around $x=-3$ and $x=3$:

The plot shown is from an online tool at academo.org, and is for $x^2 - y^2 - 0.05x^4 -1>0$ - the positive class shown as value 1 in the plot above, and is typically where $\frac{1}{1+e^{-z}} > 0.5$ in logistic regression or just $z>0$

An optimiser will find best values, you would just need to use $1, x^2, y^2, x^4$ as your expansion terms (although note these specific terms are limited to matching the same basic shape reflected around the $y$ axis - in practice you would want to have multiple terms up to fourth degree polynomial to find more arbitrary disjoint groups in a classifier).

In fact any problem you can solve with a deep neural network - of any depth - you can solve with a flat structure using linear regression (for regression problems) or logistic regression (for classification problems). It is "just" a matter of finding the right feature expansion. The difference is that neural networks will attempt to discover a working feature expansion directly, whilst feature engineering using polynomials or any other scheme is hard work and not always obvious how to even start: Consider for example how you might create polynomial approximations to what convolutional neural networks do for images? It seems impossible. It is likely to be extremely impractical, too. But it does exist.


Classification based on a Clustering Result

I think you need to do some Feature Engineering, i.e., as you explained in the question, those values mean something to your application.

For example : 1-3 : Bad, 4-6 : Average, 7-10 : Good

V1 new_V1
3   Bad
5   Average
7   Good

Something like this, so that clustering algorithm can make sense out of it.

Assuming that you don't have any constraint in using R.
If you have both Numeric and Categorical variable(as above) then you can usethis package : ClustMixType, it can understand and do the clustering based on the data fed. As traditional K Means algorithm is not applicable here as it works only for Numeric Data.

If there is any discrepancy in data you can convert them explicitly by
new_V1 <- as.factor(new_V1);
if it is a numeric variable V2 <- as.numeric(V2) before feeding the data to Clustering algorithm.

Once you get it, then you can use this package: Bruota, use this package to get Predictor Importance(what all variables are important). This can be done only when you know the Target Variable, Most likely your target variable would be the Clustering outcome.

Implementation is over here, please follow this Link

Most likely you would end up getting your desired result, Do let me know if you have any additional questions.


How can I have the same initialization for all my networks?

You need to specify the seed in the initializer, e.g:

from keras.initializers import RandomUniform

seed = 0

model.add(Dense(64, kernel_initializer = RandomUniform(minval = -0.05,
                                                       maxval =  0.05,
                                                       seed = seed)))


Data Scientist Consulting Interview Guide

This might be a resource that you would be interested in. It's a preliminary assessment for health centers, but a lot of the structure can be used in any industry.


How to do node classification without deep learning?

That problem is commonly called graph classification. Common non-deep learning approaches for graph classification are spectral and kernel based.


Where does the random in Random Forests come from?

For each tree you randomly select from the variables that you can use to split tree nodes. Generally you randomly select 1/3 of the variables per tree.


R code that gives results like Wolfram Alpha for the expectation of a function of a random variable?

You can simulate one of these lines like this:
library(ggplot2)
X<-rnorm(1000, 1, sqrt(4))
f<-function(x) exp(-x*x)
df<-data.frame(sample_size=seq_len(1000), sample_means=cumsum(f(X))/seq_len(1000))
ggplot(df, aes(x=sample_size, y=sample_means)) +
geom_line() + theme_minimal() + labs(x=NULL, y=NULL)


User based recommendation factoring in user data

Collaborative Filtering:

Match users to people with similar tastes –recommend what they like
Commonly used in e-retail
Avoids the issue of users only being recommended more of what they already like (allows serendipity)

Example:
Your Scenario.
Methods: Euclidean Distance, Cosine Distance, Pearson Correlation Coefficient (most common).
Content-Based Recommendation:

Match users directly to products and content
Recommend based on what you have bought or viewed in the past
Commonly used for document recommendation: webpages, news articles,
blogs etc.

Examples:
Which Movie(s) might our user like?
Option1: The user selects preferences for the various features using pull-down menus etc. We match against the movies using Vector-Space methods (described next)
Methods: Vector-Space Method, K Nearest Neighbour(kNN)
Option2: The user rates a sample of the movies (explicitly or implicitly) as like/dislike. we then build a user profile model for that user using machine learning.
Method: Decision Trees


Keyword/phrase extraction from Text using Deep Learning libraries

The Google Research Blog should be helpful in the context of TensorFlow.

In the above article, there is a reference to the Annotated English Gigaword dataset which is routinely used for text summarization.

The 2014 paper by Sutskever et al titled Sequence to Sequence Learning with Neural Networks could be a meaningful start on your journey as it turns out that for shorter texts, summarization can be learned end-to-end with a deep learning technique.

Lastly, here is a great Github repository demonstrating text summarization while making use of TensorFlow.


Since CV data is mostly structured you can use Stanford NER for named entity recognition or use CRF to train and model your own named entity recognizer.


Adding bias in deconvolution (transposed convolution) layer

We are going backwards in the sense that we are upsampling and so doing the opposite to a standard conv layer, like you say, but we are more generally still moving forward in the neural network. For that reason I would add the bias after the convolution operations. This is standard practice: apply a matrix dot-product (a.k.a affine transformation) first, then add a bias before finally applying a non-linearity.

With a transpose convolution, we are not exactly reversing a forward (downsampling) convolution - such an operation would be referred to as the inverse convolution, or a deconvolution,  within mathematics. We are performing a (transpose) convolution operation that returns the same input dimensions that produced the activation map in question, with no guarantee that the actual values are identical to the original input.

You can see from the animations of various convolutional operations here, that the transpose convolution is basically a normal convolution, but with added dilation/padding to obtain the desired output dimensions. The trick is to retain the mappings of localisation between the pixels.

In the paper from which those animations are taken, they explain how a transpose convolution is essentially the convolution steps performed in reverse:

  ..., although the kernel w defines a convolution whose forward and
  backward passes are computed by multiplying with $\textbf{C}$ and $\textbf{C}^T$ respectively, it also defines a transposed convolution whose forward and backward passes are computed by multiplying with $\textbf{C}^T$ and $(\textbf{C}^T)^T = \textbf{C}$ respectively.

One other source to back up my opinion: in the PyTorch implementation it seems the bias is added the output of the convolution's result.


Batch norm: why the initial normalization?

I hope I got your question correctly...

It’s called “Batch” Normalization because we perform this transformation and calculate the statistics only for a subpart (a batch) of the entire training set not as a whole.


The downvotes are because of the topic, but I'll attempt to answer your question as best I can since it's here.

Data science is a term that is thrown around as loosely as Big Data.  Everyone has a rough idea of what they mean by the term, but when you look at the actual work tasks, a data scientist's responsibilities will vary greatly from company to company.

Statistical analysis could encompass the entirety of the workload in one job, and not even be a consideration for another.

I wouldn't chase after a job title per se.  If you are interested in the field, network (like you are doing now) and find a good fit.  If you are perusing job ads, just look for the ones that stress statistical and informatics backgrounds.  Hadoop and SQL are both easy to become familiar with given the time and motivation, but I would stick with the areas you are strongest in and go from there.


PyTorch MultiLayer Perceptron Classification Size of Features vs Labels Wrong

You have stated that:

Batch size: 30
Input dimensionality: 10
Label dimensionality: 1

With this in mind, the output dimensionality of the MLP must be the same dimensionality as the labels, which is 1. Therefore, the output size of the MLP cannot be 10, but 1.

Apart from that, please note that concatenating 2 linear transformations (without any non-linearity in between) is equivalent to a single linear layer. Your MLP should have a non-linearity (e.g. ReLU) applied after fc1 and before fc2.


Cannot interpret feed_dict key as Tensor: Tensor Tensor(“Placeholder:0”, shape=(3, 3, 3, 32), dtype=float32) is not an element of this graph

I had the same issue in Flask. The following code resolved it for me:

from keras import backend as K
K.clear_session()

Also read this discussion.


Where can I get a comprehensive criminal dataset?

Take a look at the Bureau of Justice Statistics site. Other sites that you might find interesting would be U.S. City Census Crimes Data. Of course, Knoema, will provide more of an international flare that you might find useful.


how to encode labels for relationship extraction

I would reommend parsing the sentences using something like spaCy, which will be able to build such relationships for you.

It will then be a matter of extracting the realtionships and deciding how you want to label them. Let's work through a small example. Imagine your big block of text contains many sentence. We first parse the big block, then could (optionally) extract just the sentence that we care about:

import spacy

nlp = spacy.load('en_core_web_sm')      # this defines our 'parser' for English
doc = nlp(big_block_of_text)            # you may want to clean the text beforehand

Assume we only want sentences that contain the word 'went':

sents = [sent for sent in doc.sents if 'went' in sent.string]

Here is an example of grammatically walking down the parse tree. I say semantically because we are using the grammar parts-of-speech to conditionally navigate (using e.g. nouns):

for sent in sents:                          # loop over each sentence
    for word in sent:                       # analyse each word
        if word.pos_ == 'NOUN':             # if it is a noun
            for child in word.children:     # find children (go down the tree)
                if child.pos_ == 'NOUN':    # select only nouns from the children
                    print(word, child)

I believe this snippet would extract the pairs in your given example, so the output would hopefully indeed be:

"He movie"
"they school"

There are many other parts-of-speech that you will be able to use to be more specific or match other use cases - check out the relevant spaCy docs.

How to actually encode the relationships between your target words will depend on their final purpose, i.e. what you eventually want to do with the encodings. You now have the noun pairs. If you want to retain the knowledge of where they actually came from (from which sentences), you could indeed build a hierarchy, that keeps a code for the origin sentence and then a code for each pair of words.


Lazy learner:

Just store Data set without learning from it

Start classifying data when it receive Test data

So it takes less time learning and more time classifying data

Eager learner:

When it receive data set it starts classifying (learning)

Then it does not wait for test data to learn

So it takes long time learning and less time classifying data

Hint : In supervised learning

Some examples are :
Lazy : K - Nearest Neighbour, Case - Based Reasoning
Eager : Decision Tree, Naive Bayes, Artificial Neural Networks


I would rather say, that in your training, underfitting occurs, because the train values are much better, than the test values. This can occur, when your memory cells cannot store enough information, so maybe try to increase the units or nodes of your LSTM, so the loss can converge better.


How to create a new column based on two other columns in Pandas?

You can try something like this to get a new dataframe that has pairs of (EventID, TeamCount):

event_id_team_count = data.groupby('EventID').agg({'TeamID': lambda x: x.nunique()})
event_id_team_count.rename(columns={"TeamID": "TeamCount"}, inplace=True)

If you want to have this new column in the original dataframe, all you need to do is to join the original dataframe with the one you have just created:

data = data.join(other=event_id_team_count, on="EventID")


Checking linearity for a linear regression model?

I like GAM (Generalised additive model) with regression splines:
# Load data
library(ISLR)
a = Auto

# Run GAM with splines
library(gam)
g = gam(mpg~s(displacement,5)+s(horsepower,5),data=a)
plot(g, se=T)

The result (plot) shows you that displacement can be well approximated by a linear function for lower values (< 250 or so). However, there is a "kink" at about 250, so that overall, a linear approximation would not be very good here.
See ISL, Chapter 7 for more details. There are also Labs for Python and R where you can see code details.
Also see this example in R with simulated data for more details.
Alternatively, look at a Q–Q plot after regression, e.g. in R:
l = lm(mpg~s(displacement,5)+s(horsepower,5),data=a)
plot(l)


According to the book, the problem of initializing weights with too big of a standard deviation is that it is very likely to cause neurons to saturate.

But with L2 regularization, when saturation occurred only the L2 term will affect the gradient, and cause weight decay.

And when weights get small enough not to cause saturation (for example around $1 / \sqrt n$), the other term comes to affect the gradeint.

So the relative influence of L2 term decreases.

(And of course, the absolute effect of L2 term will decrease by decaying the weight.)

Why $1 / \sqrt n$?

If all of the $n$ input neurons are 1 and the standard deviations of weights are $\sigma$, the standard deviation of the input to hidden neurons will be $\sqrt{n}\sigma$.

If you want $\sqrt{n}\sigma$ to be 1 to avoid saturation, $\sigma$ should be $1 / \sqrt n$.


How to structure and train an LSTM for real-time predictions in Keras?

The tough part about this problem is evaluating what moves are "correct" per se. In a fighting game sequence, there may be 2 or more moves which both work in theory in the current frame. If you are optimizing for knocking the other character out of frame, it would be useful to build a reward set which optimizes for this.
It may be worthwhile to build a data set that takes into account corresponding stick positional data and then labeling past and future time locations as advantageous/disadvantageous based on simulation, similar to a chess computer.
First you have to build something that can identify perfect combos- situations in which the enemy, regardless of their inputs, has no reliable way to escape the combo which results in high % or death. And then the problem becomes easier because the ai can just identify positions to set up this combo, and then executes them.
Train a cpu to combo via reinforcement, and then integrate with the identifier, and you'll have a pretty damn good bot.


Advice on the learning resources for deep learning

http://neuralnetworksanddeeplearning.com/
http://www.deeplearningbook.org/

They are the two very popular online free books. The first link even has working deep learning code.


BERT is free. Any Sentiment Analysis model will achieve the task.


Sorry I'm not going to read the paper, but in general it's perfectly possible to use binary values 0/1 as numerical regression targets. A regression model doesn't care about having a continuous sample in the training set, it just calculates the relationship between the features and the target variable. For example a linear regression needs only two points to learn a model in theory. Of course this doesn't work if there are specific patterns either between the two target values or outside them, the model can only generalize on the full continuous range. As a result it will mostly predict values in the range [0,1] but it can also predict values outside of it.


Is it possible to identify what data a neural net was trained on?

If you can see the loss for a specific dataset, then the most probable set would be the one with the lowest loss.


How to interpret dummy variable in ML prediction?

1) Using drop_first=True is more common in statistics and often referred to as "dummy encoding" while using drop_first=False gives you "one hot-encoding" which is more common in ML. For algorithmic approaches like Random Forests it does not make a difference. Also see "Introduction to Machine Learning with Python"; Mueller, Guido; 2016:

  The one-hot encoding we use is quite similar, but not identical, to
  the dummy encoding used in statistics. For simplicity, we encode
  each category with a different binary feature. In statistics, it is common
  to encode a categorical feature with k different possible values
  into k–1 features (the last one is represented as all zeros). This is
  done to simplify the analysis (more technically, this will avoid making
  the data matrix rank-deficient).

However, using dummy encoding on a binary variable does not mean that a 0 has no relevance. If gender_male has high importance that does not generally say anything about the importance of gender_male==0 vs gender_male==1. It is variable importance and accordingly calculated per variable. If you, for example, use impurity based estimates in Trees it only gives you the average reduction in impurity achieved by splitting on this very variable.

Moreover, if your gender variable is binary, gender_male==1 is equivalent to gender_female==0. Therefore from a high variable importance of gender_male you cannot infer that being female (or not) is not relevant.

2) In this case gender_male==0 AND gender_female==0 means Transgender is true.

3) see 1). For algorithmic approaches in ML there is no statistical disadvantage using one-hot-encoding. (as pointed out in the comments it might even be advantageous since tree-based models can directly split on all features when none is being dropped)


Regression vs Random Forest - Combination of features

I think it is true. Tree based algorithms especially the ones with multiple trees has the capability of capturing different feature interactions. Please see  this article from xgboost official documentation and this discussion. You can say it's a perk of being a non parametric model (trees are non parametric and linear regression is not). I hope this will shed some light on this thought.


Best way to deploy and Schedule Deep Learning Model

There are two steps necessary, you need a trained model and you need to apply this model regularly to new data.

Step 1: After training your model you need to serialize this in some way. This could be via the tooling in your deep learning framework, or by saving the parameters somewhere and the code somewhere else or by pickling the whole object. After storing this model artifact we can reuse this fitted model in step 2.

Step 2: Schedule some job that loads this model artifact, loads the data from BigQuery (in chunks or the full scope of the rows for the last time period), runs these through your model and saves the results. The schedule only kicks off some script, how to properly run these predictions depends a bit on the scale. If it is a very big job you would likely have to distribute this via something like Spark. If it is managable you could just do this on a single machine, maybe by chunking the data.


I will try to answer your question as shortly as possible.

Yes, if you define probability as a risk, then the probabilities are risk scores. But, there's a catch in these scenarios, you will have to include the prevalence of a class to calibrate them. If Person A has a risk score of 0.9 but you have observed that the positive class is only 20% of the data, then the actual risk is much lower than the probability it self. You can use clf.predict_proba() function to get these probabilities.
Once you have these probabilities, you should be able to set your custom threshold.
Yes, most of these models in different libraries have predict_proba() functions.

Let me know if you have more questions.


Pandas: access fields within field in a DataFrame

Multi-indexing might be helpful. See this.
But the below was the immediate solution that came to mind. I think it's a little more elegant than what you came up with (fewer obscure numbers, more interpretable natural language):
import pandas as pd
df = pd.read_json('test_file.json')
df = df.append(df) # just to give us an extra row to loop through below
df.reset_index(inplace=True) # not really important except to distinguish your rows
for _ , row in df.iterrows():
    currNumbDict = len(row['analytics'])
    print(currNumbDict)


Does using user-specific accumulative variables causes data leakage?

If the data is available at prediction time, then it is not data leakeage.
In your specific example, historical user data should be used at prediction time to build a better model.
You do have to decide on how to do the train/test split. You can split by user, by time, or combination.


Information extraction with reinforcement learning, feasible?

You ideally want to use Reinforcement Learning in situations where there is delayed feedback and stochastic transitions in the environment. Although you could potentially apply RL, in your case, you might be better off with a Sequence to Sequence learning framework (https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf) since you have access to the entire sentence and there is no stochasticity involved.

On the topic of RL with Information Extraction, this might be of interest:
Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning (http://arxiv.org/abs/1603.07954)


How does Naive Bayes classifier work for continuous variables?

The difference boils down to "how we define $P(x_i|C_k)$?", where $x_i$ is a single feature, and $C_k$ is a class from a total of $K$ classes.

Discrete

In discrete case,  $P(x_i|C_k)$ is represented by a table as follows:

x_i   P(x_i|C_k)
a     0.5
b     0.2
c     0.3

We have one of these tables for each feature-class pair $(i, k)$.

Lets denote $i$-th feature of data point $n$ as $x_{n,i}$. Each row of this table can be estimated using

$$\hat{P}(x_i=a|C_k) = \frac{\sum_{n:n \in C_k}{\mathbb{1}_{x_{n,i}=a}}}{N_k}$$

which divides the number of samples that have i-th feature equal to $a$ by total number of samples in class $C_k$ (of course from the training set).

Also, check out pseudocount that avoids zero estimations.

Continuous

In continuous case, we either discretize the continuous interval of $x_i$ into bins $\{b_1,..,b_m\}$ and proceed the same as discrete case, or we assume a function like Gaussian (or any other one), as follows:

$$P(x_i|C_k)=\frac{1}{\sqrt{2\pi}\sigma_{i,k}}e^{-(x_i-\mu_{i,k})^2/2\sigma_{i,k}^2}$$

This way, for each feature-class pair $(i, k)$, $P(x_i|C_k)$ is represented with two parameters $\{\mu_{i,k}, \sigma_{i, k}\}$ instead of a table in discrete case. The estimation of the parameters is the same as fitting a Gaussian distribution to one dimensional data, that is:

$$\hat{\mu}_{i,k} = \frac{\sum_{n:n \in C_k}{x_{n,i}}}{N_k}, \hat{\sigma}^2_{i, k} = \frac{\sum_{n:n \in C_k}{(x_{n,i} - \hat{\mu}_{i,k})^2}}{N_k-1}$$

Instead of Gaussian, we can opt for a more complex function, even a neural network. In that case, we should look for a technique to fit the function to data just like what we did with Gaussian.

The rest is independent of feature type

Representing $P(C_k)$ is the same for both discrete and continuous cases and is estimated as $\hat{P}(C_k)=N_k/N$.

Finally, the classifier is $$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}\hat{P}(C_k)\prod_{i}\hat{P}(x_i=x_{n,i}|C_k)$$

Or equivalently using log-probabilities,

$$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}\mbox{log}\hat{P}(C_k)+\sum_{i}\mbox{log}\hat{P}(x_i=x_{n,i}|C_k)$$


ValueError: operands could not be broadcast together with shapes while using two sample independent t test

The answer to this question would be :
The objects created by the xs method of the Pandas DataFrame look like two-dimensional arrays. These must be flattened to look like one-dimensional arrays when passed to ttest_ind. The values attribute of the Pandas objects gives a numpy array, and the ravel() method flattens the array to one-dimension.
It would go like :

df  = pd.read_csv('final_out_transposed.csv', header=[1,2], index_col=[0])
case = df.xs('0', axis=1, level=0).dropna()
ctrl = df.xs('1', axis=1, level=0).dropna()
(tt_val,p_ttest ) = ttest_ind(case.values.ravel(), ctrl.values.ravel(),
equal_var=False)
print (tt_val)
print(p_ttest)


Check this example (a reproduction of the IRNN experiment with pixel-by-pixel sequential MNIST in "A Simple Way to Initialize Recurrent Networks of Rectified Linear Units" by Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton).

Relevant code:

model.add(SimpleRNN(hidden_units,
                    kernel_initializer=initializers.RandomNormal(stddev=0.001),
                    recurrent_initializer=initializers.Identity(gain=1.0),
                    activation='relu',
                    input_shape=x_train.shape[1:]))


Try something like this:

import matplotlib.pyplot as plt
plt.figure(figsize=(30, 20))  # the size you want

# your code goes here


What is the largest public wearable accelerometer dataset?

You can check these additional ones:

HAR: http://archive.ics.uci.edu/ml/datasets/Human+Activity+Recognition+Using+Smartphones
It is collected by attaching a smart-phone and it has accelerometer and gyroscope data.
It has 30 participants conducting 6 different activities. The acceleration is at 50 Hz and they are using 3D accelerometers.

USCHAD: https://dl.acm.org/citation.cfm?doid=2370216.2370438 Have a look at this paper. Here you will find the data as well.

It has 14 subjects (7 male, 7 female) and there are 12 different activities. Additionally, it has accelerometers and gyroscopes, and it has a ground truth labeling by an observer standing nearby. They are sampled on 100Hz. Again they are using 3D accelerometers.

I would strongly suggest feature-engineering by defining time-windows where you segment the time-series data. After defining the small windows of a few seconds, you can have one activity per window. Then you can easily extract some time-dependent features such as mean, correlation, acceleration, orientation, etc.. Also, Fast Fourier transform on these small windows can be applied so you can extract many frequency domain features like energy, coherence, etc.


What is the best way to visualize range of data?

I believe boxplot or violin plot is a good idea and you could overlay datapoints with a bit of jitter to the former. See below an example in seaborn taken from a relevant question:
import seaborn as sns
import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.violinplot(x="day", y="total_bill", data=tips, color="0.8")
sns.stripplot(x="day", y="total_bill", data=tips, jitter=True, zorder=1)
plt.show()


Machine learning algorithm that can use many instances to predict 1 continuous outcome per person

I think you would benefit from keeping your data as a time series and use a sequence model. Commonly you would use an RNN, the most popular flavor would be a LSTM.

Here is an example using Keras that I think is relevant for you

Multivariate example from same blog (but forecasting instead of classification)

LSTM with regression


You could plot a graph of update versus iteration and analyze the variation of each update as the number of iteration increases. Like in here, where they are comparing the variance of the standard gradient descent algorithm versus its stochastic version.


How to find correlation between two junctions of a road

Sorry to be posting as an answer, but the comment section doesn't allow me to place preformatted text. Using your example above, is the graphic representation similar to the one below?

X=== a -10- b ---- c
     |      |      |
     |      |      |
Y=== a ---- b -20- c

So you are wondering if the 10 (Road X, junction a - b) and 20 (Road Y, junction b - c) are correlated?

Update: Thank you for confirming.
Sorry I am totally not in the traffic domain, but I will give my 2 cents given the little I know about ML in general.

I assume that you want to understand the correlation between 10 and 20 to help you build a prediction model. If you unpack the question a bit, I think perhaps we can restructure the question to.

(If you are simply looking at all the available combination of and their correlations, it will probably take forever, and it might not be as actionable unless you normalize the information into your prediction model)

Label  = number of cars

Features = date, Road, junction a, junction b, [Number of cars in nearby intersections] + other related features

If it is the case, perhaps we can try to normalize the data to the Road, junction a, b through feature engineering.

Expanding on the diagram

W z-4--a --6- b --8- c
  |    |     |      |
  3    20    28     9
  |    |     |      |
X-z-1- a -10-b -19- c
       |     |      |
       15    12     7
       |     |      |
Y--10- a -14-b -20- c

For Road X, junction a - b, we have 10 as the Label

Reconsider the nearby roads as:

level 0 -- Immediate connecting segments

level 1 -- Immediate connecting segments to level 0 roads

level 2 -- Immediate connecting segments to level 1 roads (if required)

W z-1--a -1-- b --1- c
  |    |     |      |
  1    0     0      1
  |    |     |      |
X-z-0- a -**-b -0- c
       |     |      |
       0     0      1
       |     |      |
Y--1-- a -1 -b --1- c

You will know best how further away you want your output to be counted. you just create the necessary number of features to accommodate that. Perhaps distance to the junction in question, etc. Whatever makes more sense for your model
You will know best the necessary granularity. To start, perhaps you can count the number of level 0 and 1, recording each with the average in your model, etc.

Example:

Road     Jun A     Jun B    L0 Connection  L0 Avg Car    L1 Connection   L1 Avg
   X         a         b                6        17.5                9    [you get the idea]

Not sure how to handle bidirectional traffic. does the order of a and b imply direction already?
Your model will probably take care of the influence of your level 0 and level 1 (and level 2, etc.) connections.

Sorry for the long-winded response; hopefully it helps brainstorm some ways to tackle your problem.


One hot vector output in classification task

Here it's a mistake to one-hot-encode the class, because it turns the task into multi-label classification instead of regular multi-class classification. In your task an instance can only have a single class, so the class should be encoded as an int (for example with LabelEncoder).
This is why the predicted probabilities don't sum to 1, because in multi-label classification the classes are independent of each other. For example the output [0.9,0.4,0.7] means that class 1 label is predicted true at 90%  and false at 10%, class 2 label is predicted true at 40% and false at 60%, class 3 label is predicted true at 70% and false at 30%. It wouldn't make sense to pick the maximum probability of the 3 classes in this scenario.
See also this answer.


There are multifarious studies about object detection. The method you are going to use depends on your data. For instance, if you have centre of the objects alongside their height and width, you can simply use YOLO method which is one of the pioneer approaches for detection. As another example, If you have data which is classified pixel-wise, you can employ RCNN methods or Fully Convolutional methods like FCN itself or SegNet. There are open source implementations with appropriate trained parameters for these models on GitHub.

Apart from the methods, you have to consider the restrictions they may exist for your conditions. For instance, there may be situations where your task is online and you have limited computational power. In such cases, I don't think you can employ very deep architectures like SegNet.


What level of math is required for machine learning research

There can be no objective answer to this question. Obviously the more one understands the better, but the field of ML is vast, quite specialized and ranges from very theoretical to very applied research, so it's perfectly reasonable to publish in ML without a strong background in maths.

A better way to estimate your own ability to publish papers in a particular area or journal is to study recent papers published in this area/journal. You should be able not only to understand them but to redo the reasoning:

Understand the problem and the solution proposed by the authors
starting from the initial problem, how would you solve it? Can you think of alternatives to the authors' solution?
Can you find limitations to their approach and improve on it?

If you reach step 3 then congratulations: you are ready to publish your own research!


How to handle the target variable being in the features

I think you need to formalize your problem a little bit.

If I were to predict the winner of a tennis match, I would do something like this. I will be predicting the probability of Player 1 winning (or Player 2, without loss of generality). Then, this is a simple classification problem.

Then, for the features, I will organize it in a way that uses player 1 or player 2 to represent the features.

For example, if the variables are: players-current-ranking and face-to-face-win-rate. Then the feature vector will be something like:

player1_current_rank: 10
player1_2_face_to_face_win_rate: 0.7
player2_current_rank: 20
player2_1_face_to_face_win_rate: 0.3

You can never have target variable in your feature, otherwise it won't make any sense predicting it.


Why are support vector machines good at classifying images?

Support vector machines have one built-in "layer" that helps with having an interpretation of the data - the kernel. You could even use output from some other image classifier, including a neural network, as the kernel. E.g. you could measure how far apart two images are in "classifier space" from a trained neural network (maybe one trained against different targets from the ones you want to classify with the SVM)

There is also an implied "layer" in image feature construction.  Common image features are histograms of pixel values, histograms of pixel differences (edge detection), bags of "visual words" and there are many more possible transformations and statistics.

Typically if you are using a classifier other than a deep neural network (which is supposed to discover the features automatically), then you will pre-process the image into a set of features. That pre-processing covers the majority of an approach being "good at images".


Ignoring symbols and select only numerical values with pandas

Use str.strip if the prefix is fixed or str.replace if not:

data = pandas.Series(["U$ 192.0"])
data.str.replace('^[^\d]*', '').astype(float)

This removes all the non-numeric characters to the left of the number, and casts to float.


Why might my validation loss flatten out while my training loss continues to decrease?

This is totally normal and reflects a fundamental phenomenon in data science: overfitting. When the validation loss stops decreasing, while the training loss continues to decrease, your model starts overfitting. This means that the model starts sticking too much to the training set and looses its generalization power. As an example, the model might learn the noise present in the training set as if it was a relevant feature.
When training your model, you should monitor the validation loss and stop the training when the validation loss ceases decreasing significantly. It is also the validation loss that you should monitor while tuning hyperparameters or comparing different preprocessing strategies.


Can LSTM Predict The Next Few Days Of Stock Price?

The reason why you find the same dataset everywhere is that financial data is extremely secretive and expensive and nobody is willing to share their financial information because in doing so they will be effectively losing an edge in the market. For that reason public financial data is mostly gathered by academics and not too often.

Second, LSTM can make predictions for the next n-hours, n-days, n-months and even n-years like the article STOCK PRICE PREDICTION USING LSTM,RNN
AND CNN-SLIDING WINDOW MODEL by Sreelekshmy Selvin et. al. and many others show. You just need to build a good training vector and the target can be anything you want. As it's natural in financial markets, accuracy decreases as time increases.


How can we perform STS(Semantic Textual Similarity) on UnSupervised dataset using Deep Learning?

You could take pretrained embedder and look for distances between the embeddings.
There's LASER from Facebook. This is an unofficial pypi package, it replaces some of the internal tools used for tokenization and BPE encodings. I've used it extensively and it works just fine. It encodes your text as 1024-element numerical vector. Then you can just calculate distance metric between embeddings i.e. euclidean.


Rolling window features for multiclass classification

Usually the point of a rolling window is to calculate and use a moving average:

A moving average is commonly used with time series data to smooth out short-term fluctuations and highlight longer-term trends or cycles.

By definition the only meaningful aggregation for this kind of purpose is the mean: the idea is to represent the same type of value as the original series while correcting noisy variations.
Anything else is not the standard use case for a rolling window, so the answer to questions 2 and 3 would completely depend on the application.


TypeError: Image data of dtype object cannot be converted to float

The data you are trying to plot (data[i]) contains data of type object (i.e. a string) which of course cannot be plotted as they are not numbers. This is caused by the fact that one of the columns in your dataset is a string which you are trying to plot, likely the Name column. Try removing the Name column from the dataset and see if there are other column which contains strings.


Which type auto encoder gives best results for text

A working example of a Variational Autoencoder for Text Generation in Keras can be found here.

Cross-entropy loss, aka log loss, measures the performance of a model whose output is a probability value between 0 and 1 for classification. Cross-entropy loss goes up as the predicted probability diverges from the actual label. In the case of character-by-character autoencoder, each character in the vocabulary would be a label.

Cross-entropy works if the input and output are the same size, that is the case in character-by-character autoencoder. Often times in text analysis, the input and output sequences are different lengths so a second term, encoder loss, is added to the objective function.


What is the appropriate method to find n-grams/sub-phrases/parts-of-sequences that are referring to a specific topic or belong to a certain category?

An important question to solve this problem would be: what is the range of input topics? Is the topic selected among a predefined closed list? Can it be any search query?
A similar question might be asked about the target documents and/or terms: can they be processed so that any candidate term is extracted in advance, and the task only consists in identifying the right terms for the particular topic?

Assuming the most open variant of the question (i.e. nothing is available beforehand), I think that one would need:

a terminology extraction system which extracts any candidate term from the text (preferably specific to the data to be processed).
a third-party resource in order to calculate a semantic representation (typically a vector) of any possible term including the input topic query, so that the topic can be matched/compared against any term.


Why is the idf important in tf-idf when it seems to just re-scale your features?

To complement my comment I'm taking those paragraphs from data camp tutorial in which they explain this in a very clear way


How to optimize client's portafolio with analytical models?

Your problem could be framed as a recommendation system. Given historical information about preferences, what is the best item to recommend next to a user?


Computing confidence interval for average from individual predictions

OK, we have two dozen predictions-with-uncertainty, and actual point values. I'm still not sure we know enough to answer, so I'm going to just outline some points and hope someone can improve upon this. A quick search found three similar StackExchange questions:

Combining 13 independent regressions seems the most similar -- Note comment warning this might be meaningless. The best-answer suggests a sampling approach because the result has no closed form. Using a Normal approximation would have a closed form, but I think your plot suggests the errors are not Normal, or your 95% CI is much too narrow.
Combining 2 -- See esp. the reply discussing meta-analysis and whether you have a fixed effects or a random effects model. This may be what you are doing, in which case re-formulating the question for that community may be best.
Delta Method -- Seems to me the same question, but takes a direct maximization approach, assuming approximations hold.

A couple of observations suggesting any simple average is not going to give you what you want, and you need to be very clear what is being combined, and what question the result is going to answer:

If these are 95% Confidence Intervals, then your error model is much too conservative. Nearly all of them should include the true value (gray dashed diagonal) but only about 5 do. Therefore any parametric combination will be at least as unreliable. But suppose you have a black box with uncalibrated estimates, and you just want to know the "average" from that black box, however uncalibrated.
If these are two-dozen unrelated estimates from a crowdsourcing platform, the "average" is not meaningful. On the other hand, if they are performance measures of the crowd, then the average is something like the average performance of the crowd across all question types. In that case:

1st approximation would just average the two-dozen points and use the standard deviation of that. (Each point can be weighted.)
2nd approximation would treat each estimate and CI as Normal, and use one of the formulas for combining Normal CIs. Note however that it seems error bars are wider towards the edges. That may require special handling.
3rd approximation: sample say 100 forecasts from the two-dozen distributions and then combine those ~2400 samples. (Weighting can be done by varying sample size for each forecast.) Works even if the distributions are not assumed to be Normal.

If instead you have two-dozen forecasts from a single statistical model on different input data, and you want to know the average forecast across all input combinations, don't combine them. Just run the model with no factors specified, and use that result.

Hopefully a bona fide statistician can supply a more definitive answer.


Why the validation accuracy is constant for all the pre-trained models while applying on medical datasets with .czi format?

weights: One of None (random initialization), 'imagenet' (pre-training on ImageNet), or the path to the weights file to be loaded. Default to imagenet.

With weights=None, it is randomly initialized and
With base_model.trainable = False, it is not being trained.
So, basically your Conv. layer is not getting trained.


How to understand the return values of scipy.interpolate.splrep

This documentation may work:

BSpline

Univariate spline in the B-spline basis.

$S(x) = \sum_{j=0}^{n-1} c_jB_{j,k;t}(x)$

where $B_{j,k;t}$ are B-spline basis functions of degree k and knots t.


Here is an example of r lattice xyplot using log scale on the x axis and the difference of your two measures I(expected - simulated)

df <- data.frame(
x         = c(2, 4, 8, 16, 32, 64),
expected  = c(47.9995, 95.9783,
191.9127, 383.9708, 767.8831,
1457.2771),
simulated = c(48, 96, 191.8, 383.8,
767.4, 1458.1228))
xy <- xyplot(I(expected - simulated) ~ x ,
auto.key=TRUE,
data =  df ,    type=c("p","g"),
scales=list(x=list(log = 10) ),
ylab="difference expected - simulated", xlab="x", main="Simulation Results" )
print (xy)

Note, that I added a 6th result to your sample data,that was missing.


Why does storing the output of .count() result in all 1's or 0's?

Because on each loop you assign a single value to the entire column:
   tots_df['orc_4_totals'] = s4    # entire column orc_4_totals = s4!

(the same thing for the other column.
It equals all ones then, just because your final loop value fills it with 1.
You need to insert single values instead, for each row of your target dataframe.
However, because I don't know if your target dataframe tots_df has an existing index, I would suggest first saving the values in a list, then assigning them:
orc_4_values = []
orc_5_values = []
for name in pros_unique:
    # perform your checks, then assign using:
    s4 = df.loc[(df.pros_split == name) & (df.orc_4 == 1)].orc_4.count()
    s5 = df.loc[(df.pros_split == name) & (df.orc_5 == 1)].orc_4.count()
    orc_4_values.append(s4)
    orc_5_values.append(s5)

# Add to dataframe after the loop
tots_df['orc_4_totals'] = orc_4_values
tots_df['orc_5_totals'] = orc_5_values


Batch normalization variance calculation

Well, the only reason, I would say here, is the numerical stability. For instance, if the Batch size $m =1$ during the training, then you may end up with $NaN$ variance!

Update: I doubled check the BN paper again, they clearly mentioned the condition that $m>1$. However, I found an interesting post discussing the cases when biased estimator preferable than the unbiased one. This might explain why the authors used the biased one during the training and the unbiased one inference.


Why should the initialization of weights and bias be chosen around 0?

Assuming fairly reasonable data normalization, the expectation of the weights should be zero or close to it. It might be reasonable, then, to set all of the initial weights to zero because a positive initial weight will have further to go if it should actually be a negative weight and visa versa. This, however, does not work. If all of the weights are the same, they will all have the same error and the model will not learn anything - there is no source of asymmetry between the neurons.

What we could do, instead, is to keep the weights very close to zero but make them different by initializing them to small, non-zero numbers. This is what is suggested in the tutorial that you linked. It has the same advantage of all-zero initialization in that it is close to the 'best guess' expectation value but the symmetry has also been broken enough for the algorithm to work.

This approach has additional problems. It is not necessarily true that smaller numbers will work better, especially if the neural network is deep. The gradients calculated in backpropagation are proportional to the weights; very small weights lead to very small gradients and can lead to the network taking much, much longer to train or never completing.

Another potential issue is that the distribution of the outputs of each neuron, when using random initialization values, has a variance that gets larger with more inputs. A common additional step is to normalize the neuron's output variance to 1 by dividing its weights by $sqrt(d)$ where $d$ is the number of inputs to the neuron. The resulting weights are normally distributed between $\left[\frac{-1}{\sqrt{d}}, \frac{1}{\sqrt{d}}\right]$


Relu with not gradient vanishing function is possible?

If you are using an activation like y=x, then your model is a simple linear one. Multiple layers with such activation will be equivalent/reduced to only one layer with a linear activation! Thus you can only map linear function satisfactorily with this type of model. To be able to learn complex non-linear functions, you need to use multiple layers with non-linear activation in between to make the whole model non-linear

To prevent the vanishing gradient problem, there is a variant of relu called Leaky ReLU. This activation is same as relu in the positive region of x. For negative region of x, it is a linear function with a small slope (e.g. 0.2). This makes Leaky ReLU a non linear activation at x=0 point.


Tool to Label Images for Supervised Classification

If you are specifically looking to outline the whales, seems like FastAnnotationTool could work:
https://github.com/christopher5106/FastAnnotationTool

Other options here:
https://en.wikipedia.org/wiki/List_of_Manual_Image_Annotation_Tools

When had to annotate many images for a project, I built a fairly simple MATLAB gui that displayed images.  I cycled through each image and if I clicked on a point on the image, that image was annotated and the point where I clicked was saved in a corresponding .mat file (.csv would have been better). It worked... mostly.  I'd guess between python and OpenCV, this could be done easily without MATLAB.


Python - AttributeError: type object 'DataFrame' has no attribute 'read_csv'

read_csv() is not available on DataFrame. to read csvs using pandas -

import pandas as pd
data = pd.read_csv("file_name")

if you check type(data), it will be pandas DataFrame.


Salesman problem with additional conditions and features

I couldn't tell for sure if it is applicable without seeing the exact program, but you could look into constraint programming for a solution.

Constraint programming aims to find the best solution to an optimization problem with a series of constraints, which can be hard or soft. Hard constraints rule out a possible solution (for example, no more than 10 hours drive on the same day), while soft constraints would dictate which of the remaining solution is the best, for instance awarding points for staying the night in cities with good acommodation, or having roughly the same amount of miles every day.

Constraint programming is based on backtracking, so it doesn't scale well at all for problems with many possible solutions, so it might not be for you. Worth taking a look at at least. Hope it helps!