size of deep learning object detection pre-train models

If you are researching different architectures, I would recommend you to look at https://www.modelzoo.co/. It contains plenty of links to implementations in different architectures. This can help you get an intuitive understanding of how the models are built via code in case the papers aren't intuitive.

I use PyTorch more so I focused on finding PyTorch implementations. Here are links to Github repositories for the models you listed:

YOLO v2
YOLO v3
Cascade R-CNN
Faster R-CNN
Retina Net

With PyTorch networks, you can count the number of weights/parameters in multiple ways, this StackOverflow question has multiple answers all showing different ways to do it. The simplest one being:

pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

In the post, there are also ways which detail the size of each layer in the network.


Model Predictive Power and better prediction of 1 or 0 in Python scikitlearn?

Which statistic can show me model predictive power ? For example CV-f1 score
Which statistic can show me whether my model better predict event 1 or event 0 ? True negative and true positive rate


Distinguish Multi-Task vs Single-incremental Task in Continual Learning

TL;DR/Summary: The classes ($y1$, $y2$, $y3$ below) in multi-task can be anything (it may be that $y1 \cap y2 = \emptyset$, $y1 \cap y2 = \emptyset$, and so on).  In Incremental we take the labels (and data from a common set, (i.e. $y[:2] \subseteq y[:4] \subseteq y$, by the definition of subsetting)

It is just a question of the interpretation of the definition.  The definitions look very similar to each other but the devil's in the details.

Assuming that the model can distinguish the upper bond number of classes: For example, it is an ANN with $N$ neurons in the output layer and the number of classes ($k$) in the task with most classes is $k <= N$ (multi-task) or the total number of classes ($k$) is also $k <= N$ (single-incremental-task); we can say that:

Multi Task

Here we will train the model on different tasks over time, this is often called as reinforcement learning.  In semi-python pseudo code (where .train already includes things like cross validation):

model = Whatever(...)

X1 = [[1, 0],
      [2, 2],
      [3, 0]]
y1 = [0, 1, 0]
model.train(X1, y1)

X2 = [[4, 4],
      [5, 5],
      [6, 0]]
y2 = [1, 2, 0]
model.train(X2, y2)

X3 = [[7, 0],
      [8, 8],
      [9, 9]]
y3 = [0, 1, 1]
model.train(X3, y3)
score = model.score(X3, y3)

Here the tasks may or may not be related.  Or often are slightly related (e.g. identifying different types of objects in each training).

Single Incremental Task

This is also training the model several times, here we have a single task in X but do not feed the entire dataset at once.  In semi-python pseudo code:

model = Whatever(...)

X = [[1, 0],
     [2, 2],
     [3, 0],
     [4, 4],
     [5, 5],
     [6, 0]]
y = [0, 1, 0, 3, 2, 0]

model.train(X[:2, :], y[:2])
score1 = model.score(X[:2, :], y[:2])
model.train(X[:4, :], y[:4])
score2 = model.score(X[:4, :], y[:4])
model.train(X, y)
score3 = model.score(X, y)

Here the task is one but it may be a big one.  One place where this technique is used is to build a learning curve, which is one way of evaluating if we have enough data to understand the variation of the task.

Extra note: in the multitask case we said that $y1 \cap y2 \cap y3 = \emptyset$ could be (and most likely is) possible.  One example would be: $y1$ are different models of cars and $y2$ are different models of ships. And the question is: Do understanding different models of cars help with differentiation different models of ships?

(P.S. ys will always be enumerated from 0 up to the number of classes, i.e. the numeric values of y will always be the same but their class meaning does not need to be).


GRU is related to LSTM as both are utilizing different way if gating information to prevent vanishing gradient problem. Here are some pin-points about GRU vs LSTM-

The GRU controls the flow of information like the LSTM unit, but without having to use a memory unit. It just exposes the full hidden content without any control.
GRU is relatively new,  and from my perspective, the performance is on par with LSTM,  but computationally more efficient (less complex structure as pointed out).  So we are seeing it being used more and more.

For a detailed description, you can explore this Research Paper - Arxiv.org. The paper explains all this brilliantly.

Plus, you can also explore these blogs for a better idea-

WildML
Colah - Github

Hope it helps!


Is it ethical to claim experience with big data when the data isn't a part of the new advertising/social media/retail fad?

Data can be big in various ways. It can be large N (observations) and small P (variables per observation), like access logs for example. It can be large P and small N, like in biostatistics where you have tens of thousands of gene expressions from tens or hundreds of people. Or it can be both N and P, like in Facebook data. Data can also be made big, by normalizing every single detail of it, meaning that you'll be able to make a massive relational database schema that actually represents simple data.

All kinds of big data needs different approaches and companies mostly care about their own version of big data. That's why the answer isn't black and white.

To answer the question you posed in title then I believe it's ethical to claim experience with big data as long as you can put it into context of the business or domain where you encountered it and can describe some situations where a different set of tools or skills was needed than is used for smaller toy examples.


LSTM to multivariate sequence classification

Since you are using LSTMs for classification using the
multivariate time series data, you need to model your time-series data into a supervised learning problem and specify the previous
time steps you need to look before by specifying the time-lag count

You need to look into the to_supervised function and specify the number of outputs your model has. In your case, it is 4.

Split your train and test set from the whole set of data. A 70:30 ratio for the train, test would be a good start.

Also, note that you need to scale your values using the
sklearn.preprocessing.MinMaxScaler() function.

You need to reshape your train and test values into (batch_size/sample_size, time_steps, feature_size) as an LSTM Layer in Keras expects your data to be fed in a 3D array format.

For eg: Your training shape would be train.shape = (batch_size, 100, 3)

And for the model building in Keras
model = Sequential()
model.add(LSTM(number_of_hidden_units, activation='relu', input_shape=(n_timesteps = 100, n_features = 3)))
model.add(Dense(4, activation='softmax')) #since number of output classes is 4
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()
model.fit(X_train, y_train, validation_data=(X_test, y_test), no_of_epochs, batch_size)

Note that I have just given a rough outline of the model building and left out the hyperparameters at your convenience. You can either stack more LSTM layers onto the model or tune the number of hidden_layers in the model Refer Multi-Class Classification for more details on it.


There is no problem doing that, you can define your variable in every way. Just make sure you don't use the variables used to define your Y as predictors.

The only problem is that, after you defined your new variable, your model could be facing nonlinearities which are difficult to model (meaning: I can get a true from Decrease,Decrease,Decrease and from Increase,Increase,Increase, so some variables could be working for the first case, but not for the second).

If you have obtained better results with this new Y variable, I think you are fine.


What Machine Learning Technique can I use to judge boxing fights?

It depends by what you mean by the result of each round. In boxing after each round fighters receive from $7-10$ points in case if none of the fighters was KO, submitted or disqualified. An extract from Wikipedia :

The 10-point Must System is the most widely used scoring system since the mid-twentieth century. It is so named because a judge "must" award ten points to at least one fighter each round (before deductions for fouls). Most rounds are scored 10–9, with 10 points for the fighter who won the round, and 9 points for the fighter the judge believes lost the round. If a round is judged to be even, it is scored 10-10. For each knockdown in a round, the judge deducts an additional point from the fighter knocked down, resulting in a 10–8 score if there is one knockdown or a 10–7 score if there are two knockdowns. If the referee instructs the judges to deduct a point for a foul, this deduction is applied after the preliminary computation. So, if a fighter wins a round, but is penalised for a foul, the score changes from 10–9 to 9-9. If that same fighter scored a knockdown in the round, the score would change from 10–8 in his favour to 9–8. While uncommon, if a fighter completely dominates a round but does not score a knockdown, a judge can still score that round 10–8.

Whether the points for each round given, the decision is straightforward - simply a sum of all scores for each round and doesn't require any machine learning.
However, if the question is - how to judge a particular round, the question is more involved. One may collect the scores on referee cards with the number of significant strikes, strikes landed, knockdown's, control of the ring, e.t.c, or even attempt to implement an algorithm, which given the video, detects, whether the strike has landed on the opponent, was significant. But it seems to be rather a tough task, and the data mining for supervised algorithm would be quite expensive.


Grouping numeric data into efficient group/pool

Sort all devices.

Take the first x elements until their average would be over 5 MB.

Then take the next x2 devices until their average would hit 25 MB.

Straightforward with sorted data. It's not the global optimum but will get you reasonably close.


Is there any technique, which can decide no. of bags for Bagged Logistic regression?

From the start I have to state that I am not aware about any paper regarding number of bags for bagged logistic regression. Therefore my assertions apply generally, for any bagged ensemble.

The short answer is no, and I do not think that if possible to be constructed. There are a couple of reasons that I will describe above.

The first reason is that it depends on the complexity of the joint probability you want to estimate. Technically any model you build for prediction purposes aims to estimate a conditional probability space on output variable giving the joint input variables. Doing bagging puts the problem if the estimations from selected samples does cover the that relation in all places. To exemplify, you can have one input categorical variable with 2 levels or with 10 levels. I think more estimations are needed for the later case.

Second reason if the sample itself is representative. Often in practice the sample is not purely a random sample. It contains various interdependencies which can lead to problems. For example if you collect data from multiple countries and you have more samples from some countries simply because of costs, but not related with the phenomena you study.

Third reason is that 20%. A typical bootstrap sample have the same size as original. Reducing the size is interesting because of performance gain or because of induced artificially variance which makes the samples "more independent". However estimating what is the effect of decreasing the sample size is again hard to estimate.

Fourth factor could be also the signal to noise ratio. In other words if you data is noisy enough you need more samples to surface to interesting signal in a stable way.

What you can do?

Since finding the answer to your question is hard, and knowing it is close to similar with knowing the true model, there are things which can be follow in order to find a proper number of bags. What I recommend would be to build repeatedly models with increasing number of bags and see the error estimates. Usually when those estimates have a low variance and adding bags does improve the performance significantly than you can assume you need no more bags and bagging with your chosen model is not able to capture any other structure in your data.


If you want to get the time duration between the start and end in seconds, you can try this :

data %>% group_by(SessionID) %>%
summarise(start_time = min(Timestamp), end_time =max(Timestamp)) %>% # using your code until here
mutate(`session end_time in seconds` = as.numeric(difftime(end_time, start_time, units = "secs"))) # gives the time interval in seconds


Python : Approaches to mine for time ordered sequences

Tru to make some features from the datetime columns -
attr = ['Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Is_month_end', 'Is_month_start',
'Is_quarter_end', 'Is_quarter_start', 'Is_year_end', 'Is_year_start']

you can google search about making date features.
2) Find the cumulative sum of the events -
cumsum_df_all = df.groupby('column')[other-columns].cumsum()
cumsum_14_day = (df.groupby('column')[other-columns].rolling(14, min_periods=1).sum().reset_index(0, drop=True)

like wise for 7,3,1 day, just look how frequent the data is and fast it changes.

Likewise find the roling std dev. and avg.

rolling_avg_14_day = (df.groupby('column')[other-columns].rolling(14,min_periods=1).mean().reset_index(0, drop=True))
set the rolling period accordingly.

find the difference between the event values for a specific period -
diff_7_day = df.groupby('column')[other-columns].diff(periods=7)

basically, we are trying to convert time ordered sequences to linear regression problem.


Using a low-level library, such as Theano or TensorFlow, it is likely that you can construct new schemes for reducing tensors (maybe via some learnable weight vector etc). In TensorFlow, you also get automated gradient calculations, so you should be able to still define a cost function and use existing optimisers, without needing to analyse the new design yourself or re-write back propagation formulae.

The universal approximation theorem essentially states that in order to have a network that could learn a specific function, you don't need anything more than one hidden layer using the standard matrix multiplying
plus non-linear activation function. So, invention and use of new architectures needs some reason. The reason is usually that these additions improve the speed or scope of what can be learned, they generalise better from training data, or they model something from a problem domain really well.

No doubt there have been explorations of all sorts of variations to the standard NN model over time. The ones that have become popular have all proven themselves on some task or other, and often have associated papers demonstrating their usefulness.

For example Convolutional Neural Networks have proven themselves good at image-based tasks. The design of a 2-dimensional CNN layer has a logical match to how pixels in an image relate to each other locally - defining edges, textures etc, so the architecture in the model nicely matches to some problem domains.

If you can find a good match from your vector multiplication model to a specific problem domain, then that's an indication that it may be worth implementing in order to test that theory. If you just want to explore other NN structures to see whether they work, you can still do so, but without a specific target you will be searching for a problem to solve (unless by chance you stumble upon something generally useful that has been overlooked before).

To address the question in the title:

  Why neural networks models do not allow for multiplication of inputs?

Some low-level ones do (e.g. TensorFlow). However, this is not an architecture that has proven itself useful, so authors of higher-level libraries (e.g. Keras) have no reason to implement or support it. There is the possibility that this situation is an oversight, and the idea is generally useful. In which case, once someone can show this you will find it would get support across actively-developed libraries pretty quickly since it seems straightforward to implement.


partial fitting, how to ensure one hot captures all features consistently

If you're using sklearn, this is a great use of CountVectorizer as a workaround since you can specify a vocabulary.

To start, get a list of all the 1000 categories and set that as the vocabulary in the transformer. Then convert the column to a string data type and apply this transformer to each batch:

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(vocabulary = pre_made_category_list)
endoded_variable_matrix = cv.fit_transform(dfX[categorical_column])

Even though it's technically a count vectorizor, since there is only one word in the string, the counts for each row will be 1 for the category it is and 0 for everything else, so its effectively one hot encoding the variable. The order of the columns will be the order of the vocabulary, so the matrix will be consistent between folds.


Groupby date and find number of occurrences of a value a in another column using pandas

This can work:

import pandas as pd
import numpy as np

unstacked_df = stacked_df.unstack()

The next time you post, you could post a reproducible code so we could help you a little more.


Detecting abnormal 'cat' behaviour via Supervised Learning

At what resolution are you performing you classification? At sample level?

If so, what does your baseline model look like? Did you try for example just feeding the last 5-10 raw samples as features and see what accuracy you get that way?

Also, you could try seeing what happens if you do some kind of "smoothing" at the output - e.g. if you assume that one "good" sample, surrounded by 10 "bad" samples should be reclassified as "bad" also -- because the system cannot jump so quickly between normal and faulty modes of operation.

Finally (or should be firstly), what is the accuracy that is actually necessary for your model to be useful in practice?


The function you have described is a loss function. It is the function which we want to minimize in order to train our model. The loss function is called ordinary least squares. We can also see that the model that we are trying to fit is a linear function with model parameters $\beta$. This is an optimization function and regularizarion is often used in optimization problems to attain a solution which is less likely to be a result of overfitting.

No regularization

Without any regularization the solution to a linear regression is

$\beta=(X^TX)^{-1}X^TY$

L2 regularization

To understand regression it is much easier to first start with the more widely used L2 regularization, ridge regression. This is defined as

$\min_\beta (Y-X\beta)^T(Y-X\beta) + \lambda ||\beta||^2$

We can solve this to get a closed-form solution using a derivative

$0 = -2X^TY + 2X^TX\beta+2\lambda\beta$

$X^TY = (X^TX + \lambda I)\beta$

$\beta = (X^TX + \lambda I)^{-1}X^TY$

Let's analyze this result in contrast to the solution without regularization and see what it means. We can see that the only difference is the added regularization term $\lambda$ within the inverse. As we should know, the inverse of a larger value, will cause its result to get smaller. For example the inverse of 1/5, is larger than 1/2. So by adding the regularization term we actually reduced the associated weight $\beta$.

This is why regularizers are often called a penalty term. They further constrain the optimization function. By adding an additional cost associated to each weight, we can reduce the impact that each weight will have on our model. Thus reducing the potential for overfitting.

L1 regularization

L1 regularization is the penalty used for LASSO regression, it's cost function is defined as

$\min_\beta(Y-X\beta)^T(Y-X\beta) + \lambda||\beta||$

From what we learned above we can already tell that this additional cost will cause the resulting weights to be penalized. Unfortunately L1 regularization does not have a closed form solution because it is not differentiable when a weight $\beta$ falls to 0. Thus this requires some more work to solve. LASSO is an algorithm for finding the solution.

First let's remember that our constraint here is $||\beta||\leq c$ where c is a real number which is inversely related to $\lambda$. The plan is to find an appropriate $\lambda$ such that the minimum of the function falls within or on the contour of the constraint. This constraint has very sharp edges which lie on each dimensional axis at a distance $c$ from the origin. You can imagine it like a diamond in 2D space, an octohedron in 3D space, etc.

In high dimensional space these spikes have a very high likelyhood to being hit by the function you wish to optimize thus this will cause many of the features to have an associated weight of 0. For example if we compare the regularization of this line in 2D space using both L2 and L1 regularization.

In this picture the distance $r$ is equivalent to the factor $c$ I used above which is inversely related to the regularization term $\lambda$. The function being optimized (the cost function) is the described using the ellipses which represent their levels (like an altitude map). The shape at the center of the axis is the regularizer which is used to constrain the weights associated to each feature.

We can first notice that the original function is minimized exactly at its minimum. However, choosing this solution has a high probability of being one which overfits the training data.

We see that L2 regularization did add a penalty to the weights, we ended up with a constrained weight set. However both weights are still represented in your final solution. The function being optimized touches the surface of the regularizer in the first quadrant.

However, in the L1 case we can see that $\theta_2$ is the only feature that is required for this regression problem and $\theta_1$ will have a weight set to 0. Thus we can ignore it. It just happens that smooth functions encountering a function with pronounced edges tends to hit these edges with higher probability than hit the vertices. I do not know the proof for this, but I have seen it in a course as supplemental material and it was complicated. This is why L1 regularization is often used for feature selection.

Combining both together

What is often done is first using L1 regularization to find out what features have lasso weights which tend to 0, these are then removed from the original feature set. Then with this new feature set we can apply L2 regularization in order to still control these new features to control for overfitting.


Value error array with 0 features in linear regression scikit

The error says that the array you feed into predict has shape=(1,0) meaning that it must be an empty iterable.

And by looking at your code, it's easy to see why. x_test starts of as empty, and in your code, the only way to add things to it is if i > le. But since your loop is defined as for i in range(le), by definition i will never be greater than le since the last value of range is le-1


Maximize the margin formula in support vector machines algorithm

The separating hyperplane that we are looking for is $w^Tx+b=0$. That is $w$ is the coefficient or the slope of the separating hyperplane and $m$ is the margin.

By scaling, for the positive class, we want the closest data point (support vectors) to satisfies $w^Tx+b=1$ and for the negative data class, we want the closest data point to satisfies $w^Tx+b=-1$.

Recall from geometry class, (a $3$ dimensional case is discussed here) that the distance from a point $y$ from a plane $w^Tx+b=0$ is $$\frac{|w^Ty+b|}{\|w\|}.$$

Hence as a result, the distance of the closest point to the hyperplane would be $\frac{|\pm1|}{\|w\|}=\frac{1}{\|w\|}$ and hence, the margin, that is the closest distance between the two classes would be $\frac2{\|w\|}$.

We want to maximize the margin, $\frac2{\|w\|}$ which is equivalent to minimize $\frac{\|w\|}2$ which is equivalent to minimize $\frac{\|w\|^2}2$ subject to the constraints that we classify the points correctly.


In self-attention, it is not the decoder attending the encoder, but the layer attends itself, i.e., the queries and values are the same.

In practice, this is usually done in the multi-head setup. You can view that as every head focusing on collecting different kinds of information from the hidden states. In multi-headed attention with $H$ heads, you first linearly project the states in $H$ query vectors, $H$ key vectors, and $H$ value vectors, apply the attention, concatenate the resulting context vectors and project them back into the same dimension.


This question is highly subjective, like asking what colour keyboard is best for programming faster.

A large number of Computer Science research at universities is done using Unix based operating systems (normally a Linux variant like Ubuntu) because it is completely open source and allows the user to tweak things as they please. Windows never allowed this officially. Apple's OSX doesn't either. However, OSX is based on Unix like Linux, and so there are many more similarities in every day usage between Linux and OSX than Linux and Windows. Many of the poster-boys/girls from data science have an academic background, where they likely used Linux.

Because many of the latest research ideas come from universities (where I claim most researchers use Linux), it is common to see open-source projects that have only been tested and packaged for Linux usage. Making them run on OSX is a smaller jump than going for Windows, as OSX and Linux are more alike.

Other guesses as to why many pick Apple products, are that they generally have higher build quality, they have a reputation for many thing just working (though IMO, Microsoft and Linux have become almost as good in that respect) and they are marketed as being luxury consumer products - and so are visually appealing. I'm sure there are books written on how social psychology with respect to this. All this ends up with the price being somewhat inflated for the actual hardware parts that you get.

In the end, someone who wants to get work done and has the necessary skills  will manage it on a 10 year old netbook running Windows. Likewise, owning a Linux server with 8 GPUs and a Macbook Pro to remotely connect to that server is no guarantee of good work. I for one have worked on all three platforms, for data science as well as other tasks, and although I have my preferences, the operating system rarely poses the biggest problem.


Nominal categorical variable with two levels: Label Encoding or One Hot encoding?

If you use one-hot here, you're just adding an unnecessary variable that is perfectly correlated with another variable in your model. Rather than thinking of it as a label encoding of gender where "0=male, 1=female", think of it as a binary flag for is_female, where "0=false, 1=true".


linear regression with "partitioned" data

It is due to the fact that the 6th predictor is a linear combination of the other 5. You can write:

electricity = 100%-kerosene-candle-generator-solar-other

Linear regression models do not handle linear combinations at all. However the solution is easy, since the 6th variable is embedded in the first 5, you can just drop a column since it is implied by the other 5.


How to access maximum volume of tweets using Twitter Streaming API, without firehose access?

Topsy - The site provides much more rate limit, and has archived all the tweets from 2006.
Its API provides a free trial.

A solution that is paid will be Gnip's Full history search API


When can it be called transfer learning?

Good question but rather besides the point.
What I mean is that even though estimating european house prices can be considered the same task (as estimating american house prices), nevertheless in this example the NN was re-purposed using transfer learning methods from another task.
Whether one would like to call the european house prices as same or related task, is not so much the point here (which can become pedantic), but rather if transfer learning methods were used and how succesfull they were.
IMHO


They call the same underlying method, so there is no functional difference.

Calling the dataframe member function is preferred for OOP patterns, but there are many redundancies/aliases in pandas and python in general.

In case you are curious, here is how the source code breaks down (it is a mess).

The DataFrame (pandas/core/frame.py) method is simply:

def isna(self):
    return super().isna()

Where DataFrame extends NDFrame (implemented in pandas/core/generic.py). NDFrame subsequently invokes:

def isna(self):
    return isna(self).__finalize__(self)

Which was imported here:

from pandas.core.dtypes.missing import isna, notna

In pandas/core/dtypes/missing.py:

def isna(obj):
    return _isna(obj)

The _isna function is later aliased as _isna = _isna_new because there is a deprecated method _isna_old(obj).

The _isna_new(obj) function then performs the logic operations:

def _isna_new(obj)
    if is_scalar(obj):
        return libmissing.checknull(obj)
    # hack (for now) because MI registers as ndarray
    elif isinstance(obj, ABCMultiIndex):
        raise NotImplementedError("isna is not defined for MultiIndex")
    elif isinstance(obj, type):
        return False
    elif isinstance(
        obj,
        (
            ABCSeries,
            np.ndarray,
            ABCIndexClass,
            ABCExtensionArray,
            ABCDatetimeArray,
            ABCTimedeltaArray,
        ),
    ):
        return _isna_ndarraylike(obj)
    elif isinstance(obj, ABCGeneric):
        return obj._constructor(obj._data.isna(func=isna))
    elif isinstance(obj, list):
        return _isna_ndarraylike(np.asarray(obj, dtype=object))
    elif hasattr(obj, "__array__"):
        return _isna_ndarraylike(np.asarray(obj))
    else:
        return obj is None

Ultimately, the DataFrame method passes itself as a parameter to the same function that you call with pandas.isna().


No. There is no similar mechanism for continuous variable.

If it worries you, that overall prices fluctuate in relatively low range,
you can

1) demean the price, that is subtract mean price from all price values. Then negative values will clearly show below-average and positive above-average prices.

2) after demeaning you can divide values by the standard deviation of the price. (1) and (2) together is called "standardization".

Alternatively you can

3) rescale your price to the range of values you want. Usual choice is (0,1) range.

If you do this for one feature (price in your case), then it makes sense to do the same for other features.

Whether or not this will help to get better prediction results depends on the model. Some models, a typical example would be SVM, do require such transformation.


Looking for an algorithm that correctly clusters visually separable clusters

With appropriate parameters, DBSCAN and single linkage hierarchical agglomerative clustering should work very well. Epsilon=0.2 or so.

But why? You know the data, just use a threshold.

If you just want an algorithm to "confirm" your desired outcome then you are using it wrong. Be honest: if you want your result to be "if $F-factor-1 > 1.5 then cluster1 else cluster2", then just say so, instead of attempting to find a clustering algorithm to fit to your desired solution!


Can I scrape data from government websites if there is no mention about commercial usage?

Short answer: It all depends on how you plan to use the data you scraped. If you plan to upload it to your website, that is generally not OK since content can be copy write protected and you could get sued. If you plan to just use the data for personal use offline you should generally not have a problem.

I suggest you take a look at these links:

https://www.archives.gov/global-pages/privacy.html
http://www.ic.gc.ca/eic/site/icgc.nsf/eng/h_07033.html

Those are the privacy pages found at the bottom of the links you provided.

You will find all the information you need there, including email addresses to which you can write to if you have any questions.

Though to help you out a bit, both of the sites you provided state that most of the content or all of it is copy write protected.

In conclusion, ask permission if you are going to re post the data somewhere on the internet. Websites that allow you to do so clearly state it in their privacy policies or terms of use.


How to implement a YOLO model using TensorFlow?

I personally prefer to use darknet rather than Tensorflow. However, use these codes to build YOLOV3 in TensorFlow.

Also, take a look at this link


How to compute accuracy for CNN when outputs are one-hot encoded

Predicted class is the one with highest probability in output vector (class B in your case) & accuracy is correct predictions %, unless I'm missing your point.

The problem that you have mentioned is representative of multi-class classification which is solved using Softmax output layer in neutral net. NOTE that Softmax uses Cross Entropy as loss function.

I feel that calculating Cross Entropy on output may not make sense.


You may encode each level and concatenate -
If we ignore the path till file1 as its same across all the names.
Then we need 1 digit for subfile and 2 digit for targetfile

c:/users/file1/subfile1/targetfile_0  - [0 00]
c:/users/file1/subfile1/targetfile_1  - [0 01]
c:/users/file1/subfile1/targetfile_2  - [0 10]
c:/users/file1/subfile2/targetfile_0  - [1 00]
c:/users/file1/subfile2/targetfile_1  - [1 01]
c:/users/file1/subfile2/targetfile_2  - [1 10]

Another approach can treat all paths as a feature. With this, you may try all the many encoding approaches e.g. Here.
+-----------+------------+--------------+
| Feature#1 | Feature#2  |  Feature#3   |
+-----------+------------+--------------+
| File_1    | Subfile_1  | Targetfile_1 |
| File_1    | Subfile_1  | Targetfile_2 |
| File_1    | Subfile_1  | Targetfile_3 |
+-----------+------------+--------------+


Chat Bot Answering based on Data Corpus Self-Training

It seems like your problem is some form of FAQ matching. The user asks a question, the NLU part matches this question to a relevant answer located in your files.

For the training aspect, you could do two things:

Similarity measure

You could simply use a string matching distance like Levenshtein between your input and the different outputs, and return the output that is the closest. This way you do not need to train your model per se. Since this is a form of a K-Nearest Neighbour with one neighbour, the learning happens during the inference. The advantage of this is that if your files change, or you have more answers, you won't have to change anything.

This is what you mean by self-learning.

This blog explains how to use fuzzywuzzy (a Python package to compute such distances): https://marcobonzanini.com/2015/02/25/fuzzy-string-matching-in-python/

Trained model

Obviously, simple distance measures might not scale well or might simply not perform great. Your other option is to treat this as a classification problem. Your classes are your different answers from your file, and your inputs are the user inputs. This will require the user input to be labelled.

This is what you refer to as manual training.

Question answering

Everything that was mentioned above was assuming that you have a known list of possible answers. If what you want is to have your bot "query" your text files and figure out the answer on its own, it will be much more complicated task. Question answering from open text is a complicated field of research. Just think of Google. It is able to sometimes answer simple questions like "Who?", "What time?" etc. but it's not very good yet at giving concise answers to complex queries.


What's a reasonable distribution to model views over time of... this question?

One way too look at this problem, by looking at the literature, seems to frame it as an Hawkes process, as e.g. in  Rizoiu et al., https://arxiv.org/abs/1602.06033 .

The paper also cites previous work using other modeling approaches:
One set  of  approaches  describe  popularity  dynamics  [...] as being power-law shapes from either an  exogenous  shock  or  endogenous  relaxation,  a  combination of power-law and exponential decay, multiple
power-law decays with periodicity or a collection of recurrence peaks.


how to apply MC dropout to an LSTM network keras

Well, in order to enable dropout during test phase you can do something like this:

keras.layers.Dropout(0.5)(x, training=True)

Then you'll probably want to run it multiple times. If you don't care about the inference time just run forward pass multiple times and at the end calculate mean and variance of your output.


Generating Image data sets for training CNNs

I decided turning my comment into an answer.

If you want to go pro, use a framework such as scrapy.

Personally, I find them overly-cumbersome and I have been successful using the following approach. I think your use case is simple enough for it to be of use to you as well.

Assuming you are using Python3 as well, you can grab a webpage easily, and then get what you want using XPath notation.

from lxml import html
import urllib.request

# keep running until there are no "next" pages
for page in range(999):
    url = 'http://blablabla.com/?page=%d' % page
    text = urllib.request.urlopen(url).read()
    tree = html.fromstring(text)
    images = tree.xpath('//img[@class="car"]/href()')
    types = tree.xpath('//div[@class="type"]/text()')
    if not images:
        break
    for i, (cartype, image) in enumerate(zip(types, images)):
        urllib.request.urlretrieve(image, '%s-page%d-img%d.png' % (cartype, page, i))

(Purely illustrative example.)

Now adjust as you may. XPath is an incredibly powerful notation of accessing XML nodes. Much more is possible than I am writing here. Take this tutorial to learn the full XPath syntax.

Some web designers make it much harder to access whatever you want because they do not properly class-ify their HTML objects. In those cases, you may have to access a parent node and ask for their child. Or access a sibling and then get the siblings. Anyway, XPath and Python's lxml package make all this incredibly easy.

Any modern browser like Chrome and Firefox also lets you easily explore the DOM of any webpage. Just right-click and press Inspect or go to Developer Tools in the Tools menu or some such.

Note: some website like scholar.google.com disallow scrapers and are very good at detecting if that's what you're doing. You can specify an user-agent for urllib, but it might be futile. Even advanced frameworks may not be able to help you there.

EDIT: I have made a blog post where I elaborate a little more.


Image features (produced by VGG19) do not properly train an ANN in Keras

After doing a bunch of searching and trial & error, I found out that using the sigmoid activation function for all the layers and using a binary_crossentropy loss fixed the issue. The newly added hidden layers were too few and too small for the ReLU function to work.


If i use use BERT embeddings for if cosine(sent1,sent2) > 0.9, then is it fair to assume s1 and s2 are similar

They might or might not be similar, the embeddings extracted by mean pooling the BERT output usually have high cosine similarity even though the input sentences are completely different.
Bert embeddings are not meant for sentence similarity task(SST), but there is some research combining Bert and SST.
Here are those resources,
SBERT paper: https://arxiv.org/abs/1908.10084
SBERT implementation: https://github.com/UKPLab/sentence-transformers


Reducing the dimensionality of word embeddings

There is a paper on this subject calle

  Simple and Effective Dimensionality Reduction for Word Embeddings, Vikas Raunak

You can read it here

You can also find the implementation here

In my opinion it works quite well


How to distinguish informative and non-informative feature - Feature importance?

My approach is quite different. After the creation of feature importance, I usually formulate the question "Is a feature of low importance really non-informative or its informative nature is shadowed by other features?"

How do I answer this question?

I use a simple method - decreasing the number of features in each iteration* (a greater chance that the most important features are not between them), adding a dummy feature containing random numbers, and creating a new feature importance. The random dummy feature is non-informative by definition, so assumption, that features of importance less than this dummy feature are non-informative is IMHO not so strange. So, the next step is to remove these features.

Then I usually make a selection starting from the other side. Sometimes more valuable, more important features, being surely informative, cause overfitting and removing them increases accuracy.

*) The default partition of features in each iteration (colsample_bytree) is equal to 1, but it may be set in the stage of feature selection to a much lower value of the range 0.2-0.4.


As far as I know LDA is the state of the art approach for topic modeling, but I'm not following the field very closely. So I would say that it's pretty safe to use LDA, and my guess is that the different approaches are likely to give similar results overall.
In case you want to try different methods, the question of evaluating topic models is quite complex. This article might help.
Side note: there is a non-parametric variant (no need to choose the number of topics) of LDA called Hierarchical Dirichlet Processes.


Should Discriminator Loss increase or decrease?

Discriminator consist of two loss parts (1st: detect real image as real; 2nd detect fake image as fake). 'Full discriminator loss' is sum of these two parts.
The loss should be as small as possible for both the generator and the discriminator. But there is a catch: the smaller the discriminator loss becomes, the more the generator loss increases and vice versa.
Discriminator loss: Ideally the full discriminator's loss should be around 0.5 for one instance, which would mean the discriminator is GUESSING whether the image is real or fake (e.g. the same as coin toss: you try to guess is it a tail or a head).
Generator loss: Ultimately it should decrease over the next epoch (important: we should choose the optimal number of epoch so as not to overfit our a neural network).
I could recommend this article to understand it better.


Machine Learning using NVIDIA DIGITS - Can't Classify Left or Right Direction of a Ball Throw

a few things,
these images look huge. Are they resized or cropped before going into the net. Try resizing instead of cropping. Also, if you use flip/rotation augmentation that will mess up your network.

Second, in the training data what % are right and what % are left throws? ideally, you have about 50:50 in the training set. If images in training are mostly left, it might learn to guess left preferentially.

BTW, there might be an easier way to solve this problem. First, extract small patches and classify each patch as: 1 (contains a ball) or 0 (doesn't contain a ball) - this is segmentation.

After segmentation, a simple algorithm that tells you if the ball is left of center or right of center will give you the correct answer.


How can the performance of a neural network vary considerably without changing any parameters?

In general, there is no guarantee that ANNs such as a multi-layer Perceptron network will converge to the global minimum squared error (MSE) solution. The final state of the network can be heavily dependent on how the network weights are initialized. Since most initialization schemes (including Nguyen-Widrow) use random numbers to generate the initial weights, it is quite possible that some initial states will result in convergence on local minima, whereas others will converge on the MSE solution.


Retrieve dropped column names from `sklearn.impute.SimpleImputer`

I've got the same issue today, and it's a shame your post got no answers. I think this question is not well addressed in the sklearn documentation. I can show you my workaround to this issue:

headers = X.columns.values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

empty_train_columns =  []
for col in X_train.columns.values:
    # all the values for this feature are null
    if sum(X_train[col].isnull()) == X_train.shape[0]:
        empty_train_columns.append(col)
print(empty_train_columns)

The idea is to keep all your column names, and after you split your data check which of them completely empty in your training set. If I'm not wrong the Imputer respects the column order so, for example, you can correlate every feature with its importance if you are using Decision-Tree-based models.

I'm not satisfied with this ugly piece of code but I couldn't find a more elegant (and simple) solution.


Is there any library available for balancing imbalanced text dataset?

from imblearn.over_sampling import ADASYN, SMOTE, RandomOverSampler
from imblearn.under_sampling import NearMiss, RandomUnderSampler

ros = RandomOverSampler(random_state=777)
X_ROS, y_ROS = ros.fit_sample(testing_tfidf, testing_target)

smt = SMOTE(random_state=777, k_neighbors=1)
rus = RandomUnderSampler(random_state=777)

Good article for reference


LabelEncoding selected columns in a Dataframe using for loop

LabelEncoding your features is a bad practice

You should avoid using LabelEncoder to encode your input features! Don't believe me? Here's what scikit-learn's official documentation for LabelEncoder says:

  This transformer should be used to encode target values, i.e. y, and not the input X.

That's why it's called LabelEncoding.

Why you shouldn't use LabelEncoder to encode features.

This encoder simply makes a mapping of a feature's unique values to integers. For example, let's say we want to encode a feature called shirt color, which represents the color of the shirt someone's wearing. This feature has values ['red', 'green', 'blue', ...]. If you encode these into integers, i.e. [1, 2, 3, ...], you might confuse your model by because you have now given relationships to these values that don't exist in the real world, e.g. red < greed < blue or red + green = blue. This type of feature is called nominal and preferably should be one-hot encoded.

There are features however, where you might want to map their values to integers. These are called ordinal. For example, the feature rating, which has values ['bad', 'good', 'excellent', ...]. By mapping these to integers you actually preserve the relationsips these values hold in the real world, e.g. bad < good < excellent. There is a catch to this however, in order to do the above, you need to map each value with a specific integer (e.g. we can't map 'good' -> 1, 'bad' -> 2, 'excellent' -> 3, because that doesn't preserve the real-world relationship of these values). The computer doesn't know which number to map to each value, though, so if you use LabelEncoder even on ordinal variables, it most likely won't generate the correct encoding.

How to properly encode ordinal features

A more proper way of encoding ordinal variables is manually choosing the mapping. This requires more work and isn't as elegant as a one-liner that encodes all values, but is the only correct way. Let's see how we can do this in pandas.

custom_mapping = {'bad': 1, 'good': 2, 'excellent': 3}

df['rating'] = df['rating'].map(custom_mapping)

Obviously this needs to be done for each ordinal feature.

At this point I think it's clear that I strongly recommend against using LabelEncoder, but if you still want to do it at least do it correctly.

If you still want to use LabelEncoding

While both answers by @ggordon and @Anan Srivastava will do what you want, they don't have much value in practice. The problem isthat by not bounding the fitted LabelEncoder to a variable, you are loosing the mapping from categories to numbers. If you want to predict on future data, you won't know which number to encode each category with.

Expanding upon @ggordon's answer

columns_to_be_encoded = [...]  # list of column names you want encoded

# Instantiate the encoders
encoders = {column: LabelEncoder() for column in columns_to_be_encoded}

for column in columns_to_be_encoded:
    df[column] = encoders[column].fit_transform(df[column])

This way you have a dictionary of fitted encoders so that you can reuse the same encoding if you wish.


Identify "steady-state" time series window

This could be defined as a classification problem over your time series.
One approach I can think of is to build an LSTM based classifier with a couple of dense layers for binary classification (whether the window is steady or not) and define your own loss function.
So in theory, you're training the LSTM network to sample batches of your time series and classify them as steady state or not.


Why is it important to have sufficient number of instances in your dataset for each stratum?

Lets take an example that you are trying to evaluate average income of a company. Assume company has 100 employees in total and they belong to two stratum (70 management and 30 technical executives) and management executives on average are paid higher compared to technical executives.

You are the surveyor:
Due to resource constraints, you have access to survey of any 20 individuals chosen at random.

Case 1: you did a random sampling from 100 employees and end up with 10 management and 10 technical executives. you will get an estimate for average income that is biased towards technical executives. In other words, you are giving more importance/ representation to income of technical executives than
actual representation in the population.

Similarly, in other attempt:
Case 2: you did a random sampling from 100 employees and end up with 17 management and 3 technical executives. you will get an estimate for average income that is biased towards management executives.

Both the above sampled cases have bias and are not representative of the population. Therefore, when we have the stratum identified before the sampling, stratified sampling should be done. For eg. random sample for 14 out of 70 management executives and  6 out of 30 technical executives.

In your given example of the housing and income data, income scale has been shrunk by a factor of 1.5, so as to create less no of income categories for the same bin width. Any other factor > 1 could also be used to ensure each stratum has considerable number of instances.


Existing pre-trained NLP models to detect if a text input is a question

There are multiple threads on Stack Overflow talking about this. Couple of examples can be found below:

Determine if a sentence is an inquiry
NLTK. Detecting whether a sentence is Interogative or Not?

You can also take a look at Natty. It's a bot on Stack Overflow, which reviews answers to old questions. You can see how effective it is in terms of detecting questions (every bot needs training, and Natty has been trained for quite a while). However it is written in Java, it may give you some ideas. Like detecting keywords alongside checking for presence of question mark.


predict future sale from past 3 month data

The Facebook Prophet API might be the place to start. I imagine that the data is daily.

This package is avaiable either for R or Python, and it's very simple to use. You can get predictions easily, and if you have some knowledge about Time Series analysis you can tune some parameters to best fit your solution.

From the link above:

  Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.

Here and Example with R.


Using standard ML models for modeling a derivative when the data set only contains function values

This is actually really easy to implement in any deep learning framework with automatic differentiation capability.

You need two neural networks. One to approximate the $X$, another to approximate the $dX/dt$.

The first step is easy. You have samples of input and target for $X$.
Just follow the normal process to train you network to learn $X$
For the second step is that you don't have the target values (i.e. gradient) directly. The trick is that you already got an approximator for $X$ from the first step, so use that to get your target for the second network.

I am most familiar with pytorch, so I will use it as an example.

import torch
from torch.nn import Sequential, Linear, ReLU
from torch.autograd import Variable
from torch.optim import Adam
import numpy as np

x = np.linspace(0, 10, 101).reshape(-1, 1)
y = np.sin(x) # Using sin as an example

x = Variable(torch.Tensor(x), requires_grad=True)
y = Variable(torch.Tensor(y), requires_grad=False)

f = Sequential(....) # Build f approximator
f_optimizer = Adam(f.parameters())

for epoch in range(1000):
    loss= f(x) - y
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

# Build grad approximator
f_grad = Sequential(....)

# need a new optimizer to train f_grad
f_optimizer = Adam(f_grad.parameters())

for epoch in range(1000):
    # Directly backprop from the output to the gradient of the f
    model(x).backward()
    loss= f_grad(x) - x.grad # x.grad is the grad of f w.r.t x
    f_optimizer.zero_grad()
    loss.backward()
    f_optimizer.step()

f_grad # this is the gradient function you are looking for

(I don't have a machine with pytorch install at the moment, so don't expect the code to run with some debugging)


Random Forest vs K Nearest Neighbor as non linear classifier

It is unusual to have such a bad question as an assignement. Maybe it's just misguided phrasing by the question author, but I'm pretty sure that the vast majority of experts in ML consider the idea of "best ML algorithm in general" as a mistake.
First let me explain why this is a bad question: in ML there is no way to guarantee that a particular algorithm/method performs better than any other on any dataset in general. There can be exceptions under very specific constraints, but there is no algorithm which is universally the best for non-linearly separable classes. There is actually a theoretical result called the No Free Lunch Theorem which is often interpreted as a proof that there cannot be a universally best classification/regression algorithm.
Now that the context of this bad question is clearly established, as a subjective personal opinion my answer would be based on interpreting the question as: which algorithm is the least sensitive to linear separability? In this context I would say kNN because this method is not concerned at all about linear separability: a new instance is classified based on its closest instances in the training data, so it does not rely on separating the data points into groups at all.


You can set the environment variable via python's built in os module.

import os

os.environ["KERAS_BACKEND"] = "tf"   # or theano

Making changes in one notebook does not seem to carry over to other notebooks that are started in the same Jupyter session (run from a single terminal).

Here I try to get the environment variable in notebook 2, and we see it doesn't exist:

Create environment variable in Notebook 1:

Now make the changes in notebook 1 (after also confirming that the environment variable didn't exist - we catch the exception)

And now go back to notebook 2 to see if it picks up on the changed value:

There are also settings that should be read by Keras on import. Have a look at the documentation. If you are on windows, there are some separate steps you should check out there.


How to extract insights from the given data?

The language that are used on review text just happened to be my native language. I can confirm that the review_text at least from what you showed above is a direct translation of the raw text (although I would say the translation is not perfect).

Maybe you can consider making unsupervised model and probably compare between these two, see how much they match (theoretically since they are the same text they should have many overlaps).


The "optimal" stopping point for XGBoost really depends on the data you feed into it.

Using Chunk N and Chunk N+1 for instance, consider the two scenarios:

data in Chunk N and Chunk N+1 is very different - it's likely
that the optimal stopping points for each chunk are very different.
data in Chunk N and Chunk N+1 is very similar - it's likely that the optimal stopping points for each chunk are quite similar.

I would suggest using a different approach - so for each chunk:

use the xgb.cv function to first determine the optimal stopping point
use the results from xgb.cv in xgb.train [i.e. set nrounds = xgb.cv$best.iteration]

As a side-note, have you explored any other techniques / tools to handle large data sets? In R, there are packages designed to tackle similar problems - e.g.bigmemory and ff.


You could give the plotlypackage a try. You basically code up the plot you want first with ggplot2 and then call ggplotly() at the end, which will render an interactive version of it. You get zooming in on mouse selection for example and labels of points on hover. It has a pretty extensive documentation too. I can try and help you out with a specific example if you post some code. Another cool package you can check out is trelliscopejs. hth


No accuracy in Keras RNN Model with Bitcoin Data

Remember, accuracy is a classification measure. That is, it's used to evaluate models that attempt to predict membership in one of a few discrete values. No accuracy in this case means that you haven't predicted and values exactly correctly- a pretty common occurance in regression problems- that is, problems that measure the scale of a phenomenon or "how much" of something happens.

A better metric to evaluate your regression model might be MSE, such as used for your function.

Try:

    # Compile model
model.compile(
    loss='mse',
    optimizer='adam',
    metrics=['mse'],
)


How to construct validation set for time series for NN?

Im new to the topic too but I think the Idea is to create a Train/Test-Set and then take the TrainSet and Split it again in 2 Sets (mostly called Train and Development Set) for example with a KFold-CV. Train your model on the Train Set and improve it with the Developement Set. Then take the final model and use it on the whole trainingset.

The picture give you a clearer idea I think.


Data driven project for a self-catering accommodation business

I would start with a simple decision tree regression with the occupancy rate as target value. By visualizing the tree after training you can see which features impact the occupancy rate. You can do this in Weka with M5P regression: http://weka.sourceforge.net/doc.stable/weka/classifiers/trees/M5P.html


How to connect data-mining with machine learner process

The Data Science Toolkit is a powerful library (or collection of libraries, technically) which are available in a number of languages. For instance, I use the implementation called RDSTK in R.

In the case of your preferred language, Google Go, there's a list of web-related libraries here which looks very useful.


Clustering, Mixed Data Set with Ordinal and Nominal Scale Data

Yes, it causes almost no trouble. The only caution you must have is the possibility of having a regular ordinality where the ordinality is actually irregular. For example, having shirt size 1, 2 and 3 but the size "3" is not three times bigger than size "1".
The dummy variable creation is a very valid approach when having categorical variables in your dataset, not only for clustering but for almost every model you could construct.


How to compare performance between SVM and Keras models

If you are training classifiers, you can check: plain accuracy percentage, F1 score and other metrics (sensitivity, specificity, etc.), you can visualise true vs predicted values with confusion matrices. There's plenty of options.

Please remember that the quality of a model can be assessed only on test data, i.e. data that you model didn't see in training phase.


Three tables: animal, observation, and sibling. The observation has an animal_id column which links to the animal table, and the sibling table has animal_1_id and animal_2_id columns that indicates two animals are siblings for each row.

Even with 5000 animals and 100000 observations I don't think query time will be a problem for something like PostgreSQL for most reasonable queries (obviously you can construct unreasonable queries but you can do that in any system).

So I don't see how this is "relatively painful". Relative to what? The only complexity is the sibling table. In NOSQL you might store the full list of siblings in the record for each animal, but then when you add a sibling relationship you have to add it to both sibling's animal records. With the relational table approach I've outlined, it only exists once, but at the expense of having to test against both columns to find an animal's siblings.

I'd use PostgreSQL, and that gives you the option of using PostGIS if you have location data - this is a geospatial extension to PostgreSQL that lets you do spatial queries (point in polygon, points near a point etc) which might be something for you.

I really don't think the properties of NOSQL databases are a problem here for you - you aren't changing your schema every ten minutes, you probably do care that your database is ACID-compliant, and you don't need something web-scale.

http://www.mongodb-is-web-scale.com/ [warning: strong language]


your grammar is correct!

grammar = """MEDIA: {<DT>?<JJ>*<NN.*>+}
           RELATION: {<V.*>}
                     {<DT>?<JJ>*<NN.*>+}
           ENTITY: {<NN.*>}"""

by specifying

RELATION: {<V.*>}
          {<DT>?<JJ>*<NN.*>+}

you are indicating that there are two ways to generate the RELATION chunk i.e. {<V.*>} or {<DT>?<JJ>*<NN.*>+}

so

grammar = """MEDIA: {<DT>?<JJ>*<NN.*>+}
               RELATION: {<V.*>}
                         {<DT>?<JJ>*<NN.*>+}
               ENTITY: {<NN.*>}"""
    chunkParser = nltk.RegexpParser(grammar)
    tagged = nltk.pos_tag(nltk.word_tokenize("adventure movies between 2000 and 2015 featuring performances by daniel craig"))

    tree = chunkParser.parse(tagged)

    for subtree in tree.subtrees():
        if subtree.label() == "RELATION":
            print("RELATION: "+str(subtree.leaves()))

gives

RELATION: [('featuring', 'VBG')]


@Shagun's answer is right actually. I just expand it!

There are 2 different approaches to your problem:

Graph Approach

As stated in @Shagun's answer you have a weighted directed graph and you want to cluster the paths. I mention again because it's important to know that your problem is not a Graph Clustering or Community Detection problem where vertices are clustered!
Cunstructing a Graph in networkx using the last two column of the data, you can add time spent as weight and users who passed that link as an edge attribute. After all you'll have different features for clustering: the set of all vertices an individual ever met in the graph, total, mean and std of time spent, shortest path distribution parameters, ... which can be used for clustering the user behaviors.

Standard Data

All above can be done by reading data efficiently in a matrix. If you consider each edge for a specified user as a single row (i.e. you'll have MxN rows where M is the number of users and N the number of edges in case you stick with 100 case!) and add properties as columns you'll probably able to cluster behaviors. if a user passed an edge n times, in the row corresponding to that user and that edge add a count column with value n and same for time spend, etc. Starting and ending edges are also informative. Be careful that node names are categorical variables.

Regarding clustering algorithms you can find enough if you have a quick look at SKlearn.

Hope it helped.
Good Luck :)


Yes, this could be possible if your dev/test data comes from the same domain as the training data, in which case word2vec will encounter fewer OOV tokens that mess up the loss.
This could also mean that the benefits of BERT - subword tokenization to handle OOV characters in generalized domains - are lost. If your vocabulary size is small, your word2vec model needs to capture relationships between fewer tokens and can model those relationships better than a subword model which loses the relationships between fixed tokens in your data and instead tries to generalize relationships across >30K subword tokens (in the bert-based-uncased model), which could lead to noise.


The most common way of processing images in python through numpy arrays. Since you have already loaded your image through nibabel, you need to get the data from the image object and then cast it as a numpy array.

import nibabel as nib
import numpy as np

# Get nibabel image object
img = nib.load("path/to/image.nii")

# Get data from nibabel image object (returns numpy memmap object)
img_data = img.get_data()

# Convert to numpy ndarray (dtype: uint16)
img_data_arr = np.asarray(img_data)

By default this should be a 3D numpy array with a shape of (height,width,image). If you want to access one image you can always use PIL. For example to save the first image of the .nii file:

from PIL import Image
img = Image.fromarray(img_data_arr_norm[:,:,0], 'L')
img.save("image.jpeg")


Backpropagation extends to the full model, through all decoder and encoder layers up to the embedding tables.


Custom loss function which is included gradient in Keras

I think it is necessary to perform all operations using the backend versions, allowing Keras to perform backpropagation on every step of the function. You use the common + for example.... try K.add, which should work (based on the available arithmetic operation of the tensorflow backend).

Also, have a look at a related question, where some of the mechanics around creating a custom loss function in Keras are discussed.

The two main loops in your function that compute the gradients should be candidates for vecotisation, where you could compute the differences in one operation. Try something along these lines:

diffs = K.subtract(u_true[i+1, j], u_true[i, j])

and then the quotient in a second operation:

quot = K.divide(diffs, dx)

and then of course the same for v_true and dy.


The assumption regarding IID variables is to ensure a unique solution. By IID, it means they should be uncorrelated. You can't make sure data is identically distributed. Scaling and standardization are the obvious for you to find the solution. One justifies the use by getting satisfied by the solution found by the SVM algorithm (which is just one of the solution it has found).
But, you can the parameters for SVM and use a different kernel altogether. Remove redundant features yourself.
Or if you are looking at stock price prediction, look at Jane Market Street Prediction on Kaggle. I have seen heavy use of trees, mixture models and NNs.


Change aliases of filter items in Tableau

Found it myself.

Go to context menu right clicking to the dimension field.
Go to Aliases... and change the labels.


Data science degree or Computer Science degree?

This is a bit opinion based, but here are my thoughts:

  How relevant is the curriculum? (For example, the UCSD data science major)

Since most DS majors have been developed the past 2-3 years they tend to be well structured and curated. Their subjects are usually a mix of some theoretical background (e.g. stats, linear algebra), some basic programming courses (usually python or R) and some ML topics at a near state-of-the-art level.

However, some face organization issues due to them operating only for a couple of years.

  How reliable is the degree in the upcoming future? (Would I be able to easily get a job like if I had a CS BS)

No one can tell, but the trend seems to be that DS will be adopted universally by tech companies in the future. From what I can tell you'll have no problem at all finding a well paying job right out of college.

  Am I better off doing CS and minoring in stats or something else?

Personally, I think no. DS at the moment is very undermanned in the labor market and demand is rising like crazy. To fill this demand there will be a lot of former programmers with a heavy CS background and years of experience making the transition. This, however, will change when people with DS degrees flood the market and I don't think it will be as easy in the future for CS to shift to DS, as it is now...


Preprocessing to-be-predicted data in ML with R - "learn" and "apply" features

In short, the correct practice is to use the scaling/preprocessing parameters you found and used on your training set, and apply them to your test set.  the motivation for this is that you should be able to deal with someone coming to you with one new observation they want a prediction on.  You will be able to preprocess that single observation using your training set preprocessing parameters.

Ill go through some of the issues you brought up:

Dummy Variables:
This might entail you using setdiff() in R to find what columns are in your training set but not in your test set, then make a column in your test set with a column of zeros.  This is the simplest way to do it, but it is very data/purpose dependent.  You may also want to change this or look for better options depending on what modeling technique you're using.

Min/Max Normalize:
When you scale your training set, save the min and max of each column, and scale your test set using those parameters (NOT the min/max of your test set).

PCA:
When you perform PCA on your training set, you are transform your data based on rotations that were calculated on your training set.  You want to perform those pre-calculated rotations on your test set.  Check out ?predict.prcomp.  This will show you how to apply your rotations from your training set onto your test set.


Does the SVM require lots of features most of the time?

I stumbled upon some rules of thumb for dataset sizes, but not specific to the ratio of features / samples.

Also, it seems easier to guess to the "positive" side, i.e. "might it work?" then "might it fail?".
I witnessed both small datasets (<1000) that functioned well with even 20-50 features, and huge datasets (~10M) that functioned well with very few (<5) features.

The stage in the processing pipeline to tackle this, is the feature selection one. I'd start with gathering all of your make-sense features and playing with different k-best values for the feature selection object to find. (ranges like: 10, 50, 100, ..)

You didn't mention a specific research environment, but I'd like to suggest scikit-learn sklearn.feature_selection.SelectKBest running inside a GridSearchCV pipeline. This way the grid object will help you choose the closest k-best featues out of a given list. Something like:

from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.svm import LinearSVC

pipe = []
pipe.append(SelectKBest())
pipe.append(LinearSVC())

est_pipe = make_pipeline(*pipe)
print 'all possible grid params: {0}'.format(est_pipe.get_params().keys())
param_grid = {
  # some svc grid params..,
  'selectkbest__k': [5, 10, 50, 100]
}

grid = GridSearchCV(est_pipe, param_grid=param_grid)
grid.fit(X, y)


Can categorical features be linearly distributed?

I am not sure what you mean with a linear distribution, but you can use a categorical features with a linear regression (that is, if your dependent variable is continuous). One thing to make though is to encode the categorical feature as such, e.g. use one hot encoding.


A surrogate model is an approximation model for a given function.
The original function is generally a black box function that we can sample from and based on the samples we can optimize our surrogate model to approximate the behaviour of the original function.
A surrogate model can be a neural network, an ensemble method, a gaussian process which can be interpreted and used later for explainability.


How to discard trash topics from topic models?

I assume you have inspected your topic-by-word matrix, and that is why you say you have 'trash' topics.  If no words are strongly associated with a given topic, it may not be a useful topic.  If you find many of these, you can re-run the experiment with fewer topics.

If instead there are words strongly associated with these topics, you just don't see the cohesive meaning behind the words associated with the topic, then these are not exactly "trash" topics.  They are still capturing the latent structure of the data, it just isn't something easily deemed a "topic" in the traditional sense of the word.

Another thing to remember is that the hyperparameters also control the shape of your resulting distributions.  This answer does a great job explaining alpha and beta.


When you are in the inference phase, there is no loss to be calculated unless you are using a validation set to fine tune your model on.
Using two models with shared layers in between is fine yet I suggest to start training the model and don't waste your time writing perfect code.


Shouldn't an autoencoder with #(neurons in hidden layer) = #(neurons in input layer) be "perfect"?

I think a part of your problem could be related to the activation function - you're probably using tanh or relu for the hidden layer. If the middle layer did not have an activation function, the network would learn to simply pass the information forward - ideally, there would be a weight of 1 between every input and a corresponding hidden layer neuron and the other weights would be 0, completely replicating the information. But since you have activation functions, this replication wouldn't work, the activation would distort the result...

The activation function, which breaks the linearity of the problem is what makes NN so powerful, but it also causes these problems.. they are also the cause that NN have multople local minimum, as Lucas said, so the NN can get stuck in a "good enough" solution.

In short: remove the activaction function (make it linear) and it should work...


How to do feature analyzing : pandas groupby(). mean

Means are calculated just to have an idea of the relationship. For more definite analysis, one can use simple statistics to know relationship between two variables. The test to be applied depends on type of variables:

If both are numeric: Correlation (Pearson or Spearman)

If both are grouping (nominal) variables: Chi-square test
   Fisher exact test can also be used if only 2 groups in each variable.

If one is grouping and other numeric:
   if only 2 groups: Student t-test or Mann-Whitney U test
   if more than 2 groups: ANOVA or Kruskal–Wallis test

Means of different groups will give you only partial information about relationship. However, it may be sufficient to know if the feature is likely to be important for prediction. As a method of feature selection, if groups' mean do not differ significantly, this feature may be dropped from analysis.


First layer weights for transfer learning with new input tensor in keras.applications models?

The first layers are convolution and pooling ones:

For the convolutional layers, the only weights are the kernels and the biases, and they have fixed size (e.g. 3x3x3, 5x5x3) and do not depend on the input tensor shape.
The pooling layers do not have weights at all.

That's why you can reuse the weights independently from the input tensor shape.

With dense layers (i.e. the final layers), you need shapes to match, so you cannot reuse them if they do not.


Split the data between the Training Data and Test Data using sklearn

What you are looking for is called Stratified sampling

From this CrossValidated question, we have a short explanation

  Stratified sampling aims at splitting one data set so that each split
  are similar with respect to something. In a classification setting, it
  is often chosen to ensure that the train and test sets have
  approximately the same percentage of samples of each target class as
  the complete set.

To get a stratified split in ScikitLearn, you just need to edit this part of your code

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y ,
                                                  test_size = 0.30 ,
                                                  random_state = 0,
                                          ----->  stratify = y)

This will automatically split your dataset in train and test but also keep the same proportion of positives and negatives as the original dataset.

So, if you have a dataset where the positive class is 70% of the records and negative class 30%, after a stratified split, both train and test will have the same 70-30 distribution.


Accuracy graph of binary classification by CNN

There is no way to definitely note how many epochs is required. Your accuracy graphs may be unstable for several reasons such as -

Irregular data distribution
Faulty model
Large batch size (as mentioned in the comments)
Overfitting and underfitting (in this case im not too sure which one)

You can fix it through the following methods -

Analyse your data well and see if one class is heavily grater than other.
Make sure all duplicate images are removed.
Try different models, start with pre-trained. Shouldn't take much compute power.
Approach transfer learning (neural networks)
Adjust learning rate (as mentioned above)
General hyperparameter tuning.

if the dataset is from somewhere public, see the models and the approaches others have used. You might see where your model actually lacks.


You are probably trying to load word vectors that are shared in the fasttext.cc website or are trying to load using models that are trained with the latest one on master. There is an issue in github with a lot of comments. One solution is to try to load from the text files the code for which is given here. I was able to work around this using java but had to train it in java only. Take a look at the examples in this package.

https://github.com/vinhkhuc/JFastText/tree/master/examples/api


Feature extraction in time series data as input variables for machine learning algorithms

You don’t need to extract any features. You should feed the pressure data to a LSTM network. LSTM is a type of recurrent ANN that can be used to predict time series.


Several questions and thoughts come to mind.

What languages are in the corpus?  This may impact what services you can leverage.
I like the "Sentiment Idea" for languages that are supported natively by the services you mentioned.
I would keep the "Language Idea" as the last resort as it is possible that the translation engine may not capture the sentiment of the original language.
Mechanical Turk would be a good option if you can limit the number of samples sent for classification.  For each language, you could try clustering the passages by, for example, word count into 30 (you pick) clusters and then perform sampling within the clusters to identify candidate passages to send to Mechanical Turk. I have used this technique to try to sample across the vector space more uniformly.

Don't dismiss oW_'s comment.  You should seriously consider breaking the articles into paragraphs. You can always aggregate the paragraph scores to the article, but it's hard to get one representative score as the text gets longer.

HTH


Root Mean Squared Error (RMSE) - significance of square root

It depends on what you are using the RMSE for. If you are merely trying to compare two models/estimators, then there is no significance to the square root. However, if you are trying to plot the error in terms of the same units as you made the measurements/estimates, then you need to take the square root to transform the squared units to the original units (much like variance vs standard deviation)


You don't need word embeddings. Actually, in neural machine translation is frequent not to use them and simply train the embeddings along with the task.

Nevertheless, word embeddings work as a data augmentation technique, as you normally use a different (and much larger) dataset to train them, so they can be useful when you don't have much training data.

Therefore, the decision of using pre-trained word embeddings or not should be driven by the available data.


Noisy data is meaningless data.

• It includes any data that cannot be understood and interpreted correctly by machines, such as unstructured text.

• Noisy data unnecessarily increases the amount of storage space required and can also adversely affect the results of any data mining analysis.

• Noisy data can be caused by faulty data collection instruments, human or computer errors occurring at data entry, data transmission errors, limited buffer size for coordinating synchronized data transfer, inconsistencies in naming conventions or data codes used and inconsistent formats for input fields( eg:date).

Noisy data can be handled by following the given procedures:

Binning:
• Binning methods smooth a sorted data value by consulting the values around it.

• The sorted values are distributed into a number of “buckets,” or bins.

• Because binning methods consult the values around it, they perform local smoothing.

• Similarly, smoothing by bin medianscan be employed, in which each bin value is replaced by the bin median.

• In smoothing by bin boundaries, the minimum and maximum values in a given bin are identified as the bin boundaries.

• Each bin value is then replaced by the closest boundary value.

• In general, the larger the width, the greater the effect of the smoothing.

• Alternatively, bins may be equal-width, where the interval range of values in each bin is constant.

• Binning is also used as a discretization technique.

Regression:
• Here data can be smoothed by fitting the data to a function.

• Linear regression involves finding the “best” line to fit two attributes, so that one attribute can be used to predict the other.

• Multiple linear regressionis an extension of linear regression, where more than two attributes are involved and the data are fit to a multidimensional surface.

Clustering:
• Outliers may be detected by clustering, where similar values are organized into groups, or “clusters.”

• Similarly, values that fall outside of the set of clusters may also be considered outliers.


Algorithms, techniques, papers for regression with vector output

Perhaps you could frame this as predicting a distribution (I hope I haven't mis-understood your question!).

You could try using Markov Chain Monte Carlo (don't be scared by the name!). What this essentially does, it to sample a posterior distribution from your input data and see how likely it was that is is related to your prior distribution (your data). If it thinks the sample fits with your data, it keeps puts it records that sample, otherwise it discards it. It keeps searching like this until some stop-criterion is reached. All the accepted points that were recorded then represent your target distribution, which can be N-dimensional (and so hopefully matches your use-case).

Here is a quick video introduction to the topic in general, and here is a deeper video explanation. Here is a more thorough text-based introduction.

You can get started doing this with packages like:

PyMC3 (code and docs) if you use Python. There are interactive examples here, e.g. the PyMC3_tips_and_heuristic notebook too.
If you use R, then there is the mcmc package.


Benchmarking is the process of comparing your result to existing methods. You may compare to published results using another paper, for example. If there is no other obvious methodology against which you can benchmark, you might compare to the best naive solution (guessing the mean, guessing the majority class, etc) or a very simple model (a simple regression, K Nearest Neighbors).  If the field is well studied, you should probably benchmark against the current published state of the art (and possibly against human performance when relevant).


I'll go through your question one by one:

  My initial assumption was that the point came at around epoch 28 since the validation error almost remains constant and then increases ever so slightly. However, I still wanted to know if this is fine or the model is indeed overfitting.

Commond knowledge from DS/ML handbooks says you're right. You should keep training until validation Loss doesn't start going up. Please keep in mind that some overfitting is inevitable (a good model is not one that eliminates overfitting, that would be impossible, but a model that is able to keep it at bay).

  Another concern I have is that the training and validation curves are very very close to each other. Is this an expected behavior?

You should keep training even after validation loss gets higher than training loss, and stop only when validation loss starts growing. Ideally, you should have a very slight overfitting, with validation set just above training, and its slope is flat. That means: I can't train my model more than that without making things worse.


How do I build a recommend system based on user's past purchases?

Content-based filtering is relevant whenever you have features to learn from (features about the products \ user). Sound like this is not your case as you mostly rely on the historical choices \ preferences of your user base. Sounds like a classical use case for Collaborative-Filtering (CF) based approach.
There are many possible CF implementations, most common ones are:
Memory-based: User to User \ Item to Item based similarity
Model-based: Matrix factorization
Of course, there are much more implementation options but these ones could give you a good starting point


Calculation of relative standard deviation with a custom function in R

I find these sorts of operations much easier to do if the input data is in a long instead of a wide format, this has the additional advantage of being faster on larger datasets. I first transform the dataset such that each row is a sample for a specific element and the columns hold the concentration and standard deviation for that combination. I then use the same logic with dplyr::mutate() to create the new rsd column.
data %>%
  gather(type, value, -Sample) %>%
  separate(type, c("element", "type"), sep="_") %>%
  spread(type, value) %>%
  mutate(
    rsd = ifelse(
      is.na(conc) | is.na(std) | conc == 0,
      NA,
      ifelse(
        std >= conc,
        100,
        abs(100 * std / conc)
      )
    )
  )

# A tibble: 50 x 5
   Sample element    conc     std    rsd
   <chr>  <chr>     <dbl>   <dbl>  <dbl>
 1 A      Cu      104.    0.784   0.753
 2 A      Fe        0.729 0.00637 0.874
 3 A      K       284.    1.54    0.544
 4 A      Mg       21.3   0.100   0.469
 5 A      Mn      441.    4.59    1.04
 6 B      Cu       32.4   0.774   2.39
 7 B      Fe        0.783 0.0148  1.89
 8 B      K       452.    0.474   0.105
 9 B      Mg       43.2   0.00714 0.0165
10 B      Mn      727.    0.232   0.0319
# … with 40 more rows


How to Make Meaningful Conclusions here?

It is very difficult (arguably impossible, if you want to get philosophical about it) to be absolutely, 100%, for sure about anything. For this reason, we talk in terms of probability/significance/confidence sets. A refresher on statistical hypothesis testing might help.
So to answer this type of question, people would usually try to attain a well-agreed-upon p-value for their problem, below which we can reject the null hypothesis, and above which we accept the null hypothesis. The null hypothesis in this case is that the bone disease is not a symptom of COVID, and the alternative would be that it is a symptom.

Edit for demonstration as requested in the comments:
Firstly, these methods are purely for association analysis, not for proving whether bone disease is a symptom of COVID—again, correlation $\ne$ causation!
Given that we are dealing with binary variables, you could use the Phi coefficient to measure the association of bone disease with COVID.
consider this contingency matrix:
|----------|---------|---------|-----------|
|          | Bone =0 | Bone =1 | total     |
| COVID =0 |    A    |  B      | I=A+B     |
| COVID =1 |    C    |  D      | J=C+D     |
|------------------------------|-----------|
|  total   |  K=A+C  |  L=B+D  | E=I+J+K+L |
|----------|---------|---------|-----------|

Which we can represent visually via a Venn diagram:

then you can calculate
$ \phi=\frac{AD-BC}{\sqrt{IJKL}}=\frac{ED-IK}{\sqrt{IK(E-I)(E-K)}} $
This is related to the Chi-squared test: $ \phi= \sqrt{\frac{\chi^2}{n}} $ So you can easily retrieve the p-value given that you know the degrees of freedom (in this case, it is just 1).
And you interpret it similar to the Pearson correlation coefficient (both from the same statistician—Pearson).

So given $7.5\%$ of the total has the bone disease, and $25\%$ of COVID patients have it, we can construct our contingency table in terms of $N$ (the number of samples). Where $Q$ is the percent of people with COVID:
$ I=N(1-Q)= $ number of people without COVID
$ J=NQ= $ number of people with COVID
$ K=N(1-0.075)=$ number of people without bone disease
$ L=N(0.075)=$ number of people with bone disease
We know that 25% of people with COVID also have the bone disease, so $D=J(0.25)\Rightarrow$
$\phi=\frac{E(0.25\cdot J)-IK}{\sqrt{IK(E-I)(E-K)}}$
Finally, we can calculate:
$\phi=\frac{0.25\cdot EQ-N(1-Q)(1-0.075)}{\sqrt{(1-Q)(1-0.075)(E-N(1-Q))(E-N(1-0.075))}}$
From here, we can find the associated p-value easily by looking it up in a Chi-Square p-value table, such as this one: http://chisquaretable.net/. Then you can accept/reject the null hypothesis given your predefined $\alpha$ threshold.


Try updating to Tensorflow 2.0. With it you can fiddle with your network, when it's running.


How to determine the function is linear in linear regression problem?

A function $x = (x_1, \dots, x_n) \mapsto f(x_1, \dots, x_n) \in \mathbb{R}$  is linear if and only if for every $a,b \in \mathbb{R}$ it holds that $f(a (x_1, \dots, x_n) + b (\xi_1, \dots, \xi_n)) = a  f(x_1, \dots, x_n) + b f(\xi_1, \dots,\xi_n)$.

In your second example try computing $f(a x + b \xi)$ to see that the last term $w_3 x_1 x_3$ will make $f(a x + b \xi) \neq a f(x) + b f(\xi)$.

The third  example has a coefficient which is expressed as a product, but the function remains linear (just rename $w_3 w_4$ to $w$).


I would try to analyze and solve one or more of the problems published on Kaggle Competitions. Note that the competitions are grouped by their expected complexity, from 101 (bottom of the list) to Research and Featured (top of the list). A color-coded vertical band is a visual guideline for grouping. You can assess time you could spend on a project by adjusting the expected length of corresponding competition, based on your skills and experience.
A number of data science project ideas can be found by browsing Coursolve webpage.
If you have skills and desire to work on a real data science project, focused on social impacts, visit DataKind projects page. More projects with social impacts focus can be found at Data Science for Social Good webpage.
Science Project Ideas page at My NASA Data site looks like another place to visit for inspiration.
If you would like to use open data, this long list of applications on Data.gov can provide you with some interesting data science project ideas.


Is it possible to train the neural networks for programming

It depends on what kind of functionality you would want to achieve. In essence, a trained neural network is a mapping function, which takes some input and produces some output, which then may be variously interpreted (for example using a certain threshold for classification).

Therefore, if the desired functionality of your program would be coming up with a computational function, you could treat your trained neural network as the program itself. Multiple machine learning libraries allow that easily, producing a class instance which contains the trained neural network inside and allows you to call it with given parameters just like any other function.

However, I assume that you are looking for a way to output complex computer code with control structures, etc. - that unfortunately is very difficult to achieve, since you'd need an effective way to interpret the neural network outputs as code - not just the commands, but their structure and ordering as well. Such interpretations would be so complex that they'd make it very difficult for any Neural Network training method to converge to anything sensible.

To mitigate this problem, you can try machine learning techniques which manipulate and optimize the code "directly". The most famous of such is Genetic Programming - in which a computer program in encoded as a set of "genes" which are then improved upon through genetic algorithm techniques (crossovers which recombine large parts of code and mutations which randomly change small parts of the code). This has been successfully applied to some programming problems such as the Towers of Hanoi (see http://www.oakgp.org/towers-of-hanoi ).


Will my neural network get lazy if I give it an "easy" feature?

Interesting theoretical question. While I cannot answer this with 100% certainty, my own intuition and experience say:

1) I would be very surprised if this is not the case. If we look at different problems, where we have one very important feature that holds most of the information but some other features with some signal in there the model generally still improves by adding these lower value features. That is exactly what you are doing here, you have the MNIST pixels that individually hold little value but combined tell you quite a bit. The value of these pixels goes down quite a bit by adding such a strong feature, but the value is absolutely not reduced to 0. I think in theory this should lead to a strictly better model.

2) I think it will take around the same time to converge on average. Fitting to the one-hot encoded feature will be very fast which means most of the time will be spent on fitting to the features, which is a similar problem as without the feature. However, instead of going from 10% to 99% accuracy, we are now kind of going from 95% accuracy to 99.7% accuracy.

I'm less certain about the second question but I think this is what you would find if you do the (interesting) experiment. Let us know if you do implement it!


Residuals in a gradient boosted classification

It's a similar trick to logistic regression. We use an unbounded value that we can map to a probability by using the sigmoid function. Only at the end of the gradient boosting tree model we map it to a probability. The loss function used for deciding the weights of the terminal nodes is adapted from the normal sigmoid loss to not have to map directly to probabilities. I cannot find it at the moment but it should be easy to derive.

EDIT: I found the post that I read this from the other day, it's over at stats stackexchange: https://stats.stackexchange.com/questions/204154/classification-with-gradient-boosting-how-to-keep-the-prediction-in-0-1


Constructing an image from a dense layer output

Concatenating the two output layers is not such a pain as you might think. For starters, there are a number of things that can possibly go wrong with the model. Let's look at them one by one:

1) If the loss curve is unstable or is not coverging, try using batchnorm in the intermediate dense layers. Since there a lot of dense layers, it can lead to overfitting as well so try using dropout too. Experiment with batchnorm and dropout separately as well as together to see which one helps.

2) Check the range of the input and output of your decoder network: Normalise the data to be within 0 and 1 by simply multiplying the RGB values with 255. Use this normalised value as the true target value if your output image is same as input image or else use the normalised target image as your target output.  Similarly, to obtain the output value between 0 and 1, simply clip the output from relu to 1 and store it in a float variable as its a fraction. Use this float variable for computing the loss. Multiply it with 255 and store it in integer variable for visualisation purpose.

If these don't work you might want to look at other upsampling methods out there like: Deconvolution layers and Pixel Shuffle layers. You might also have a look at the decoder of generative models like GANs, specifically the CGAN (conditional GAN) which do a similar thing of merging information from an image with some known prior.


Data sets that have strings and numerical data all in one column

Normally pandas should identify numeric types automatically. If it doesn't in your case, there seems to be a formatting issue.

Instead of reading everything into string, I'd rather try to enable pandas to read the types directly in the correct form. First I'd try to pass the column types in a dict to dtype, as in:

pd.read_csv(file_name, dtype={'int_column_name': 'int32', 'float_column_name': 'float32'})

If that is not applicable, because the format is not recognized by pandas automatically, you might want to try some other options.

E.g. if it's just the decimal point or the thousand separator that is different in your data, you can set it over the corresponding keywords in read_csv (thousands and/or decimal).

If that isn't sufficient because you have some special formatting etc., you can also pass your own converter to parse the string into the data type you require for the column, like this as an example of how you could parse numeric values with a currency unit:

import re
mon_re=re.compile('(?P<value>[0-9.]*)([^0-9].*)?')
def strip_off_currency(currency_string):
    m=mon_re.match(currency_string)
    if m is not None:
        return np.float32(m.group('value'))
    else:
        return np.NaN

pd.read_csv(file_name, dtype={'int_column_name': 'int32', 'float_column_name': 'float32'}, converters={'monetary_ammount': strip_off_currency})

The reason why I would prefer this way over reading everything into memory and processing it there is, that it needs less memory and probably also is faster if you process the values just once (at least if you don't need to pass converters).