Successful ETL Automation: Libraries, Review papers, Use Cases

I don't think you'll find anything that checks all of your requirements, but here are some things to look at:

Automated ETL mapping: There is a tool called Karma started by a team at USC's Information Sciences Institute. It learns from your ETL mappings and helps automate future mappings. It's the only open source tool I'm aware of that helps automate the ETL process, but I would be very interested if there are others out there.
Large scale ETL: There are many many tools you could look at for the scalability you are looking for. I can personally recommend looking at Storm and Spark. Storm is excellent for stringing a connection of processing steps together that, given enough resources, can compute in near real time on streaming data. Not too dissimilar Spark has a streaming component with a similar use case, but standard Spark may fit your needs better if the data you are needing to ETL is a fixed set to be processed once.
Data storage: You may also need to consider where all this data will live during the ETL lifetime. You may need something like Kafka to deal with large streams of data. Or maybe HDFS to store a static collection of files.


Are there any python libraries for sequences clustering?

Is there libraries to analyze sequence with python

You can take a look at here. You can also use TensorFlow if your task is sequence classification, but based on comments you have referred that your task is unsupervised. Actually, LSTMs can be used for unsupervised tasks too depending on what you want. Take a look at here.

  And is it right way to use Hidden Markov Models to cluster sequences?

Markov hidden models are those that your current state does not depend on all previous states. If you your task has longterm dependencies, you can use LSTM networks. If your data does not have longterm dependencies you can use simple RNNs.


How to create an ensemble that gives precedence to a specific classifier

Feature-Weighted Linear Stacking might be what you are looking for.

  FWLS combines model predictions linearly using coefficients that are
  themselves linear functions of meta-features.

In your example you can use the meta-feature "Does A label the example as True?"


I'm assuming you understand the original gan paper.

So there are 2 distribution at the start - first distribution that the original images follow and a random distribution that the fake images follow.

Discriminator task is to figure out which image came from what distribution, whereas generator is trying to learn the real distribution and make the random distribution similar to the real one.

Now giving image labels as input is like giving extra bit of information about the distribution. This doesn't change the game, it's the same unsupervised one.

There will two consequences of adding this extra bit of information :

Even the random distribution that the fake images follow will have some pattern. Hence convergence will be faster.
You can control the output of generator at test time by giving label for image you want to generate.


In images, some frequently used techniques for feature extraction are binarizing and blurring

Binarizing: converts the image array into 1s and 0s. This is done while converting the image to a 2D image. Even gray-scaling can also be used. It gives you a numerical matrix of the image. Grayscale takes much lesser space when stored on Disc.

This is how you do it in Python:

from PIL import Image

%matplotlib inline

#Import an image
image = Image.open("xyz.jpg")

image

Example Image:

Now, convert into gray-scale:

im = image.convert('L')

im

will return you this image:

And the matrix can be seen by running this:

array(im)

The array would look something like this:

array([[213, 213, 213, ..., 176, 176, 176],
       [213, 213, 213, ..., 176, 176, 176],
       [213, 213, 213, ..., 175, 175, 175],
       ...,
       [173, 173, 173, ..., 204, 204, 204],
       [173, 173, 173, ..., 205, 205, 204],
       [173, 173, 173, ..., 205, 205, 205]], dtype=uint8)

Now, use a histogram plot and/or a contour plot to have a look at the image features:

from pylab import *

# create a new figure
figure()
gray()
# show contours with origin upper left corner
contour(im, origin='image')
axis('equal')
axis('off')

figure()

hist(im_array.flatten(), 128)

show()

This would return you a plot, which looks something like this:

Blurring: Blurring algorithm takes weighted average of neighbouring pixels to incorporate surroundings color into every pixel. It enhances the contours better and helps in understanding the features and their importance better.

And this is how you do it in Python:

from PIL import *

figure()
p = image.convert("L").filter(ImageFilter.GaussianBlur(radius = 2))
p.show()

And the blurred image is:

So, these are some ways in which you can do feature engineering. And for advanced methods, you have to understand the basics of Computer Vision and neural networks, and also the different types of filters and their significance and the math behind them.


How to deal with a biased feature in Machine Learning (date)

What you are describing are censored observations in survival analysis. Indeed, those who are still alive will obviously die but just at a later, observable date in a time frame past our study date. By counting them as survived on a logistic regression we will clearly bias our model to favor survivals for those particular horses, unrealistically.

Hence, you have survival data and should approach it like so. The Cox proportional hazards rate model allows you to relate hazard rate functions to predictors, and while many use linear models for the "predictor" part there are other ML methods like xgboost and random forests that have the ability to return Cox PH models. These models allow for censored observations as you describe.


It is overfitting if you have an accuracy on training of 100%, but the test accuracy would be 5%. That's overfitting.

In your case, there is a good match between training and testing accuracy. To check for any sampling bias, do a K-fold cross validation with the same hyper-parameters.


Should I consider feature scaling for all gradient descent based algorithms?

About the tips regarding plot cost vs. iteration, they are generally applicable to gradient descent approaches, including deep learning, where hyperparameter tuning (e.g. learning rate) is crucially important.

About the proper input scaling, it is not only related to the machine learning approach, but to the specific problem under consideration. Sometimes machine learning algorithms rely on distances to compute the similarity between individuals. Scaling changes some of these distances. In these cases, the resulting distance after scaling should be assessed to check whether it is more appropriate than without scaling. Here you can find examples for clustering. For some machine learning algorithms, you need standardized features, e.g. regularized linear/logistic regression. For most optimization-based machine learning algorithms, it makes sense to have feature scaling. On the other hand, there are problems where scaling doesn't even make sense (e.g. discrete input problems, like token-based natural language processing).


Why does a horizontal cross section of a squared error surface yield ellipsis?

Hinton refers to a shape like this, where clearly the horizontal cross-sections are ellipses (or circles).

(In 3D space things can be perpendicular to each other in two ways)
So no matter how steep the gradient at one point is, we need to travel perpendicular (in one way) to the contour lines in order to reach the minimum of the bowl. Think about it :D


What Introductory Statistics book would you recommend?

I would recommend Elements of Statistical Learning, by Trevor Hastie and Rob Tibshirani. That link gets the book directly from their Stanford website.

In addition, there are also accompanying videos (they follow the structure of a similar book), which are really helpful!

They offer a simpler set of books and online courses, which you can find listed here.


Are there enough databases for all learning tasks?

A naive observer might be tempted to think that with the vast amounts of data being collected, there's enough to answer all learning tasks. But this is misleading at least for the following reasons:

There's no such thing as "all learning". Humanity has not yet perceived what could be the limits of knowledge. With the boldly stated ambitions of AI to match human intelligence, this also transfers to the lack of limits to what could machine learning aim to learn.

One feature that a lot of data scientists downplay, but that comes as heritage from traditional science fields, is that sound research starts with identifying what you want to achieve and focus data collection on that objective. In other words, in an ideal data analysis setup, the analyst would have a say in how fit-for-purpose data is collected.

A big part of machine learning is preprocessing data in order to clean, normalise or annotate it. This could also be done to pre-existing data as a way to adapt it for certain - previously unenvisaged - purposes. If the data is not collected with attention to how it is going to be used, it would need more preprocessing, and thus - generally speaking - would be less reliable due to the accumulation of steps where noise could be introduced.

These three arguments are only a scratch on the surface of the topic, but they come to show that even if it is sometimes useful, the widespread data collection does not always respond to all questions one might want to ask. Even if in machine learning it sounds a bit more complicated, it all boils down to this amazing quote from Alice in Wonderland:

“Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don't much care where.
The Cheshire Cat: Then it doesn't much matter which way you go.
Alice: ...So long as I get somewhere.
The Cheshire Cat: Oh, you're sure to do that, if only you walk long enough.”

Examples:

Comprehending information from random receipts or invoices.
Automatically judging football situations.
Telling Google (think captcha) if that thing on the image is a street sign, a bridge or a number.


Doing some research, I found a paper which proves that sorting can be done with at most 3 layers, and that their solution is optimal if you restrict the size of the network to be polynomial w.r.t. to the number of input numbers:

Depth Efficient Neural Networks for Division and Related Problems, see Theorem 7 on page 955 (page 10 in the PDF).


Combine RepeatedStratifiedKFold and crossval

I found a recent paper dealing with repeated cross-validation:

On estimating model accuracy with repeated cross-validation

It states: If $A_i$ is the result returned by a single k-fold crossvalidation, then
the cross-validation result for the whole repeated cross-validation (n repeats) is: $A = \frac{\sum_i A_i}{n}$.

So to calculate the final score, we need a "chunks" function: (Code from stackoverflow)

def chunks(lst, n):
    means = []
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        means.append(np.mean(lst[i:i + n]))
    return means

Then the final score can be calculated by:

np.mean(chunks(scores, n_splits))


How to split data frame into groups, combine rows

If I understood you correctly, you want to vertically split your data frame and horizontally union the resulting data frames. Here is an example for what I would do in this case:

#assuming exemplary dataset
dataset <- data.frame(matrix(rnorm(n = 27 * 15 * 10), nrow = 10));
colnames(dataset) <- paste(
  as.character(sites18)
  ,rep(1:27, length.out = 27 * 15)
  ,sep = "_");
str(dataset);

#create list of data frames using a vertical split
(list_df_by_group <- lapply(
  names
  ,function(name) dataset[, paste(rep(name, 27), 1:27, sep = "_")]));
names(list_df_by_group) <- names;
str(list_df_by_group);

#horizontally union data frames
final_dataset <- data.frame(matrix(ncol = 28, nrow = 0));
colnames(final_dataset) <- c("group", as.character(1:27));
for(name in names){
  colnames(list_df_by_group[[name]]) <- as.character(1:27);
  final_dataset <- rbind(
    final_dataset,
    cbind(
      data.frame(group = rep(name, nrow(list_df_by_group[[name]])))
      ,list_df_by_group[[name]]));
}
str(final_dataset);


Generating Dinosaur names with Tensorflow RNN

Check out this answer:

https://stackoverflow.com/a/60230236/12642230

Alternative Solution:

Tensorflow provides a method pad_sequences() to do that:

https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences

The default value of padding is 'pre', you might wanna change that to 'post' to do what you want, along with providing the maximum length, which is 26 in your case. You also would need to add a special padding character to you dictionary of characters to indices, and use its index to provide padding value for the method.


Clustering geo location coordinates (lat,long pairs)

K-means is not the most appropriate algorithm here.
The reason is that k-means is designed to minimize variance. This is, of course, appearling from a statistical and signal procssing point of view, but your data is not "linear".
Since your data is in latitude, longitude format, you should use an algorithm that can handle arbitrary distance functions, in particular geodetic distance functions. Hierarchical clustering, PAM, CLARA, and DBSCAN are popular examples of this.
This recommends OPTICS clustering.
The problems of k-means are easy to see when you consider points close to the +-180 degrees wrap-around. Even if you hacked k-means to use Haversine distance, in the update step when it recomputes the mean the result will be badly screwed. Worst case is, k-means will never converge!


train_test_split() error: Found input variables with inconsistent numbers of sample

Your independent variables need to be passed to the classifier's fit as a 2D array, but yours is 1D; you can use numpy's reshape: https://stackoverflow.com/questions/30813044/sklearn-found-arrays-with-inconsistent-numbers-of-samples-when-calling-linearre

Separately, I find it odd that you are trying to use quality to predict alcohol content, for which you should be using regression instead of classification.  Perhaps you meant to switch the inputs to train_test_split?


How to apply TFIDF in structured dataset in Python?

You could use pandas pivot_table() to transform your data frame into a count matrix, and then apply sklearn TfidfTransformer() to the count matrix in order to obtain the tf-idfs.
import pandas as pd
from sklearn.feature_extraction.text import TfidfTransformer

# input data
df = pd.DataFrame({
    'RepID': [1, 1, 1, 2, 2, 5684, 5684, 5684],
    'Word': ['Doctor', 'diabetes', 'patient', 'patient', 'arm', 'cough', 'Xray', 'Covid'],
    'BOW': [3, 4, 1, 2, 7, 9, 3, 5]
})

# count matrix
df = pd.pivot_table(df, index='RepID', columns='Word', values='BOW', aggfunc='sum')
df = df.fillna(value=0)
print(df)
# Word   Covid  Doctor  Xray  arm  cough  diabetes  patient
# RepID
# 1        0.0     3.0   0.0  0.0    0.0       4.0      1.0
# 2        0.0     0.0   0.0  7.0    0.0       0.0      2.0
# 5684     5.0     0.0   3.0  0.0    9.0       0.0      0.0

# tf-idf transform
X = TfidfTransformer().fit(df.values)
print(X.idf_)
# [1.69314718 1.69314718 1.69314718 1.69314718 1.69314718 1.69314718 1.28768207]


Here you actually do not have a system of linear equations that needs to be seen at a whole and solved together. Here you have 3 independent equations, each of them with infinite valid answers. So:

$\begin{bmatrix}
a_{1} & a_{2} & a_{3}\\
a_{4} & a_{5} & a_{6}\\
a_{7} & a_{8} & a_{9}
\end{bmatrix}\times \begin{bmatrix} x_{1}\\ x_{2} \\ x_{3} \end{bmatrix} = \begin{bmatrix} b_{1}\\ b_{2} \\ b_{3} \end{bmatrix}$

equals solving

$a_{1}x_{1} + a_{2}x_{2} + a_{3}x_{3} = b_{1}$

and

$a_{4}x_{1} + a_{5}x_{2} + a_{6}x_{3} = b_{2}$

and

$a_{7}x_{1} + a_{8}x_{2} + a_{9}x_{3} = b_{3}$

independently.

Each of them in for example in $\mathbb{R}$ has infinite right answers. For instance in $a_{1}x_{1} + a_{2}x_{2} + a_{3}x_{3} = b_{1}$ you just need to randomly choose two of $a_{i}$s and simply get the last one. For example for known $x$ vector $\begin{bmatrix} 1\\ 2 \\ 3 \end{bmatrix}$, $b$ vector $\begin{bmatrix} 4\\ 5 \\ 6 \end{bmatrix}$ and $a_{1}=1$ and $a_{2}=2$, we get $a_{3}$:

$5 + 3\times a_{3} = 4$

so

$a_{3}=-\frac{1}{3}$!

Same procedure applies to other two equations as well. Randomly choose two $a_{i}$s for each equation and get the last one.

PS: I feel either something is missing in your question or I did not understand well. In either cases, please drop a comment here and I will update the answer.


What is the difference between "offline trained model" and "pretrained model"?

Every pre-trained model is an offline-trained model, but not the
  reverse.

Offline training is any training that leaves the model unchanged when new observations arrive, i.e. it has an end. Online training constantly updates the model with the help of new, incoming observations without using the previous training points, although, having a limited memory of previous samples compared to all seen samples is OK. Therefore, training offline periodically on all the training points, no matter how frequent, is not the same as online training.

We can use offline training for a model that supports online training, however, to train online, the model must allow such training. For example, common implementation of SVM cannot be trained on N new data points without re-using the previous training points. In contrast, Bayesian models are natural candidates for online learning, as they are trained by updating the belief (model) upon new observations, i.e. posterior update.

Pre-training is an offline training followed by a main, task-specific training, hence the prefix "pre".

For example, a $512 \times 512 \times 3 \rightarrow 128$ dimensionality reduction model is pre-trained on a large dataset of RGB images (either supervised, or self-supervised). Then, the pre-trained model is used to reduce the dimension of our [new] images to $128$, which is then being fed to the main, task-specific model.

Note that the word "self-supervised" (supervised by input itself) is currently used for models that try to reconstruct, or predict the whole, or part of the input as close as possible; e.g., auto-encoders, or some language models such as Word2Vec.


How can I appropriately handle cleaning of gender data?

There are at least two general considerations to make:

Domain-related

If an attribute potentially has predictive power in your domain and more specifically for your task your models might benefit from a direct encoding. For example: if being trans is correlated with different psychological disorders then I'd include a direct feature for this. This way it is easier for your model to make a prediction since it does not need to combine two features in the first place (e.g. no need to combine "sex at birth" and "gender identification" to identify a transsexual person (which would not even accurate since "trans" is a much broader term than just sex at birth != gender identity)).

Moreover, I'd apply the same thinking to other feature engineering questions. Sex has predictive power for many tasks related to mental disorders, e.g. because mood disorders are more common among women and anti-social personality disorders are more common among men. However, whether these are rather related to the sex at birth or the gender a person identifies with is another question. So if your hypothesis is that in your task the gender a person identifies with is important then, again, it makes sense to include this in addition to the sex at birth.

Model-related

Different models are able to handle predictors differently. For example, tree-based models can more easily work with two separate attributes sex == female and trans == True to implicitly derive trans female == True. However, linear models like neural networks might benefit from having a combined binary feature female trans.


I'm one of the SWEs who worked on TensorFlow Quantum. I'll do my best to answer your three questions.

TensorFlow Quantum is a piece of software to incorporate the quantum computing primitives in Cirq into TensorFlow in a native and scalable fashion. It is primarily targeted at researchers.
Quantum computing is statistical in nature, TensorFlow is very useful for (among other things) statistical things like analyzing data and building models at very large scale. Having this ability to leverage the tools of TensorFlow alongside quantum computing workflows will hopefully lead to new research at scale that might not have otherwise been possible.
Hybrid quantum-classical machine learning is machine learning that involves both classical and quantum data. TensorFlow Quantum was designed with the goal of developing a better understanding of quantum data.

If my answers were too short for your liking, these sorts questions are also answered in greater detail over here and here.


Scikit Learn Missing Data - Categorical values

We would need more information on the prediction problem and the features to be able to give something more precise.

Anyhow, I am surprised no answer so far included all possible options since they aren't that many:

get rid of incomplete observations or features --- obviously, only viable if there are few incomplete cases since you lose too much information otherwise
replace NAs with some value like -1 --- this depends on the classifier you use; if your classifier supports categorical variables, you can create a new category for those NAs for example. In some continuous variables, sometimes there are some values that make sense (for instance, in text mining classification, if you have a title-length feature but you have no title, it might make sense to replace with title-length=0)
fill up the missing data

This last point encompasses too many things:

replace NAs with the median (this is the usual lazy approach; sklearn has a class for this)
if time series, replace with an average of the previous and following values -- in pandas, this can be done using DataFrame.resample().
use the $k$ closest neighbors. build a KNN model using the other variables and then do the average of those neighbors (if you use euclidean distance, you probably should normalize first). I never seen this done, but you probably could try predicting the missing NAs using another model as well.

But all this depends very much on what you are doing. For instance, if you have performed clustering analysis and you know your data is made up of clusters, you could use the median within each cluster.

Possibly other solutions could include things like multimodal or multiview models. These are recent techniques that can cope with missing modalities, and you can see a feature, or subset of features, as a modality. For instance, you could build a different classifier for various subsets of your features (using the complete cases in each of those subsets) and then build another classifier on top of that to merge those probabilities. I would only try these techniques if most of your data is missing. There are more advanced deep learning versions of this using autoencoders.


'tuple' object is not callable while reshaping training data using python

The problem is that train.shape is simply a tuple. So train.shape(arguments) doesn't make sense because train.shape is not callable (hence the error). Try to just replace that line by

data = train.reshape((train.shape[0], 3, train.shape[1]))


How to extract all information by using id

Define a new column, then use these ids to select the relevant rows and set that column to the appropriate id:

from pandas import DataFrame,Series
from numpy.random import rand
df = DataFrame(rand(10,10)).assign(cluster=0)
clusters = Series([[1,3,5],[0,2,4,6,7,8,9]])
for cluster,rows in clusters.iteritems():
    df.loc[rows]["cluster"] = cluster

Now you can do your groupbys on column "cluster".


How to transform entire pandas data frame in one hot representation?

You can use:: pandas.get_dummies

get_dummies will only convert string columns and will keep numerical columns as it is. You can first convert categorical columns into string type and then apply get_dummies.

concated_dataset['1stFlrSF'] = concated_dataset['1stFlrSF'].astype("string")
pd.get_dummies(cacated_dataset)


Cleaning NaNs with averages pre or post split?

Imputing all missing values before the split would mean data leakage - you would use test set information to influence the training set.
I had a similar problem recently, I ended up using complete cases of the source data for the test set and then imputing the training data using medians calculated by class.
Otherwise, you might split the data and then impute the test set using the training set average/median/mode.


What ML/DL approach better suits this problem?

Yes, you can use DL network to solve this problem. It is a easy multi-classes classification task. And just a full connected network will do this work.

To construct the network I recommend using Keras which is easy to use. And before training the network, it is better to preprocess the data(standardizing the numeric column, and embedding the string column).

EDIT:

For the data normalisation you can reference this blog: Neural Network Data Normalization and Encoding, that show the basic methods and codes.


Scikit Learn: KMeans Clustering 3D data over a time period (dimentionality reduction?)

I don't have enough reputation to ask questions in comment for clarification before answering it, so I'm going to do both here.
Here are the things that would help answering this question for now:
Can you post part of the process_set.hpc?
What's its format? Is it a numpy array? Is it a Pandas dataframe?
What's the value of cclasses?
And now the answer:
First of all, k-means algorithm is able to find clusters in any n-dimensional data. If n is too big, it is better to use PCA but for n=3 that wouldn't necessarily add any value.
The second thing that looks suspicious to me is that in the documentation for kmeans in scikit-learn, there is no compute_labels option, as seen here. However, that option exists for MiniBatchKMeans as seen here.
Also, if you make your data in the form of a pandas dataframe (if it is not already so), things would be much easier to track and you won't have to reattach the timing information to your data afterwards.
I may be able to give you a more thorough answer if I know a bit more about the format of the data.
Good luck!


I'm not sure it is a good problem setting for input to a feed-forward network (especially such a small network with only one activation over 6 neurons).

What is there even to predict here if you are just using the day of the week and the month in the year? You could simply take the average for each of the pairwise combinations and use that.

If you want to adjust for local trends over your dataset's timespan, then perhaps look int temporal decomposition. For example, you can use the seasonal_decompose method available in the statsmodels package for Python.

What this does is to find:

a global trend over the range of your data
parts of your data that vary on a repeating pattern (seasonally),
any remaining noise, or residuals not fitting parts 1. and 2.

It works by applying convolutions over the data to find a repeating frequency. You can think of it like finding correlations in time, compared to the ConvNets, which look for correlations over pixels in an image, for example.

Here is an example, using a dataset taken of total number of international airline passengers over time. I am using the statsmodels package to load this well-known R dataset as a Pandas DataFrame:

In [1]: import statsmodels.api as sm

In [2]: import matplotlib.pyplot as plt

In [3]: ap = sm.datasets.get_rdataset("AirPassengers")       # ap = AirPassengers

The actual dataframe is stored on the data attribute of ap and has two columns:

In [4]: ap.data.columns
Out[4]: Index(['time', 'value'], dtype='object')

We can see how the data looks, where there is clearly an upward trend as well as a repeating seasonality (more people fly over the summer time):

In [5]: ap.data.plot("time", "value");

In [6]: plt.show()

Now we can perform the seasonal decomposition of the data (freq=12, because the data is monthly):

In [7]: decomp = sm.tsa.seasonal_decompose(ap.data.value.values, freq=12)

This decomp result has a few attributes, but most helpful is the plot... We can see the nicely separated seasonlity and the trend, along with some noise

In [8]: decomp.plot();

In [9]: plt.show()

We see it isn't perfect, as the seasonality is best fitted to the middle section, not the start or end phases - the residuals are largest at the beginning and end.

You can access each of the values: observed, trend, seasonal, resid and nobs on the decmp object.

Now if this analysis helps you discover seasonality, you might want to look into Seasnal ARIMA models (also available in the statsmodels package).

Here is a blog post I just found that looks pretty detailed and would help you on this path.

If you are doing this excercise to learn about Keras and deep learning, it might be worth looking into LSTM models that are better for time-series analysis. You will also need to re-think how you feed data into the model for training, not to mix up the temporal information or create biases (letting the model cheat).


Balanced Linear SVM wins every class except One vs All

I think the following synthetic example explains what is going on:

from sklearn.base import clone
from sklearn.svm import SVC
import numpy as np
from sklearn.datasets import make_blobs

X, y = make_blobs(1000, centers=[[1, 1], [-1, -1], [1, -1]], cluster_std=0.8)

models = [
    ('normal', SVC()),
    ('balanced', SVC(class_weight='balanced')),
]

import matplotlib.pyplot as plot
plot.ioff()

for i, (name, m) in enumerate(models):
    for k in np.unique(y):
        m = clone(m).fit(X, (y == k).astype(int))

        xx, yy = np.meshgrid(np.arange(-3, 3, 0.1), np.arange(-3, 3, 0.1))
        z = m.predict(np.c_[xx.ravel(), yy.ravel()])
        z = z.reshape(xx.shape)
        colors = ['blue', 'green', 'red']
        plot.contour(xx, yy, z, colors=colors[k])
    plot.scatter(X[:, 0], X[:, 1], c=y)
    plot.title(name)
    plot.show()

The balanced SVM is clearly superior when considering each model individually, at least as measured by the F1 score, which punishes false positives and false negatives equally.

But One-vs-Rest uses maximum score, which in this case is given by the distance to the decision hyperplane. It is not yet clear why this is worse in some cases, but clearly it is possible for one model to beat another for individual classes, but to lose for One-vs-Rest: how well the relative distances are modeled is what counts in One-vs-Rest.


Can Machine Learning be applied in software developement

Definately - Yes.
Good question.
Was thinking about it myself.

(1) Collect the data.
The first problem you have: gather enough data.
All the attributes you mentioned (date, name, check-in title/comment, N of deffects etc) are potentially useful - gather as much as possible.
As soon as you have a big project, a number of developers, many branches, frequent commits and you have started collecting all the data, you a ready to go further.

(2) Ask good questions.
The next question you should ask yourself: what effect are you going to measure, estimate and maybe predict.
Frequency of possible bugs? Tracking inaccurate "committers"? Risky branches? Want to see some groups of users/bugs/commits according to some metrics?

(3) Select the model.
As soon as you have the questions formulated, you should follow the general approach in data science - extract needed features in your data, select appropriate model, train you model and test it, apply it. This is too broad process to discuss it this thread, so please use this site to get right answers.


Reward negative derivative on linear regression

It's a classic outlier. You could, for example, remove him or replace him with a new value(by interpolation).
You have many ways to work around this problem.

Links for you:

https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba

With some code:
https://towardsdatascience.com/5-ways-to-detect-outliers-that-every-data-scientist-should-know-python-code-70a54335a623


When is Gradient Descent invoked on the objective function while running XGboost?

I thought of throwing some historical perspective into this...

The initial algorithm is called AdaBoost. Unlike most of machine learning, this algorithm has its genesis in hard theory. The authors were trying to answer the following theoretical question:

  Is it possible somehow to combine weak models and create a very accurate predictor?

Weak models are models that are hardly better than chance. Think about this question. It is quite a philosophical question... And, as it happens, it can be answered by hard mathematics.

They actually demonstrated that yes. Their paper is actually worth reading. They have since published a lot of interesting papers from different perspectives, for instance using game theory.

The AdaBoost algorithm is also very simple and worth checking because it is the basis of xgboost.

AdaBoost works with any model. The only restriction is that the model supports giving weights to each observation. This is because it trains models sequentially by increasing (or decreasing) the weight associated to each observation to make that observation more important (or less important).

The cool thing is that the algorithm is very unlikely to overfit. But notice that this is an empirical claim, not a theoretical one. The algorithm can overfit, but it has been found (empirically) to be very resilient.

This algorithm has been expanded into Gradient Boosting, which has more flexible loss functions. This algorithm is an AdaBoost adaption by the same guy that invented random forests. They are not concerned about theoretical questions like the authors of AdaBoost, they just want to make the algorithm more flexible and efficient. For instance, they also update the weights using all previous weak models, not just the last trained model.

xgboost is just an intelligent implementation of Gradient Boosting. You can see the basic algorithm in wikipedia. Each decision tree is trained sequentially and weights are computed based on the errors from the current ensemble.

I can hear you say:

  What? xgboost trains trees in sequence? How is xgboost so fast then?

This website explains it very well. While trees are trained sequentially, each individual decision tree is trained in parallel by using highly creative techniques. Most importantly, nodes in the same depth do not depend in each other, so you can paralelize them.

Long story short: the loss function is applied between training models.

When it comes to making predictions, then the entire ensemble can be used in parallel.


Try to raise the nlp.max_length parameter (as your error message suggests):
nlp = spacy.load('en_core_web_sm')
nlp.max_length = 1500000 #or any large value, as long as you don't run out of RAM

also, when calling your spaCy pipeline, you can disable RAM-hungry intensive parts of the pipeline that are not needed for lemmatization:
doc = nlp("The sentences we'd like to do lemmatization on", disable = ['ner', 'parser'])

Finally you should get the results you expect with:
print([x.lemma_ for x in doc])


Calculate whether datapoints are part of a larger distribution

One way addressing your question is to ask if your two datasets(regular points and irregular points) come from the same distribution?
The kolmogorov-smirnof test is a common tool for putting a number on the 'likelihood' of two datasets being from the same distribution.
python scipy has an implementation of this test.

Be aware that this test is more sensitive to differences around the means of each distribution, than to differences around the edges. From the looks of the graph in the question, its not an issue in your case, so the test should work.

This is a very concise explanation of the test


How to train a spacy model for text classification?

You have several good tutorials on the web :

https://www.kaggle.com/poonaml/text-classification-using-spacy

Basically, you have to :

Import the data in python, here POSITIVE is the variable to predict and 0 and 1 are the 2 encoded classes.

TRAIN_DATA = [(Text1, {'cats': {'POSITIVE': 1}}),
(Text2, {'cats': {'POSITIVE': 0}})]

Initialize a textcat pipe in a spacy pipeline object (nlp), and add the label variable in it.

nlp = spacy.load('en_core_web_sm')
if 'textcat' not in nlp.pipe_names:
  textcat = nlp.create_pipe("textcat")
  nlp.add_pipe(textcat, last=True)
else:
  textcat = nlp.get_pipe("textcat")

textcat.add_label('POSITIVE')

Iterate the training examples to optimize the model

other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']

n_iter = 1

# Only train the textcat pipe
with nlp.disable_pipes(*other_pipes):
    optimizer = nlp.begin_training()
    print("Training model...")
    for i in range(n_iter):
        losses = {}
        batches = minibatch(train_data, size=compounding(4,32,1.001))
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(texts, annotations, sgd=optimizer,
                      drop=0.2, losses=losses)


Feature importance parameter in machine learning models like Naive Bayes

Since Naive Bayes assumes independence and outputs class probabilities most feature importance criteria are not a direct fit. The feature importance should be no different from the skewness of the feature distribution in the set: You could try to directly compare the probability of the features given the classes (implemented in sklearn for instance), the variability of those probabilities with respect to the classes should express the importance of those features.


Feature engineering for categorical variables

Before I begin let me note that there isn't a right and a wrong way to do feature engineering. If you want to be thorough, you need to try things and see if they work or not.

Now let me give you my opinions (i.e. how I might approach things), but they aren't guaranteed to work:

  1) One of the variable can take 3 values (Girls, Boys, Girls&Boys). Converting it into one-hot encoding or binary encoding will treat all three values as a different class. How can I use it efficiently retaining the information that 'Girls&Boys' include both? Is breaking into two separate columns for girls and boys is the only approach in such cases?

I think one-hot encoding works best here. Depending on what information does Girls&Boys store you can select how to encode it. If Girls&Boys == 1 if both Girls and Boys are 1, then I think you can safely omit it.

  2) Age range: (18-35, 35-50 etc.) I have broken it down to 2 columns of age_min and age_max. Is there any better way to use such kind of features with values in range?

What you've done isn't bad. Another option would be to treat this like an ordered variable and simply map each range to a number (while preserving the order). E.g. 18-35 --> 1, 35-50 --> 2, etc.

  3) Range of percentage: It can take only 5 values (0, 1, 1-5, 5-10, 10). How should I use this variable for training my model? Here I cannot break it down into 2 columns (like with age) because of "1" and "10" fixed values. How to treat variables with fixed as well as range values?

Same thing as before. I'd treat this like an ordered variable. You could even leave 0 and 1 as they are. The rest would go like this: 1-5 --> 2, 5-10 --> 3, 10 --> 4. If some of these 5 categories has very few values you could merge it with a neighboring one!

  4) Similarity: It can also take 5 values (0, 1, 1-5, 5-10, 10) but it also has an "auto" option. That means if it is "auto" it can take any random value which we do not know about. How should I incorporate that? Should I create a separate column for "whether it is auto or not"? Then, in the original similarity column, what value should I put for observations with "auto". I cannot put "0" as it is already a value and I am sure that "auto" won't be 0. Does putting it "None" will create a difference? How to treat unknown values?

This is trickier here. One thing I'd do is what you mention: to have an auto column which can be either 1 or 0 if the user chose "auto" or not. However, it's tricky to encode the range now, because how would you encode those that selected "auto"? One way is to make a separate category (e.g. auto --> 5), but this wouldn't correct for an ordinal variable. Another option would be to one-hot encode this. Your choice I think depends on two reasons: a) Given the "auto" option how important is it preserving the range information as is? Could you group categories together? could you disregard their order? b) How many of the samples are "auto"? If they are very few it might not even make a difference. If they are too many then encoding the range might have no meaning.


how to match a sentence to a cluster of keywords?

I am kind of confused by the question. But I am guessing you are asking what model to use and how to represent words.

This seems like a pretty introductory example so you may want to try simple Bag-of-words representation. Try pushing that into a Naive Bayes classifier as a first pass. If you are unhappy with the results, you can implement word2vec and more complex methods.


Format for X_train in keras using theano

I solved this (still banging my head against the wall):

estimator.fit(numpy.array(X_train),numpy.array(y_train))

this works. I am not sure why. The error message is very misleading IMHO.


It seems that you have an output size of 2 in your final layer, while you should rather have size 1 (because of your sigmoid output and binary cross entropy loss).

Also, don’t use the to_categorical transformation as you only have two classes so no need to one-hot encode.

Try to change this and see if training improves.


You can still perform regression even if your input (or part of it) is discrete.
If you think of it, even your "continues" values are actually discrete (starting from their initial measurement accuracy/resolution).

You need to decide if you want to train a model to perform estimation, or just cast your input features to a vector space and use them with some euclidean distance (to make sure some features match exactly, just add some "if" check so you can calculate distances only to relevant samples).

To train a model, it should except as input all the features (preferably normalized), and output a single number which you can later round to the closest discrete value (if you like).


Tips & Tricks on training DCGAN on small dataset

Extending a small dataset comprised of images

A deep learning algorithm will learn a mapping function from your input space to your outputs. The variations in your input images will be learned within this function. Thus, you will want to consider this fact when you augment your dataset. The distribution of your input features should be concise as to what you plan to model.

You are thus left balancing between adding data and adding variability to your input space. For example, it might not be worth it to rotate by 180 degrees when trying to generate Dragon Ball characters. Their heads should not be where their feet are. You would want your network to understand that the bottom of the image should contain some strange space boots, and the top exotic haircuts or bald heads.

Augmenting images

Here are some useful transformations that you can use to get more data

Apply transformations (rotations, translations)
Mirror the image
Add distortions
Change zoom factor
Add blurring thus to better generalize the input data
Invert colors (skew, add brightness, etc.)

In Keras

In Keras you can use the ImageDataGenerator functions.

import numpy as np
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt
%matplotlib inline

# load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.astype('float32')

# set up your data generator
datagen = ImageDataGenerator(
   featurewise_center=True,
   featurewise_std_normalization=True,
   rotation_range=60,
   width_shift_range=0.2,
   height_shift_range=0.2,
   shear_range=0.2,
   zoom_range=0.2,
   horizontal_flip=True,
   vertical_flip = True)

# Fit the generator using your data
datagen.fit(X_train.reshape((len(X_train), 28, 28, 1)))

# let's look at some generated images
image = X_train[5]

plt.figure(figsize=(12,12))
plt.subplot(4, 4, 1)
plt.imshow(image.reshape((28,28)),  cmap='gray')
for j in range(15):
   augmented = datagen.random_transform(image.reshape((28,28,1)))
    plt.subplot(4, 4, j+2)
    plt.imshow(augmented.reshape((28,28)),  cmap='gray')
plt.tight_layout()
plt.show()


my intial dataframe :

     id  p1_201501  p1_201502  p1_201803  p2_201801  p2_201812
0   x12        125         12          6        500         14
1  nz15        250         16        600        423         32
2  qz15        350          4         20        223         32

my program:

import pandas as pd

df = pd.melt(df, id_vars=["id"], var_name="product", value_name="values").reset_index(drop=True)
df['date'] = df['product'].str.split('_',n = 1, expand=True)[1]
df['year'] = df['date'].astype(str).str[:-2]
df['month'] = df['date'].astype(str).str[4:]

#drop unused columns
df.drop(['product','date'],axis=1,inplace=True)

df = df[['id', 'year','month','values']]
df = df.sort_values(['id', 'year','month'], ascending=[True,True,True]).reset_index(drop=True)
df.set_index(['id','year','month'])

#just sum each annual total sales for each id
df = df.groupby(['id','year'])['values'].sum()

print(df)

final result:

id    year
nz15  2015     266
      2018    1055
qz15  2015     354
      2018     275
x12   2015     137
      2018     520


rows to columns in data.table R (or Python)

Just two lines in plain R

X <- read.table(header = TRUE, text = "
zone       code        literal
 A         14           bicl
 B         14           bicl
 B         24          calso
 A         51           mara
 B         51           mara
 A         125           gan
 A         143          carc
 B         143          carc")

X$time <- ave(X$code, X$zone, FUN = seq_along)
reshape(X, direction = "wide", timevar = "time", idvar = "zone", sep = "")

# output
  zone code1 literal1 code2 literal2 code3 literal3 code4 literal4
1    A    14     bicl    51     mara   125      gan   143     carc
2    B    14     bicl    24    calso    51     mara   143     carc


PCA-like analysis for dataset that has both categorical and continuous variables

I'm not aware of any the dimensionality reduction algorithms (like PCA) that can work with categorical values.

However, an approach that could help you with that is to make a one-hot encoding of your categorical variables (if the number of possible values is manageable. Otherwise, try to pick only the most frequent values and assign the rest to a single variable).

If you are using Pandas DataFrames, get_dummies can be helpful.


human level performance on ImageNet, top-1 or top-5?

It comes from this paper: https://arxiv.org/abs/1409.0575

O. Russakovsky "ImageNet Large Scale Visual Recognition Challenge" 2014


Tensorflow regression model giving same prediction every time

Your network design/logic is basically correct, but you are seeing some very common problems with neural network numerical stability. This results in your weights diverging and not training accurately.

Here are the fixes, any one of them might help a little, but the first two should be used for nearly all neural network projects.

1. Inputs need to be scaled to work with neural networks.

This is called input normalisation. Usually you would do this in data preprocessing, but for your simple network we can include the scaling at the input:

x_normalised = x * 0.2 - 0.5             # Arbitrary scaling I just made up
Out1 = feedForward(x_normalised,w1,b1)   # output of first layer

The most common scaling operation is to take all the training data and convert it so that it has mean $0$ and standard deviation $1$ (typically per feature, as opposed to global scaling for all data as I have done here) - store the values used to achieve that and apply them to all following data for training, tests etc.

2. Adjust learning rate until you get a working value.

train_step = tf.train.GradientDescentOptimizer(0.001).minimize(J)

A value that is too high will cause your training to fail. A value that is to low will take ages to learn anything.

3. For small training sets, use more iterations than you might think

This is not a general thing, but specific to demos with tiny amounts of training data like your example, or the commonly-used "learning XOR function".

for _ in range(10000):                      # performing learning process

   sess.run(train_step, feed_dict = {x:xs, M:Ms})

With your very simple network actually this may cause over-fitting to the training data, so you will have to play with a value that gives you "sensible" results. However, in general how to spot and minimise over-fitting is a whole broad subject in itself, based on how you test and measure generalisation. This will need other questions if you are not sure when you learn it. It should be high on your list of things to learn though . . . it is a critical skill in producing useful neural networks that solve real problems.


what does the standard deviation plot around my learning curve indicate?

I only have a guess, but I suspect it could be simply due to random initialisation. This would mean that after few training samples the models would still be very different. After 400k training samples the models all converge to the same learning path. I could be wrong of course!


How to generate training instances from multivariate normal distribution?

This can be done by using inverse transformation theorem
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.preprocessing import QuantileTransformer

def make_gausssians_binary_classification(n_samples= 1000,
                                          n_features= 5,
                                          n_informative= 5,
                                          n_redundant= 0,
                                          random_state= 42
                                          ):

    X, y = make_classification(n_samples= n_samples,
                           n_features= n_features,
                           n_informative= n_informative,
                           n_redundant= n_redundant,
                           random_state= random_state)

    transformation = QuantileTransformer(output_distribution= "normal").fit(X)

    X_normal = transformation.transform(X)

    return X_normal, y

X, y = make_gausssians_binary_classification(n_samples=10000)


Effect of NOT changing filter weights of CNN during backprop

By not changing the weights of the convolutional layers of a CNN, you are essentially feeding your classifier (the fully connected layer) random features (i.e. not the optimal features for the classification task at hand).

MNIST is an easy enough image classification task that you can pretty much feed the input pixels to a classifier without any feature extraction and it will still score in the high 90s. Besides that, perhaps the pooling layers help a bit...

Try training an MLP (without the conv/pool layers) on the input image and see how it ranks. Here is an example where an MLP (1 hidden & 1 output layer) reached 98+% without any preprocessing/feature extraction.

Edit:

I'd also like to point out to another answer I wrote, which goes into more detail on why MNIST is so easy as an image classification task.


When to use a Sigmoid function and when to use ReLu in a Convolutional Neural Network

Relu solves the gradient vanishing problem and stops the inactive neurons. Using Sigmoid in output layer because its range is (0, 1) and it can represent the probability of binary class. When you are doing multi-classification, it is more appropriate to use Softmax function.

Maybe you need to read more materials, you can refer to CS231n. Thanks.


Although I haven't verified it, a first glance at the features and training set you used shows an obvious problem. You have just two data samples with the exact same features while you give it different labels.

Although other types of models might not break and just give both samples an equal probability of being 0 or 1, something in the Naive Bayes classifier internal calculations probably requires there to be at least some difference in differently labeled samples (not an unreasonable assumption).

I can't confirm this for myself as I'd have to go deeper into the source code to actually check the scikit implementation so perhaps someone else more familiar with it can check.

EDIT: I just tested this out by changing one line in your code to

features = np.array([[-1, 1],[-1, 0]])

It worked.


Can we use decreasing step size to replace mini-batch in SGD?

Generally answer is "it's not known". Similarity of effects of increasing minibatches size and decreasing learning rate is mostly empirical, there is no known asymptotic formula for it. Also effect of small LR and big minibatch is not the same. For example batch normalization layer would act completely different on those two approaches. Probabilistic distribution of gradients produced by minibatches and single sample (or mb of significantly different size) would be also quite different


Why do we divide the regularization term by the number of examples in regularized logistic regression?

There are several possible intuitions behind that as explained here, but the strongest argument for me is the following one.

The „best“ form of the loss function is the one that requires the least amount of tweaking, e.g. for different dataset sizes. Dividing the regularization term by the number of samples reduces its significance for larger datasets. And, indeed, since regularization is needed to prevent overfitting, its impact should be reduced (in favor of the impact of the data itself) if a larger amount of data is available.

This way the same $\lambda$ value has better chances for working with the whole dataset as well as with a small part of it without adjusting.


What is the preferred approach for this problem?

Convert your training data to the following format -

a = login_time
b = logout_time
c = person_activity
train_data_X = [
               [[a, b, c], [a, b, c], ......, [a, b, c]]
               [[a, b, c], [a, b, c], ......, [a, b, c]]
               .
               .
               .
               [[a, b, c], [a, b, c], ......, [a, b, c]]
             ]

train_data_Y = [
                 [[time_day_1], [time_day_2], ...... [time_day_60]]
                  .
                  .
                  .
                 [[time_day_1], [time_day_2], ...... [time_day_60]]
                ]

Explaination -

You Input(training_data_X) data has shape 10000 * 60 * 3 because you have 10K users, data for 60 days and each data point depends on 3 variables namely a, b and c.
Your Target(training_data_Y) data has shape 10000 * 60 * 1 because you have just one number to output i.e active time.

Now, this is a Regression Problem because you are trying to predict a continuous value i.e active time.

You can model this type of problem with a number of time-series models. But since you got a multivariate input, i.e [a, b, c](refer training_data_X). You would be benefitted a lot using a Seq2Seq model to generate your time-series. If just want to generate 61st time-step you could use it just for that as well.

Specific details and implementation


The politicians and Political parties follow a set of principles and characteristics. They always follow these things during voting on legislations and bills. you can use T-distributed Stochastic Neighbor Embedding (t-SNE), machine learning algorithm for visualization developed by Laurens van der Maaten and Geoffrey Hinton. This method helps us to visualize the clusters and similarities of the observations.

Like-minded MP's and political parties with similar principles cluster together(neighbors).

The t-SNE 1algorithm comprises two main stages. First, t-SNE constructs a probability distribution over pairs of high-dimensional objects in such a way that similar objects have a high probability of being picked, whilst dissimilar points have an extremely small probability of being picked. Second, t-SNE defines a similar probability distribution over the points in the low-dimensional map, and it minimizes the Kullback–Leibler divergence between the two distributions with respect to the locations of the points in the map. Note that whilst the original algorithm uses the Euclidean distance between objects as the base of its similarity metric, this should be changed as appropriate.


It is a bit hard to solve your problem without data but I tried to give it a go
with a how I think the data would be encoded. I used R with data.tables. You can read data.tables with fread().

Step 1

 require(data.table)

 # Assume sample_data has the following format:
 #   class: the class
 #   division: the division
 #   attendance: the attandance for a match
 #
 # I assume the table is in long format e.g. multiple rows exist per class with
 # per class one or different divisions.

 # Make the list of classes with at least 5 divisions.
 classes_of_interest <-
   sample_data[,
               .(num_divisions = length(unique(divisions))),
               by = class][num_divisions > 4, class]

Step 2

 # Only consider the classes that were in at least 5 divisions.
 attandance_by_division <-
   sample_data[class %in% classes_of_interest,
               .(attendance = sum(num_people)),
               by = list(division, class)]
 setkey(attandance_by_division, "class")

Step 3

 # Merge the data set with a datas set that contains the required number
 # of attendants per class.
 # The  format is as follows:
 #   class: the class
 #   mininum_attendance: the minimum attendance
 attendance_data <-
   merge(attendance_requirements,
         attandance_by_division, by = "class")

 # Here I exploit the fact that the true/false condition will be converted
 # to a 1 and 0. So I can sum and divide by the length of index subset created
 # by aggregating on 'class'.
 pct_of_division <-
   sample_data[,
               .(pct_with_min_attendance = (sum(attendance > minimum_attendance)
                                            / length(.I))),
                 by = class]

Hope this helps


ML algorithm with fixed number of inputs and variable number of outputs

RNN (and LSTM / GRU)can generate variable length output. For example, generation of text.

In such problems, RNN is designed to generate next character (It keeps track of what characters have generated in past).  RNN should generate "End Of Output" character to indicate end of sequence.

For example : https://chunml.github.io/ChunML.github.io/project/Creating-Text-Generator-Using-Recurrent-Neural-Network/

Following method will have to be changed :

def generate_text(model, length):
    ix = [np.random.randint(VOCAB_SIZE)]
    y_char = [ix_to_char[ix[-1]]]
    X = np.zeros((1, length, VOCAB_SIZE))
    for i in range(length):
        X[0, i, :][ix[-1]] = 1
        print(ix_to_char[ix[-1]], end="")
        ix = np.argmax(model.predict(X[:, :i+1, :])[0], 1)
        y_char.append(ix_to_char[ix[-1]])
    return ('').join(y_char)

to :

     stop_characters = set(['.','?'])
..
..
     ix = np.argmax(model.predict(X[:, :i+1, :])[0], 1)
     predicted_char=ix_to_char[ix[-1]]
     if(predicted_char in stop_characters ):
         break
     y_char.append(predicted_char)


Advantages of pandas dataframe to regular relational database

I think the premise of your question has a problem. Pandas is not a "datastore" in the way an RDBMS is. Pandas is a Python library for manipulating data that will fit in memory. Disadvantages:

Pandas does not persist data. It even has a (slow) function called TO_SQL that will persist your pandas data frame to an RDBMS table.
Pandas will only handle results that fit in memory, which is easy to fill. You can either use dask to work around that, or you can work on the data in the RDBMS (which uses all sorts of tricks like temp space) to operate on data that exceeds RAM.


Pre-processing is 50% of the entire pipeline. With better data you build better machine learning/deep learning models. But unfortunately, cleaning data is something that needs time and experience. You need to visualise it and make your hands dirty. The aim is to remove noise, garbage and outliers, in short.

There are some really good data visualisation libraries in Python such as :

MatplotLib
Seaborn
GGPlot
Bokeh
PyGal
Plotly
and more.

Since, you are comfortable with Scikit learn, as you mentioned in the question, I would suggest you to look up the preprocessing modules in scikit learn, it contains several APIs such as feature extraction, normalization, feature scaling, mean removal, variance scaling, standardization, etc.

  But, it is always better to understand the data, visualise it like a
  story and clean it manually,
  slowly, instead of passing it through predefined frameworks or
  pipelines.

These videos :

Why You Need Data-Preprocessing
Data-preprocessing tasks

can be helpful. There are many more available easily. Good luck !

Also, do upvote my answer, if it has helped you. It encourages the community to help each other.


How does SQL Server Analysis Services compare to R?

In my opinion, it seems that SSAS makes more sense for someone who:

has significantly invested in Microsoft's technology stack and platform;
prefer point-and-click interface (GUI) to command line;
focus on data warehousing (OLAP cubes, etc.);
has limited needs in terms of statistical methods and algorithms variety;
has limited needs in cross-language integration;
doesn't care much about openness, cross-platform integration and vendor lock-in.

You can find useful this blog post by Sami Badawi. However, note that the post is not recent, so some information might be outdated. Plus, the post contains an initial review, which might be not very accurate or comprehensive. If you're thinking about data science, while considering staying within Microsoft ecosystem, I suggest you to take a look at Microsoft's own machine learning platform Azure ML. This blog post presents a brief comparison of (early) Azure ML and SSAS.


How much data is needed for a GBM to be more reliable than logistic regression for binary classification?

In my experience GBM does at least as well as LR on small datasets too. The main advantages of GBM over LR occur when you have

correlated variables
nonlinearity
interactions

If you want to simplify your GBM you can use feature importances to drop unimportant features. You might also want to try PCA to see if you can reduce the number of features. After this you could try a LR model on the resulting features to see the model performance.

If you're worried that you've overfit your model to the test set, you'll need to find a new test set!


Interpretation of the loss function for word2vec

Not sure what the video said, but $T$ should not be the vocabulary size, but the training corpus size (number of all words).

For example, if your training corpus is

deep learning is popular . i love deep learning . i want to learn more about it.

Then when you sume up over $T$, you will sum up all the word pairs in the corpus including duplicates. The word pair (deep learning) is indeed calculated twice.

For details please refer to the original skip-gram paper, and notice the definitoin (1) on page 2.


Learning AI, Machine Learning, Deep Learning

Bascially this is true. Deep Learning is kind of like stacking your machine learning algorithms on top of each other. Have a look here and the associated picture.


Cost tends to infinity when relu activation is used

If a3 is 0 or 1, np.log(a3) or np.log(1-a3) is going to give an error as $\log(0)$ is undefined, and $\lim_{x -> 0} \log(x) = -\infty$


Logistic regression GD implementation in Python

Actually, i have finally made it work. I dont know, what exactly was the crucial change, but here's the extract of my changes:

replaced all np.multiply with *
Decreased learning rate and regulizers
Applied np.nan_to_num to exponents

So here is the final code:

def lossf(w, X, y, l1, l2):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    lossf1 = np.sum(ss.log1p(1 + np.nan_to_num(ss.expm1(-y * np.dot(X, w)))))
    lossf2 = l2 * (np.dot(np.transpose(w), w))
    lossf3 = l1 * sum(abs(w))
    lossf = np.float(lossf1 + lossf2 + lossf3)
    return lossf

def gradf(w, X, y, l1, l2):
    w.resize((w.shape[0],1))
    y.resize((y.shape[0],1))

    gradw1 = l2 * 2 * w
    gradw2 = l1 * np.sign(w)
    gradw3 = -y * (1 + np.nan_to_num(ss.expm1(-y * np.dot(X, w))))
    gradw3 = gradw3 / (2 + np.nan_to_num(ss.expm1(-y * np.dot(X, w))))
    gradw3 = np.sum(gradw3 * X, axis=0)
    gradw3.resize(gradw3.shape[0],1)
    gradw = gradw1 + gradw2 + gradw3
    gradw.resize(gradw.shape[0],)
    return np.transpose(gradw)
class LR(ClassifierMixin, BaseEstimator):
    def __init__(self, lr=0.000001, l1=0.1, l2=0.1, num_iter=100, verbose=0):
        self.l1 = l1
        self.l2 = l2
        self.w = None
        self.lr = lr
        self.verbose = verbose
        self.num_iter = num_iter

    def fit(self, X, y):
        n, d = X.shape
        self.w = np.zeros(shape=(d,))
        for i in range(self.num_iter):
            print "\n", "Iteration ", i
            g = gradf(self.w, X, y, self.l1, self.l2)
            g.resize((g.shape[0],1))
            self.w = self.w - g
            print "Loss: ", lossf(self.w, X, y, self.l1, self.l2)
        return self

    def predict_proba(self, X):
        probs = 1/(2 + ss.expm1(np.dot(-X, self.w)))
        return probs

    def predict(self, X):
        probs = self.predict_proba(X)
        probs = np.sign(2 * probs - 1)
        probs.resize((probs.shape[0],))
        return probs


For the formal proof I think it's best to post on maths stackexchange instead.

Here are one of the best videos on backpropagation:

(link)

(link)

Your equation shows the chain rule.

hadamard-multiplying any matrix (like $w^T$) by a vector-with-n-entries (in your case by $dz^{[2]}$) is basically saying "I scale each stripe of my matrix by the corresponding entry in vector $dz$"

regarding the "stripes":

each row of $W^{[2]}$ represents a weights from one of our neurons to all neurons in preceding layer
each column of $W^{[2]}$ represents weights from all our neurons towards one particular neuron in preceding layer.
your activations ($g$) each depends on the weights that flow into it (the weights are the parameters of these activations).
your activations also depend on the values of neurons that flow into them. Notice, although neurons are the same entities used by the activations, they are treated as separate branches on the tree diagram (link)

If you watch the "tree diagram link", here is one more hint: in the video, the girl was summing up the terms from different branches. This is exactly what the dot product is doing during backprop.


What's the difference between multi label classification and fuzzy classification?

Multi-label classification (Wiki):

Given $K$ classes, find a map $f:X \rightarrow \{0, 1\}^K$.

Fuzzy classification (a good citation is needed!):

Given $K$ classes, find a map $p: X \rightarrow [0, 1]^K$ where $\sum_{k=1}^{K} p(k)=1$.

In multi-label classification, as defined, there is no "resource limit" on classes compared to fuzzy classification.

For example, a neural network with a softmax layer does fuzzy classification (soft classification). If we only select a class with the highest score, then it will become a single-label classification (hard classification), and if we select top $k$ classes, it will be a multi-label classification (again hard classification).

Fuzzy classification:        [0.5, 0.2, 0.3, 0, 0]
Single-label classification: [1,   0,   0,   0, 0]
Multi-label classification:  [1,   0,   1,   0, 0]

As another example for multi-label classification, we could have $K$ neural networks for $K$ classes with sigmoid outputs, and assign a point to class $k$ if output of network $k$ is higher than 0.5.

Outputs:                     [0.6, 0.1, 0.6, 0.9, 0.2]
Multi-label classification:  [1,   0,   1,   1,     0]

Practical considerations

As demonstrated in the examples, the key difference is the "resource limit" that exists in fuzzy classification but not in multi-label classification. Including the limit (in the first example), or ignoring it (in the second example) depends on the task. For example, in a classification task that has  mutually exclusive labels, we want to include the "resource limit" to impose the "mutually exclusive" assumption on the model.

Note that the $\sum_{k=1}^{K} p(k)=1$ restriction in fuzzy classification is merely a "definition", there is no point in arguing about a definition. We can either propose another classification, or  argue when to use - and when not to use - such classification.


Is it possible to call from Keras unsupported backend function directly from tensorflow?

There is no possible way to directly call tensorflow functions from keras. Keras is a stand-alone project. Although, tensorflow >= 1.12 has keras built in, so you could play with the source of tensorflow and figure it out.


I get this error Expected Ptr for argument '%s' while resizing image in open cv2

I changed the data type from float16 to float32, solving the problem:

images.astype(np.float32)


How to check the randomness or the distribution of an attribute in a dataset?

One option is to check and compare feature statistics across the datasets: mean, variance (or standard deviation), kurtosis, and skewness.

If you prefer a more visual approach (I do!), you can plot density or histograms of each feature, creating overlays for each dataset.

pandas can be useful in calculating the statistics; for visualisation I like using seaborn.


Permutation feature importance vs. RandomForest feature importance

Forest:

Forest paper
"We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories."

This is saying that if a feature varies on its ability to detect based on class it will be fault. This seems to imply the methodology is sensitive to data outliers like maybe a feature if 90% responsible for classification of 5% of the variables it may be the most important feature even though that might not really be accurate.

Permutation:

 Scikit-learn
"Permutation feature importance is a model inspection technique that can be used for any fitted estimator when the data is rectangular.
This is especially useful for non-linear or opaque estimators. The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled 1. This procedure breaks the relationship between the feature and the target, thus the drop in the model score is indicative of how much the model depends on the feature. This technique benefits from being model agnostic and can be calculated many times with different permutations of the feature."

The data being rectangular means it is a multivariate feature array table. The model working in non linear scenarios means it is able to be beneficial in prediction even when outputs follow non linear functions like XOR.


You might want to take into consideration that Pylearn2 has no more developer, and now points to other Theano-based libraries:

  There are other machine learning frameworks built on top of Theano that could interest you, such as: Blocks, Keras and Lasagne.

As Dawny33 says, TensorFlow is just getting started, but it is interesting to note that the number of questions on TensorFlow (244) on Stack Overflow already surpasses Torch (166) and will probably catch up with Theano (672) in  2016.


How to apply AdaBoost to more "complex" (non-binary) classifications/data fitting?

AdaBoost is a supervised learning method; it starts with a table of 'correct' answers and generates a predictive model for a target, which is known. It is then possible to inspect this model to figure out how it works, what it judged was more important. With that in mind, here is my guess for what he did:

First, he created a training dataset. The dataset was created, according to the article, like this:

  Now he’d do the same for love. First he’d need data. While his
  dissertation work continued to run on the side, he set up 12 fake
  OkCupid accounts and wrote a Python script to manage them. The script
  would search his target demographic (heterosexual and bisexual women
  between the ages of 25 and 45), visit their pages, and scrape their
  profiles for every scrap of available information: ethnicity, height,
  smoker or nonsmoker, astrological sign—“all that crap,” he says.

  To find the survey answers, he had to do a bit of extra sleuthing.
  OkCupid lets users see the responses of others, but only to questions
  they’ve answered themselves. McKinlay set up his bots to simply answer
  each question randomly—he wasn’t using the dummy profiles to attract
  any of the women, so the answers didn’t matter—then scooped the
  women’s answers into a database. "

So he used python scripts to collect lots of information! At the end of this, he had a table of data, where each row had three pieces of information:
- A bot's answer to all the questions, and their weights
- A woman's answer to all the questions
- Their match percentage.

On this, he could use AdaBoost to create a predictive model, predicting the match percentage from the available information. The weak learners were probably decision stumps, greedily choosing one variable at a time to split on, that's the standard that people refer to when talking about AdaBoost.

Once the predictive model, was in place, it could be used for optimization - determining which weights to put on the questions when holding all other variables constant, maximizing the average match percentage to women in his target audience.

Of course, this is just a guess. The article doesn't have much detail. But it's a potential way to use AdaBoost for that purpose.


the library 'transformers' works also with older version of Tensorflow?

It does work but I recommend using TF2 for any new developments. In the end, if something does not work, you will have to port it back to TF2. Even though there are not significant difference in the semantics of the framework


Xgboost (classification problem) feature importance per input not for the model

A partial dependence plot might be what you are after: these plots show the relationship between the probability and the input variable. The mlr package in R takes care of this.

Discussion of Partial Dependence on XGBoost Git: https://github.com/dmlc/xgboost/issues/486

General Tutorial using mlr: https://mlr-org.github.io/mlr-tutorial/devel/html/partial_dependence/index.html


Uneven dataset, how I should train the model if I can't load all pictures at once

From your tags, I see that you use keras. Keras offers you the class ImageDataGenerator which has the method flow_from_directory() (see here). This method loads the images in your training directory batch by batch from the hard drive and stores only the current batch in the RAM. This eliminates the bottleneck you currently face when loading the images.

To tackle the class imbalance, the recommended method is to use the class_weight argument of the keras classifiers. This argument assigns a weight to every class in your data, allowing you to give higher importance to images from the minority classes. This answer shows how you can calculate the class weights.

In the code below, I put that all together:

# Define constants - change them according to your requirements
BATCH_SIZE = 128
EPOCHS = 50
IMAGE_SIZE = 224

# Set up Image Data Generator
train_datagen = ImageDataGenerator(dtype=np.float16) # here you can also do some data augmentation

# Set up flow from directory
train_generator = train_datagen.flow_from_directory(directory="path/to/your/directory",
                                                    class_mode="categorical",
                                                    target_size=(IMAGE_SIZE, IMAGE_SIZE),  # resize the images if required
                                                    batch_size=BATCH_SIZE)

# Calculate class weights
counter = Counter(train_generator.classes)
max_val = float(max(counter.values()))
class_weights = {class_id: max_val/num_images for class_id, num_images in counter.items()}

#
# Here you set up your model ...
#

# After compiling the model, you fit it to your data using fit_generator
model.fit_generator(train_generator,
                    steps_per_epoch=train_generator.n // BATCH_SIZE,
                    epochs=EPOCHS,
                    class_weight=class_weights,  # use the class_weights as method parameter
                    verbose=1)


How is Importance-Sampling Used in Off-Policy Monte Carlo Prediction?

After some head-scratching, I think I was able to make sense of the "transformation" that the author uses in equation 5.4 to yield the correct expectation for $v_{\pi}(s)$. I've introduced some notation to help clarify what's going on. In particular, let $G_t^{(\pi)}$ denote the return after time step $t$ by following policy $\pi$. Then our goal is to estimate $v_{\pi}(s)$ for the target policy $\pi$ from an episode generated under behaviour policy $b$. Note that the state-value function for $b$ is defined as

$$v_b(s) = E[G_t^{(b)} | S_t = s]$$

We now apply the multiplication by the importance-sampling ratio proposed in equation 5.4 and expand using the definition of the expected value of  $G_t^{(b)}$ as follows.

$$E[\rho_{t:T-1}G_t^{(b)} | S_t = s] = \sum_{g \in supp(G_t^{(b)})} \rho_{t:T-1} \cdot p_{G_t^{(b)}}(g) \cdot g$$

However, note that the probability of a particular return occurring under $b$ is just the probability of its sequence of state-action transitions (called the "trajectory" of a return in the text) occurring under $b$. Thus $p_{G_t^{(b)}}(g)$ is simply the probability of a particular trajectory
occurring under $b$, as given in the following excerpt from the text for the example of an arbitrary policy $\pi$.

Recall $p$ here is the state-transition probability. We now substitute this expression for $p_{G_t^{(b)}}(g)$ into our expanded version of equation 5.4, and substitute the definition of $\rho_{t:T-1}$ to obtain:

$$E[\rho_{t:T-1}G_t^{(b)} | S_t = s] = \sum_{g \in supp(G_t^{(b)})} \left(\frac{\prod_{k=t}^{T-1}\pi(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)}{\prod_{k=t}^{T-1}b(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)} \cdot \prod_{k=t}^{T-1}[b(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)] \cdot g \right) $$

Canceling like terms gives us

$$= \sum_{g \in supp(G_t^{(b)})} \prod_{k=t}^{T-1}[\pi(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)] \cdot g$$

Note that, as discussed earlier, $\prod_{k=t}^{T-1}[\pi(A_k|S_k) \cdot p(S_{k+1} | S_k, A_k)]$ is the probability of a given return trajectory occurring under the target policy $\pi$. In other words, we have:

$$ = \sum_{g \in supp(G_t^{(b)})} p_{G_t^{(\pi)}}(g) \cdot g$$

The support of $G_t^{(\pi)}$ is the same as the support of $G_t^{(b)}$: the support of either random variable is the set of all state-action sequences (trajectories) possible after a particular time step t. We conclude that

$$E[\rho_{t:T-1}G_t^{(b)} | S_t = s] = \sum_{g \in supp(G_t^{(\pi)})} p_{G_t^{(\pi)}}(g) \cdot g = E[G_t^{(\pi)} | S_t = s] = v_{\pi}(s)$$


Confusion on result of K-Fold Cross Validation and Independent Test set

Absolutely, I've had this happen to me. As Ben Reiniger pointed out in the comment, check your individual scores of the folds. I'd expect you to have some folds that score worse than your reported k-fold score and some that score better, maybe even better than the independent test set.

Some things I consider when this happens:

When you test on the independent test set, have you trained on the full training set?
What accuracy do you get if you predict on the training set after using it for training? Better or worse than the accuracy you got on the independent test set?

Lastly, consider the balance of classes in your training set, training folds, and your independent test set. Accuracy scores are sensitive to class imbalance, and may cause such a discrepancy in scores between sets if the balance shifts. If you suspect class imbalance is an issue, you could either balance it (e.g. undersample the test set) or use a different metric.


How to understand what each layer is learning in a Deep learning neural network?

I wanted to know if there are any papers who have tried doing this (links would be really helpful).

The best known paper on this topic is Visualizing and Understanding Convolutional Networks by Matthew Zeiler and Rob Fergus. But you can find many more papers and online articles that do the same.

  Meanwhile are there any other ways to understand what happens in each hidden layers?

Unfortunately, deep neural networks are like black boxes; we have little understanding of their inner workings. Visualizing the activations of the layers is probably the best way to tell what features they extract.


does random forest require more trees to be built if # rows increase in train data

The best way to look at this problem is to consider the extreme case: let's say you duplicate every row in your dataset.

If you train a decision tree directly, there are certain hyperparameters you will want to change. For instance, in sklearn, you will want to double hyperparameters like min_samples_split or min_samples_leaf because in each node you'll have double the observations.

With regards to a random forest, usually, a random forest is learn by bootstrapping with resampling=True in the entire dataset. By doubling every observation, you make each observation just as likely to be picked. Even if you use a random forest which subsamples, doubling the dataset makes no difference.

Now, you are not doubling your data. You are introducing new data which might change the distribution of your data. To the extent that it does, then it will mean that whatever hyperparameters you have previously used might no longer be optimal. So you should just try. But, to answer your question, no.


XGBoost feature importance has all features but decision tree doesn't

XGBoost iteratively trains many trees (=boosting). So there is not only one tree.
For example you can plot a single tree, see: https://machinelearningmastery.com/visualize-gradient-boosting-decision-trees-xgboost-python/
A basic decision tree algorithm creates just one tree. If you apply pruning to the tree not all features would be present in the tree. The first split would be the one with the highest importance, ...


Refer to my comment, I believe what you need is

df <- read.csv("./yourpath/yourfile", sep = ";", header = TRUE) # play around with the arguments per your file.


How are you solving this? How are you keeping track of the work done?
  What's your logbook tool?

This might not be the best approach. But, this is how my team does it. We believe that for pulling off an end-to-end data science experiment, proper conscience is very important. So, we use Slack for the same for our discussions and the meetings.

In addition to them, we have Rmd (R markdown) files for documenting the planning and the analysis parts.


"Zero-meaned" means the vector has been transformed so that its mean is 0.

Typically, you would do this by subtracting the mean of each column from that column. (This is for dimensional as well as algorithmic reasons; you don't want to subtract a person's weight from their height.)

It sounds like here they're actually talking about the row mean--that is, $(-0.6946377, 12.680544,  0.50395286)$ would be transformed to $(-4.857924, 8.5172577, -3.65933344, 4.1632863, 7.40047)$, where the first three are the original features minus the row mean, the fourth is the row mean, and the fifth is the standard deviation of the original features.

This would make sense if the three have the same units (if they're all accelerations at the same scale, this works), and so you want a separate measure of how much it's being accelerated at all and how much it's being accelerated in a particular direction.


No loss decay while learning Neural Network for X-OR operation in torch

Finally I found the error in my Network.
1. I haven't added the non-linear layer after the first linear layer.
2. No randomisation while running stochastic gradient descent.

By updating these two things, now it is working fine.
Updated Code:

require 'torch'
require 'nn'

m = nn.Sequential()
m:add(nn.Linear(2,2))
m:add(nn.Tanh())
m:add(nn.Linear(2,1))
m:add(nn.Sigmoid())

--print(parax_m)

criterion = nn.BCECriterion()

inputs = torch.Tensor(4,2)
inputs[1][1] = 0
inputs[1][2] = 0

inputs[2][1] = 0
inputs[2][2] = 1

inputs[3][1] = 1
inputs[3][2] = 0

inputs[4][1] = 1
inputs[4][2] = 1

targets = torch.Tensor(4,1)
targets[1][1] = 0
targets[2][1] = 1
targets[3][1] = 1
targets[4][1] = 0

function trainEpoch(m,criterion,inputs,targets)
    for i=1,inputs:size(1) do
        local idx = math.random(1,4)
        local input = inputs[idx]
        local target = targets[idx]
        local output = m:forward(input)
        --print(output)
        local loss = criterion:forward(output,target)
        print(loss)

            -- backward
        local gradOutput = criterion:backward(output,target)
        m:zeroGradParameters()
        local gradInput = m:backward(input,gradOutput)
        --update
        --module:updateGradParameters(0.9) -- momentum (require dpnn)
        m:updateParameters(0.01) -- W = W -0.1*dL/dW
    end
end

for i=1,10000 do
    trainEpoch(m,criterion,inputs,targets)
end

-- prediciton
testinput = torch.Tensor(4,2)
testinput[1][1] = 0
testinput[1][2] = 0

testinput[2][1] = 0
testinput[2][2] = 1

testinput[3][1] = 1
testinput[3][2] = 0

testinput[4][1] = 1
testinput[4][2] = 1

for i=1,testinput:size(1) do
    local output = m:forward(testinput[i])
    print(output)
end


The goal of having a "forest" (baggging ensemble) of trees is to make the prediction more solid. Individual decision tree tend to overfit, and with Random Forest the sampling, features selection and bagging helps to make a more robust score.
Its weird that you have an empty tree, since decision tree are greedy and they will fit to anything. Even if there is just random noise in your data they will make splits (depends a bit on how you have configurated it).
If you have an empty tree, random forest wont do nothing since is just a bunch of decision trees.
With out seeing your code, or nothing else I would guess that you have a coding error somewhere.


Handling features with multiple values for clustering

You can easily use get_dummy function in Pandas to convert them to numerical vectors.

The idea is that categorical variables do not have a numerical intuition e.g. when it comes to the definition of Distance. But just imagine you have one feature Genre with 3 values Comedy, Romance and Crime. Then you can model them in a 3-dimensional space by saying Comedy = (1,0,0), Crime = (0,1,0) and Romance = (0,0,1). It replaces 1 feature with three but intuitively works well.

Update

I just understood your question after editing it! It was a bit fuzzy previously. But I keep my initial answer and add an update.

In this case use the values of the feature Genre (unique values of union of all genre sets in that column) as new features and determine their presence with 1 and 0 otherwise. Should work.

Movie_ID    Action  Thriller  Drama  Romance  Comedy

1             1       1         1       0       0
2             0       0         0       1       1
3             1       0         0       1       0


Why do internet companies prefer Java/Python for data scientist job?

So you can integrate with the rest of the code base. It seems your company uses a mix of Java and python. What are you going to do if a little corner of the site needs machine learning; pass the data around with a database, or a cache, drop to R, and so on? Why not just do it all in the same language? It's faster, cleaner, and easier to maintain.

Know any online companies that run solely on R? Neither do I...

All that said Java is the last language I'd do data science in.


Removing duplicates and keeping the last entry in pandas

You can see from the documentation of the method that you can change the keep argument to be "last".

In your case, as you only want to consider the values in one of your columns (datestamp), you must specify this in the subset argument. You had tried passing all column names, which is actually the default behaviour. Now we can use this (along with the correct value for the keep argument) to get this:

For example, a dataframe with duplicates:

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'datestamp': ['A0', 'A0', 'A2', 'A2'],
                           'B': ['B0', 'B1', 'B2', 'B3'],
                           'C': ['B0', 'B1', 'B2', 'B3'],
                           'D': ['D0', 'D1', 'D2', 'D3']},
                           index=[0, 1, 2, 3]).T

In [3]: df
Out[3]:
  datestamp   B   C   D
0        A0  B0  B0  D0
1        A0  B1  B1  D1
2        A2  B2  B2  D2
3        A2  B3  B3  D3

Now we drop duplicates, passing the correct arguments:

In [4]: df.drop_duplicates(subset="datestamp", keep="last")
Out[4]:
  datestamp   B   C   D
1        A0  B1  B1  D1
3        A2  B3  B3  D3

By comparing the values across rows 0-to-1 as well as 2-to-3, you can see that only the last values within the datestamp column were kept.


You can try and use the tf.keras.preprocessing.image.ImageDataGenerator's flow-from_directory() method. Here are a few links for reference:

tf.data: Build TensorFlow input pipelines

Retraining an Image Classifier

Notebook on Image Captioning


In a Time Series Problem, is it possible to forecast quantities by learning the patterns of other items? What are my options?

If I am only concerned about Red Apples, I can just use ARIMAX with data pertaining to only red apples or VAR right?

Yes, is possible, you will lose information (perhaps valuable) but you can work with it.

But what if I suddenly introduce a new apple type such as orange apples and only have a history from T1-4?

Given I need to use T1-4 for the orange apples... Is it possible to use other data from the red, green and yellow to aid in properly calculating a QN value for orange apples at T5?

You will be limited to the knowledge of the orange apples but the other variables might help you obtain enough information until you could have a more robust model, 4 entries might not be enough, but the model might help you obtain an estimation


AUC with sklearn vary each time script is started

Ok  I miss the fact that you can set also random_state here to remove this variability... but I don't see how it works here exactly so if someone want to explain...your are welcome.

   clf = tree.DecisionTreeClassifier(random_state=5)


Improving model prediction for small single data sets by using multiple data sets to fit

One option is merge nearby data points, this is call binning or bucketing. Binning increases the amount data for a specific target and sometimes increases the signal-to-noise ratio. Binning works very will location based data.


Which supervised ML model to use for exam/grade prediction?

First you need to figure out what type of variable the response is. Is it categorical (eg. A, B, C, etc) or continuous (75%, 83%, 92%, etc)? If it's categorical, it'll definitely be ordinal. This means you can explore methods like ordered logistic regression. If it's continuous, look at something like beta regression.
For the app - if the student only keys in 3 of the 5 mock exam results, you could just get them to 'guess' their results for the other 2 mock exams. In my opinion, this is better than inputing the remaining 2 mock exam outcomes. This is since the student should have a good handle on they're going to perform.


pip install transformers

Then try this

from transformers import pipeline
nlp = pipeline("fill-mask", model="bert-base")
nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")


If you use tree-based algorithms like random forests the data distribution should not be an issue. Linear algorithms are more dependent on the distribution of your variables. To check if you overfit can try to predict your training data and compare the result with test data. The score depends on your evaluation metric. If you use scikit-learn you get R^2 as your metric.

The coefficient R^2 is defined as (1 - u/v), where u is the residual
sum of squares ((y_true - y_pred) ** 2).sum() and v is the total sum
of squares ((y_true - y_true.mean()) ** 2).sum().


How to persist data scaler for predictions

So after looking some more I found this question: save scaler model in sklearn
which says that one can picle the scaler aswell, which works.


Dealing with large data: selecting a sample

9300 cases is not a lot, this doesn't even fall within the realm of big data or anything were you would have to apply special techniques.

Also given a more restricted feature set there are almost no computational issues you should encounter unless you work on an actual brick or Atari. As an example benchmark I am able to fit a model for >4M cases with up to 20-30 features (some with ~40 levels) on my 16gb Laptop in <10mins.

So let's focus on the real issue, the complexity of the data i.e. the amount of available features/feature levels and most importantly efficient code.

Assuming you don't have the resources to simply throw some cloud processing power on this problem and call it a day we have to restrict the complexity.

Note that this does not mean "sampling the data" which would indicate using only a subset of cases, as said in the beginning this isn't your problem.

Instead let's workshop ways to cut down computational requirements:

1. Optimize your code

This sounds simple but research and make sure that you are using the best packages for the job, adjusted parameters correctly and cut fat from your code. This is hard work but it is necessary and worth it.

E.g. have you checked that all code is parallelized as much as you can and runs on all CPUs?

2. Optimize your data structure

Factor levels normally aren't a problem computationally speaking but even so >100 levels might be too hard to understand anyway. Try to transform your data by e.g. grouping certain factor levels together. Also try to pivot parts of your data e.g. by wide to long data.

3. Use your domain knowledge

You surely understand your business and this problem. Don't just throw all the data at it! Think for a minute and remove unnecessary or duplicate columns.

4. Work your way upwards

ML modeling has a simply yardstick, if the validated prediction quality is good, your model is good. If you manage to do that with two variables, nobody cares.

So start with a bare-bones data set that includes ALL cases (still keep split-sample validation in mind here) but only the 2-3 simplest, most relevant predictors (based on domain knowledge) as well as the depended variable.

Fit a model and check it's quality, is it good enough? Yeah your done! If not include the next round of variables...

5. Modulize your EDA and have patience

I cannot stress enough that your data set really isn't that big or complex, so optimize your code first. BUT if basic EDA needed to find the best features still takes too long, do the following:

Structure and modulize your code by e.g. doing a PCA for only chunks of the columns (again use all case but only a subset of variables).
Let it run one-by-one and just have patience if the code runs for an hour, sometimes that is how it goes.
Look at the results, compare and decide how to merge results.

6. Talk to your boss

Yes I said I assume you do not have the resources to throw more power on it but if this is an important business problem there should be resources for cloud computation. Fitting a model on AWS is dirt cheap and will only take seconds, so realistically we are talking about <100$ here to train multiple models.

If you don't have a boss to pay for this stuff or he doesn't want to, try free resources. There are a lot of cloud based services that offer enough free GPU or CPU time that you should be able to fit a model.


Do records with the same key in two RDDs repartitioned by key reside in the same node in spark?

The tuple of one partition is always on the same node because a partition itself is impartible. So if you do a groupBy or write your own partitioner which partitions by key, all records with the same key/partition number will be shuffled to the same node.

Otherwise, transformations like mapPartition which pass an iterator to a user defined function wouldn't work.


AttributeError: 'numpy.ndarray' object has no attribute 'img_to_array'

You are using the variable name image in your code, while you also import a module named image. This conflict results in trying to find img_to_array method in the variable rather than in the module.

Use a different name for you image variable, or directly import the img_to_array function:

from keras.preprocessing.image import img_to_array
[...]
x = img_to_array(img1)


Q&A answer comparison multiple sentences using

It depends on how sophisticated you want the system to be:

the most basic way is to compare the user answer with the gold answer using a simple string similarity measure, such as the overlap coefficient. Basically it just counts the words in common, and there would be a minimum threshold to count the answer as correct (e.g. 80% words in common). It's not very good because a small typo is enough to make the score wrong and it gives the same importance to every word.
The same idea but with TF-IDF weights, typically with cosine similarity. This requires a corpus on which to calculate the IDF weights (which reflect the importance of the words in general).
Still based on string similarity measures but more advanced: a hybrid similarity measure which combines character-level similarity between words (e.g. Jaro, Levenshtein edit distance) and similarity across words. Soft-TFIDF is a common example. Disadvantage: can be tricky to properly adapt to the task.
Beyond that there are a lot of fancy options: using semantic similarity with WordNet (synonyms), words embeddings, etc.

Note: fyi this is not related to the task called Question Answering, which is about a computer-generated answer to a question.


Your first dataframe.shape is showing you that you have no rows.  Pandas is probably treating your data as just the column names, so try pd.read_csv('train2.csv', header=None).