What kind of algorithms can be used as a stacker in stacked generalization?

There is no preference for stacked generalization. You can use any algorithm whether it be a NN, RF model, XGB model, etc. The only thing that you need to take care of is the fact that the algorithm which you are applying to your model is useful to your data or not. Also, model stacked at level 1 may not be a very good model for level 2. For example, suppose at level 1 you stacked an XGB model with max_depth=12, eta=0.01, colsample_bytree=0.7, but at second level it might happen that the same depth is just an overkill and hence you  should discard using that model at second level. So, in short there is no preference for stacked models, the preference is for the data on which you are going to train your model.


Here is one way to generate two data frames based on the data.

# read data
dat <- read.csv("sample csv1.csv", stringsAsFactors = FALSE, header = FALSE)

# find rows containing "Interval"
idx_int <- dat[[1]] == "Interval"

# calculate groups
group <- cumsum(idx_int) %% 2

# remove indices for "Interval" rows
group2 <- group[!idx_int]

# split data frame into two data frames
dat_list <- split(dat[!idx_int, ], group2)

# generate two separate data frames
dat_energy <- dat_list[[2]]  # Energy
dat_demand <- dat_list[[1]]  # Demand


What is the "novel reinforcement learning algorithm" in AlphaGo Zero?

The AlphaGo Zero article from Nature, "Mastering the Game of Go without Human Knowledge", claims four major differences from the earlier version:

Self-learning only (not trained on human games)
Using only the board and stones as input (no hand-written features).
Using a single neural network for policies and values
A new tree-search algorithm that uses this combined policy/value network to guide where to search for good moves.

Points (1) and (2) are not new in Reinforcement learning, but improve on the previous AlphaGo software as stated in the comments to your question. It just means they are now using pure Reinforcement Learning starting from randomly initialized weights. This is enabled by better, faster learning algorithms.

Their claim here is "Our primary contribution is to demonstrate that superhuman performance can be achieved without human domain knowledge." (p. 22).

Points (3) and (4) are novel in the sense that their algorithm is simpler and more general than their previous approach. They also mention that is is an improvement on previous work by Guo et al.

Unifying the policy/value network (3) enables them to implement a more efficient variant of Monte-Carlo tree search to search for good moves and simultaneous using the search tree to train the network faster (4). This is very powerful.

Furthermore, they describe a number of interesting implementation details like batching and reusing data-structures to optimize the search for new moves.

The effect is that it needs less computing power, running on 4 TPUs rather than 176 GPUs and 48 TPUs for previous versions of their software.

This definitely makes it "novel" in the context of Go software. I believe that (3) and (4) are also "novel" in a broader context and will be applicable in other Reinforcement Learning domains such as e.g. robotics.


Calibrate the predicted class probability to make it represent a true probability?

"Not all classification models are naturally probabilistic" is the first line of this chapter [1] .
This means that some models do output probabilities, and some do not. This is highly dependant on the model structure as well as of the objective function that it uses.

In particular, for neural network, a commonly used objective function is Cross Entropy. And to my understanding, Cross Entropy does try to approach the true distribution of the target variable, conditionned on the input variable. This would mean that for a hundred similar cases $X_{similar}$, that are similar in terms of the information that the model is capable to understand, if 80% of those cases are dogs, the predicted score will be a descent approach of the probability you want to estimate: $ \hat{y} \approx p(dog|X_{similar}) = 80/100$.

For other models with less "statistical" behaviour, such as Decision Tree, SVM, of models taken into a Boosting or Bagging process, the output distribution ($\hat{y}$) is skewed. In this binary classification setting, these models would try to give a score as close as possible of the target value. In the case at hand, they would consider the $X_{similar}$ as a group, and the best decision for that group in terms of accuracy beeing to classify them all as dogs, $\hat{y}$ would be skewed toward 1 value.

So, my take on this :

If your goal is to have accurate probabilities, use a model that directly outputs probabilities. If the probabilities are "perfect", a cutoff value of 0.5 would provide the best accuracy (i.e. $\hat{y}\geq{0.5}$ as a decision rule)
If your goal is to have a good classifier in terms of accuracy, use the model that... has the best accuracy. Note that in this case, the best cutoff value may not be 0.5 (it can even be far from it).
If you have both those goals, well, I do not have the answer as this is data and model dependant. If your scores are not probabilities, calibration should make them look more like it indeed. See Calibration in classification for a list of calibration methods.

[1] Wikipedia - Probabilistic Classification  - Probability calibration

[2] Wikipedia - Calibration (statistics) - In Classification


Is there any time series model which handles data at variable frequencies.?

It looks like stock price data. You can use fbprophet(python),BATS(R),TBATS(R) to predict the data.
You said your model contains different frequencies which mean it contains more than one seasonality.
To find seasonality(frequencies) you can use Fourier transform. You should have significant data to get a good prediction


Boruta feature selection in R with custom importance (xgboost feature importance)

Boruta is by universal reputation dog-slow and not very good. Boruta runs take many hours or days. VIF feature-selection algorithm is not objective, anyway.
You can program your own feature-selection that runs faster. I ran Boruta a few times on various datasets and it wasted 4 days of my time, and the result was inconclusive.

Here's the fast-and-dirty not-100%-scientific approach, culled from consensus in a lot of Kaggle competitions and suchlike. It's 10-1000x faster than Boruta and probably more accurate.

Run some fast exploratory trees (RF/XGB), i.e. not very deep, not too many trees. Extract feature importances. Repeat for several random seeds and look at average or absmax feature importances.
For now, throw out very-high-cardinality features (e.g. user-ID, zipcode etc.), since trees won't tend to split on them until very deep and small nodes, so initially exclude them. Let's define "Very-high-cardinality" as >= ~0.3 distinct values per record, or less in a larger dataset.
Throw out all near-zero features, zero- and very-low-importance features (< 0.02-0.05 f.i. or lower is a rule of thumb).
Measure the matrix of which features are highly correlated (< 0.4) to each other. Corrplot is a nice way to visualize these, and order correlated features near each other). Pick a sensible color-scale. Identify subgroups of correlated features. Arbitrarily select one feature from those groups (you can revisit this selection later). Or you can decide if this warrants using PCA.
Rerun, your CV accuracy should have gone way up, and your training time will be way faster (since tree training is quadratic to no. of features). Revisit choices of borderline-low-important or correlated feature, and again use multiple random seeds.
For exploratory, 5-fold CV is fine, but if training time is an issue, 3-fold works. 10-fold is unnecessary.
For the subset of remaining features, use a 'respectable' method. See the excellent posts on CrossValidated in feature-selection.

Somebody with a PhD in Stats will probably wag their finger at me for this :)


How can I predict rank of a team based on list of team members and past placement?

This sounds like an interesting problem. Based on what you're trying to achieve, I think you should consider a more classical approach first, rather than diving straight into a neural network. I would consider something like an ordered logistic regression model. This is an extension of logistic regression, handling two or more ordered outcomes.
Since you have your data in a json file, I'd first convert this to a csv. Then load the csv into Python to estimate/train any models. You can then use the data to estimate an ordered logistic regression model. See the Python documentation here.
Your response will be rank (1-10). You can use the characters and items used as predictors. You could also look at interaction terms between the characters and predictors - this is because certain item and character combinations impact the outcome differently. One concern is that this model might become quickly over parameterised. Next you could consider a neural network.
In the neural network, your input nodes will be the characters and items. You'll need to perform a one hot encoding on the rank before you can use it to train the network. Note that you should use a cross entropy loss function, with a softmax activation function on the output layer. It's important to note that this doesn't achieve exactly the same thing as an ordered logistic regression (we're ignoring the fact the outcome is ordered). See this question about setting up a neural network mimic an ordered logistic regression.


There is a couple of separate questions I can identify here: one is whether it is appropriate to include coordinate features and the other is whether it is appropriate to convert a multivariate regression (more than one outcome variable - 9 in your example) into a regression with single dependent variable.

Coordinate features

If you think that the dependent variable value in a particular cell of your grid is somehow affected by the location of the cell within the grid (in other words, that values in the same cell are correlated), then you should probably include coordinate features (which can be simple cell identifiers or something more complex, like a more specific identification of where the cell is located within the grid relative to a particular point).  It is possible to even use target-based encoding (i.e., use the mean of dependent variable values in the same cell as a new feature), though that gets into issues like data leakage and overfitting (which can be addressed separately).  You may also possibly want to create interaction terms, allowing for the effects of non-coordinate features to vary by level of a coordinate feature.

This is different than modeling correlation between values across different cells.  If you expect that there is some spatial autocorrelation between cells in this 2-d grid, e.g., a value in cell (1,1) is correlated with values in cells near it: (1,2), (2,1), (2,2), you should probably model that relationship by including some set of spatial features (not just coordinate identifiers), like spatially clustered covariates, or by using a spatially lagged dependent variable model.

If you are not sure if spatial correlation is present, you can estimate a model without any spatial features and then test for spatial autocorrelation (spatial dependence between residuals of regression) using Moran's I (a positive Moran’s I hints at data is clustered, a negative Moran’s I implies data is dispersed.)

Multiple vs single dependent variable

If you go with option 2 as you described it, with one output variable, you will be modeling a completely different variable than if you were modeling 9 (in your example) separate variables.  Each of the original 9 variables likely has its own statistical properties, but if you combine them into one, you will have a new variable that has its own properties (some combination of the 9).  So if you consider the grid to be 9 separate variables, you should stick with modeling 9 separate variables.


Seeking advice on database architecture -- given my problem, what tools should I learn?

I used R for several years but have since moved to Python and so I have a hard time understanding output.OLS's data hierarchy. Nonetheless, here are my thoughts.

In-Memory Databases: If you're struggling to fit your R object in memory, then I'm guessing the memory requirement is too great for your laptop (i.e., the problem isn't due a 32-bit installation of R). If that's the case, putting an in-memory db, like MongoDB, on your laptop will only transfer the memory problem to a new technology. In other words, 6 GB in R will be 6 GB in Mongo, so if your laptop simply can't handle 6 GB, then it doesn't matter how you put it there. You could set up a MongoDB on AWS, which could hold 100s of GB, but this can be expensive and also difficult if you don't have AWS experience.

On the other hand, MongoDB is good at storing hierarchical data like lists and hash tables / dictionaries (I forgot what R's term for that is) and is fast if you have enough memory.

Relational Databases (RDMS): I would recommend going this route and creating a wide and deep table. You can still get value out of an RDBMS even if your machine has limited memory because everything is efficiently stored on disk. If you create table indexes on the columns you expect to use/filter/select most, you can very quickly query the data that you want even if your data is multiple GB. If you don't create table indexes, your queries will be horribly slow, and you'll probably abandon the RDBMS effort entirely (so make sure to index).

A long time ago my go-to package for accessing MySQL through R was RMySQL. MySQL has been around for many years, so you'll be able to find most answers to your problems on Goggle, but MySQL user interfaces like Sequel Pro (for Mac only) have sort of fallen behind on maintenance, so setting up tables and databases can be a little more challenging than they used to be. You'll now have to go the command-line route.

Postgres has now replaced MySQL as the open-source DB of choice, but it lacks good user-interface support and so you'll probably have to set everything up via the command line. I started using Postgres after I stopped using R, so I never used a Postgres R package, but it looks like there is one here.

Whether you go with MySQL or Postgres, you'll probably have to use the command-line outside of R to set up the database, but after that creating, dropping, joining, or querying tables can all be done through R. RDBMS systems are also great because you can guarantee data quality (no duplicates) by creating primary keys -- I found this feature very helpful for my R modeling. In any case, SQL is a very basic language that every data-oriented person should know so setting up an RDBMS and possibly learning SQL are skills worth learning.


Risk score from Neural Network classifier (more than 2 categories)

How about doing in an easier way?

Perform the classification in normal way, the greater the score the greater the level. Then, for the final output, just reverse the level.

For example, from the classification, the lowest score comes to level 1 and the highest score comes to level 10. Then, make level 1 as level 10 and level 10 as level 1 for the final output


Predict Ethereum price up or down by given Bitcoin price

So, the question asks how to predict the price Etherium given the price of Bitcoin.
It is worth doing some Explanatory Data Analysis (EDA) to see if there is a correlation between these two variables.
You might find that the relationship between the two variables is non-linear, which indicates that maybe the model that you would make with Bitcoin price feature is going to be too simple and will lead to undercutting. Therefore, it would be worth adding more complexity by increasing the number of features, which could also be correlated with Etherium prices.


From wikipedia:

  ..., the binomial distribution with parameters n and $\rho$ is the discrete probability distribution of the number of successes in a sequence of n independent experiments, each asking a yes–no question, and each with its own boolean-valued outcome: a random variable containing a single bit of information: success/yes/true/one (with probability $\rho$) or failure/no/false/zero (with probability $\rho = 1 − \rho$).

So if you know that logistic regression is performed in order to model a binary output variable to some modelling question (i.e. to give 0 or 1, yes or no, etc.), it would make sense to base any probabilistic assumptions on a distribution, which shares this feature. Therefore, a binomial distribution may make sense compared to a continuous distribution, such as a Gaussian or Cauchy.


What is the appropriate statistical significance test for multi-class classification?

Generalisation of Mcnemars is called Cochran–Mantel–Haenszel test.
There is an implementation in R, but I suppose porting to Python should not be too hard. You can find the R version here.


Generally models can be compared by evaluating them. It depends on the specific application, for instance if the goal is to predict the next element(s) in a sequence then one can take a subset of the data as test set and see how accurately the model predicts the next element.


Use pretrained word vectors over custom trained word2vecs

The entire philosophy of Distributed Word Representations makes use of the fact that a word is understood by the context it has . When we say context , we mean the words that come in the neighborhood of a particular word . Now context in natural language is a tricky thing . As an example . the words open and create are two words that are not so similar semantically . But think of the scenario of a Bank related corpus , you will frequently see statements like

                     How do I open a new account
                                  &
                    How do I create a new account

In the context of such a document , open and create become similar words. That is why context specific word vectors are needed.

Now , when it comes to which is the better option between the two

Use pre-trained vectors
Use custom vectors ,

it depends on how much data do you have for your custom use case . If you have enough data then it's always safe/better to go for a custom vectorization as it will be very specific to the context the corpus has . But in all other cases , you can use the pre trained embeddings. These generalize quite well on a variety of docs.


sklearn.accuracy_score(y_test, y_predict) vs np.mean(y_predict == y_test)

Both of these methods act differently. You will only get the same results in very few cases or if you are testing only one row at a time.

np.mean(y_test==y_pred) first checks if all the values in y_test is
equal to corresponding values in y_pred which either results in 0 or 1. And then takes the mean of it (which is still 0 or 1).
accuracy_score(y_test, y_pred) counts all the indexes where an element
of y_test equals to an element of y_pred and then divide it with the total number of elements in the list.

For example-

  import numpy as np
  from sklearn.metrics import accuracy_score
  y_test = [2,2,3]
  y_pred = [2,2,1]
  print(accuracy_score( y_test, y_pred))
  print(np.mean(y_test==y_pred))

This code returns -

  0.6666666666666666
  0.0

You will get the same result from both the method if you have only one sample/element to test. You can find more details here on accuracy_score and np.mean.

Also, accuracy_score is only for classification data. As mentioned in the first line here.


How to decide which method to use TFIDF, or BOW

There is no specific way to deal with these kinds of experimentation. Below are some important points to remember before doing experimentation

If you are using NN to do the work, dense vectors like word2vec or fasttext may give better results than BoW/TfIdf

If you have more OOV words then fasttext may give better output than basic Word2Vec

If you are using linear algorithms like Logistic Regression/Linear SVM, BoW/TfIdf may have some advantage over averaging all the word vectors in the sentence. But it's not always true.

For the tree-based algorithms, training time may increase if we use BoW/TfIdf features because of the huge feature-length.


Difference between rbfnn and svr with gaussian kernel

Your description of the model functioning is correct. Structurally, both models are representatives of the so called kernel methods. As such they are very similar, the same in many cases.

What is completely different between the two methods is the way the kernel centers and linear coefficients are derived. My experience with SVM is limited, so I cannot go into the details of the training method, but in summary is consists of solving the constrained optimization problem of finding the best approximation subject to maximum coefficients. This is achieved by linear programming or through a method like Sequential Minimal Optimization (SMO).

Training RBFNs, on the other hand, is a different story. Identifying the kernels is done in one of two ways usually; either by using a clustering algorithm or by using Orthogonal Least Squares (OLS). In either case, the linear coefficients are identified as a second step through least squares (LS).

The differences in training mean that, even through in structure the resulting models may be the same, in functioning they will most probably be completely different, due to the different fitting procedures.

Some references:

J. C. Platt, “Fast Training of Support Vector Machines Using Sequential Minimal Optimization,” Adv. kernel methods, pp. 185–208, 1998.

J. Moody and C. J. Darken, “Fast Learning in Networks of Locally-Tuned Processing Units,” Neural Comput., vol. 1, no. 2, pp. 281–294, Jun. 1989.

S. Chen, C. F. N. Cowan, and P. M. Grant, “Orthogonal least squares learning algorithm for radial basis function networks,” IEEE Trans. Neural Networks, vol. 2, no. 2, pp. 302–309, Mar. 1991.


How to train with cross validation? and which f1 score to choose?

I think you confused some technical names. Cross-Validation is the name of the procedure, and it has some techniques or approaches such as k-fold cross-validation, train test split, etc. All are techniques to measure the performance of a model.
In your case, you have the first model that is assessed using 10-fold cross-validation and has an f1-score of 0.941, and the second model is assessed using the train test split approach and has an f1-score of 0.953. In this case, choosing the better model depends on what you want to give the privilege. In other words, whether you focus more on False Predictions or True Predictions, or False Negatives of False Positives. For this purpose, check the confusion matrix below.

There are several scenarios which mainly are:

Using Recall metric

You can use it if False Negatives are more important for you. In the medical analysis, for example, False Negatives are tried to be minimized and scientist measures model's performance essentially by using Recall. Because, for example, it is a more acceptable case to predict a person cancer but actually that person is healthy while compared to the case where a person is labeled as healthy but in actual that person has cancer.

Using Precision metric

It is preferred when False Positives are more important than False Negatives.

Using F1-score

It helps to identify the state of incorrectly classified samples. In other words, False Negative and False Positives are attached more importance.

Using Accuracy score

It is mostly used when True Positive and True Negatives are prioritized.
So back to your question, you should not choose the model that best 'fits' (or performs) in a test, but you should choose a model that fits your demands most and the test must be chosen depending on the demands. If you want to have balanced False Negatives and False positives use the f-score to choose a model, which is the model2 in the above-mentioned case. However, if you focus on a different target, make your choice according to a performance metric.


In random forest, can features with high Gini importance show insignificant p-values in t-tests?

Intuitively, wouldn't features with high Gini importances have very different values between classes? I'm wondering how to reconcile these two facts.

If I understood you correctly, these two facts perfectly reconcile in case your classes are not separable in one-dimensional feature space.

The following exaggerated toy example should illustrate such situation (and power of RF).

#uncomment the following line to remove all variables
# rm(list=ls())

library(ggplot2)
library(caret)
library(MASS)
library(GGally)
library(dplyr)

SEED <- 123456
SIMS <- 1000

set.seed(SEED)

x.signal <- rbind(
    mvrnorm(n = SIMS/2, c(0,0), matrix(c(1,0.9,0.9,1),2,2)),
    mvrnorm(n = SIMS/2, c(0,0), matrix(c(1,-0.9,-0.9,1),2,2)))

x.noise <- mvrnorm(n = SIMS, c(0,0), matrix(c(1,0.9,0.9,1),2,2))
y <- rep(c(TRUE, FALSE), each=SIMS/2)
df <- data.frame(
    x1.signal=x.signal[,1],
    x2.signal=x.signal[,2],
    x1.noise=x.noise[,1],
    x2.noise=x.noise[,2],
    y=factor(y))

yName <- "y"
xName <- setdiff(colnames(df), yName)

ggpairs(df, aes(colour=y))

ctrl <- trainControl("cv", number = 5, verboseIter = TRUE)
rffit <- train(df[,xName], df[,yName],
    method = "rf", trControl = ctrl, tuneLength = 5)

varImpPlot(rffit$finalModel)

## t = 0.061767, df = 997.98, p-value = 0.9508
t.test(filter(df, y=="TRUE")$x1.signal,
    filter(df, y=="FALSE")$x1.signal,
    var.equal = FALSE, paired = FALSE)

## t = 0.22695, df = 996.04, p-value = 0.8205
t.test(filter(df, y=="TRUE")$x2.signal,
    filter(df, y=="FALSE")$x2.signal,
    var.equal = FALSE, paired = FALSE)

dplyr::group_by(df, y) %>%
    dplyr::summarise(mx1=mean(x1.signal),
    mx2=mean(x2.signal))


Similarity measure for multivariate time series with heterogeous length and content

Try this recent paper: Consistent Algorithms for Clustering Time Series.

Your question is very much a current research topic.

Here's an older but excellent paper which talks about the fundamentals: Generalized Feature Extraction for Structural Pattern Recognition in Time-series Data.


In the above image the big square is your $f_m(x)$.

The big square is made up of small partitions. Those partitions are your smaller functions(Partitions) which performs the comparison between your variables and try to find out best possible split.

Here each partition has nearly same labels for classification and nearest numerical values for regression.

The square representation is for 2 dimensions if you want to get for higher dimensions refer the last one with 3 dimensions.

Reference: Elements of Statistical Learning, Basics of Decision trees


How Object oriented programming approach can be utilised in data science domain?

This is a really interesting question about the motivation behind Object orientated programming.
The main reason that makes OOP particularly attractive is the reusability of functions and objects which you can reuse in different analysis.
For example, for one NLP model, you create a function which cleans text (returns lowercase text, free from stopwords). A few days later you want to generate another NLP model which preprocesses text in the same way as your previous NLP model. Here you can put the clean text method instead a Preprocessing class and reuse the clean text method to clean the text in the same way as before.
Another reason behind OOP is its legibility. I find it is noticeably easier to track where bugs arise in OOP code than functional programming code. This, in turn, makes it easier to test and debug separate components within a class.
This is my take on it, and I am sure everyone has their own way of programming that suits you and the teams you work in.
I look forward to reading other people's views/answers to this query.


Looking for example infrastructure stacks/workflows/pipelines

In order to understand the variety of ways machine learning can be integrated into production applications, I think it is useful to look at open source projects and papers/blog posts from companies describing their infrastructure.

The common theme that these systems have is the separation of model training from model application. In production systems, model application needs to be fast, on the order of 100s of ms, but there is more freedom in how frequently fitted model parameters (or equivalent) need to be updated.

People use a wide range of solutions for model training and deployment:

Build a model, then export and deploy it with PMML

  AirBnB describes their model training in R/Python and deployment of PMML models via OpenScoring.
  Pattern is project related to Cascading that can consume PMML and deploy predictive models.

Build a model in MapReduce and access values in a custom system

  Conjecture is an open source project from Etsy that allows for model training with Scalding, an easier to use scala wrapper around MapReduce, and  deployment via Php.
  Kiji is an open source project from WibiData that allows for real-time model scoring (application) as well as functioanlity for persisting user data and training models on that data via Scalding.

Use an online system that allows for continuously updating model parameters.

  Google released a great paper about an online collaborative filtering they implemented to deal with recommendations in Google News.


If I have to recommend 10 movies to the users

Use a recommender system.

I suggest you begin by reading an introduction to recommender systems.


Count the number of false positives with respect to the first class

I always find the notion of false positive and negative confusing, especially when it comes to multi-class problems.

A good rule of thumb that I came up with is the following

True positive: "I predicted that you were a certain class, and I was right"

False positive: "I predicted that you were a certain class but I was wrong"

True negative: "I predicted that you weren't a certain class, and I was right"

False negative: "I predicted that you weren't a certain class, but I was wrong"

So in your case, there are 31 true positives with regard to the first class since you predicted 31 times that something was class 1 but you were wrong.


The result that you are calling awkward is indeed what we expect from the definition of PMI. Here are some identities involving PMI:

$\operatorname{pmi}(x;y) \equiv \log\frac{p(x,y)}{p(x)p(y)} = \log\frac{p(x|y)}{p(x)} = \log\frac{p(y|x)}{p(y)}$

Let $x=$'went' and $y=$'to go' and use the last expression. $p("to go"|"went") = 1$, and voilà, only $p("to go")$ remains in the denominator of the expression.


If it's just the input you like to decompose, you can preprocess the input data and use two input layers:
import tensorflow as tf

inputs_first_half = tf.keras.Input(shape=(300,))
inputs_second_half = tf.keras.Input(shape=(200,))

# do something with it
first_half = tf.keras.layers.Dense(1, activation=tf.nn.relu)(inputs_first_half)
second_half = tf.keras.layers.Dense(1, activation=tf.nn.relu)(inputs_second_half)
outputs = tf.keras.layers.Add()([first_half, second_half])

model = tf.keras.Model(inputs=[inputs_first_half,inputs_second_half],outputs=outputs)

data = np.random.randn(10,500)
out = model.predict([data[:,:300],data[:,300:]])

If you like to split after the input layer you could try reshaping and cropping, e.g,:
inputs = tf.keras.Input(shape=(500,))

# do something
intermediate = tf.keras.layers.Dense(500,activation=tf.nn.relu)(inputs)

# split vector with cropping
intermediate = tf.keras.layers.Reshape((500,1), input_shape=(500,))(intermediate)

first_half = tf.keras.layers.Cropping1D(cropping=(0,200))(intermediate)
first_half = tf.keras.layers.Reshape((300,), input_shape=(300,1))(first_half)

second_half = tf.keras.layers.Cropping1D(cropping=(300,0))(intermediate)
second_half = tf.keras.layers.Reshape((200,), input_shape=(200,1))(second_half)

# do something with decomposed vectors
first_half = tf.keras.layers.Dense(1, activation=tf.nn.relu)(first_half)
second_half = tf.keras.layers.Dense(1, activation=tf.nn.relu)(second_half)
outputs = tf.keras.layers.Add()([first_half, second_half])

model = tf.keras.Model(inputs=inputs, outputs=outputs)

data = np.random.randn(10,500)
out = model.predict(data)

The Cropping1D() function expects a three-dimensional input (batch_size, axis_to_crop, features) and only crops along the first dimension, therefore we need to add "pseudo-dimension" to our vector by reshaping it.


What is the neural network architecture behind Facebook's Starspace model?

Calling StarSpace a neural model would be misleading I think. You could certainly think of the it as a neural network with a single layer and a linear activation function, but I think don't think that would be very illuminating. They didn't discuss the architecture much in that paper for a reason- there isn't really any in terms of layers of neurons, activation functions, latent variables, or anything else except the constraints on the number of dimensions.

In fact, the most helpful way to think about StarSpace is  that, at its core like many (maybe most) popular embedding techniques across natural language, graphs, etc it's a low rank matrix factorization. What the sampling procedures are doing is using the data in some way to produce a positive definite gram matrix. It doesn't appear that way initially because that is done solely through sampling- if you were to find the expectations of each input/target pair, however, you'd find that the optimization objective is maximizing the expected value of vector similarity over a joint distribution on pairs of items minus a marginal distribution (this is due to the negative samples). Essentially, the goal is to maximize the difference in similarity between items that are frequently sampled and the similarity between items sampled independently from the marginal distribution.

If this sounds familiar to SGNS implicitly factorizing a shifted PPMI matrix or GlOVe explicitly factorizing a relaxed variant of the same, good. The specifics are different and StarSpace has significantly more flexibility in the sampling distribution it works with, but the principle is the same. "Neural Word Embedding as Implicit Matrix Factorization" and "Improving Distributional Similarity with Lessons Learned from Word Embeddings" are fantastic papers by Levy, 2014 and 2015 if I recall correctly, that discuss the connections between neural embeddings and explicit matrix factorization techniques like PPMI-SVD and glove and the principles that make them successful.

Similarly, "Network Embedding as Matrix Factorization: Unifying
DeepWalk, LINE, PTE, and node2vec" is a great discussion of the connections between neural network embeddings and the same implicit objectives as the neural word embeddings.

In short- it doesn't sound like there's much going on in StarSpace as far as architecture because there isn't. It's quite literally adjusting the placement of points in the embedding space to make associated items more similar to each other than to unrelated items.


Start with neural nets. They are the basic building blocks of Deep Learning.

Neural Nets for Newbies video is a well-explained introduction. CS-449 from Willamette University is an free and approachable class in neural networks.

That should lay the foundation for more advanced Deep Learning material. Deep Learning by Bengio, Goodfellow, and Courville is a quality book on the subject.


Unsynchronized time series visualization

As far as estimating similarity between the time series series there is a variety of methods you may want to investigate. Some of those are:

cross correlation: this will be affected by the amplitude and will not be able to estimate lagged correlations, prone to noise.
coherence: normalised frequency based correlation (cross-spectrum), not prone to amplitude or noise.
wavelet coherence: similar to above but based on wavelet transformations instead of STFFT.
dynamic time warping: measuring similarity between two temporal sequences, which may vary in speed.


Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?

The two slightly-smaller models will perform exactly the same, on average.  There is no difference baked in to the different trees: "the last tree will be the best trained" is not true.  The only difference among the trees is the random subsample they work with and random effects while building the tree (feature subsetting, e.g.).
Gradient boosted trees are a different story.  If you drop the first tree after you finish training, the resulting model will be mostly garbage.  Every subsequent tree was trained to improve upon the fit of the previous trees, and removing any single tree will put all future trees out of context.  (To give an extreme example, suppose the first tree actually captures "the correct" model.  All future trees will just fit on the remaining noise.)  On the other hand, removing the final tree is equivalent to having just trained one fewer tree, which may be good or bad depending on your bias-variance tradeoff at that point.


Siamese network return same loss value while training

Finally, I found a solution to this problem. I completely modified the architecture, Then I reduced the learning rate to 0.00005. Then the loss value of the model falls continuously.

Latest Architecture:

a = 'tanh'
model = Sequential()
model.add(Dense(self.INPUT_DIM, input_shape=(self.INPUT_DIM, ),
activation=a))
model.add(Dense(600, activation=a))
model.add(Dense(self.INPUT_DIM, activation=a))


How to make predictions of multiple input samples at once in tf 2 with keras

In my case, the Input Tensor Array was of length 30 but faulty constructed.

Make sure your inputs are right, otherwise it can lead to inaccurate results


Cluster algorithm to group events in more general domains

Some creative post-processing can be done. For instance applying Named Entity Recognition and simplify some parts (Manchester is a City). Using Knowledge Graph Analysis also gives some meta-info e.g. mapping your word to Wikipedia graph or using DBpedia may help you to recognize Named Entitiy categories (Obama and Merkel are politicians and NER does not necessarily capture their profession).

Note that the combination of named entity recognition and knowledge-base (Wikipedia, DBpedia, etc.) mapping is called Entity Linking.

Regardless of statistical learning for NLP techniques, all structures above are actually graphs so they can give you also the semantic similarity measure based on which you can use other clustering algorithms like Spectral Clustering and go to a semantically higher level of clustering.

Hope it helps :)


LightGBM and XGBoost Libraries can handle missing values

LightGBM: will ignore missing values during a split, then allocate
them to whichever side reduces the loss the most
XGBoost:   the  instance  is classified into a default    direction (the optimal default directions  are learnt from the data    somehow)

Finally, it is NOT a general property of gradient descent algorithms or tree algorithms. Only specific implementations of these algorithms have this property.

Personally, I believe that LightGBM and XGBoost can handle missing values effectively, in case one wants to keep them


Using random forest to learn Imbalanced Data (rare disease)

Use class weights, to weight errors, so that "incorrectly labelling a sick person as healthy" is penalized more than "incorrectly labelling a healthy person as sick".  Or look up any of the other standard techniques for dealing with class imbalance.


Assessing Group Similarities and Dissimilarities Post PCA

What does the distance between groups mean for your problem? Answering that problem would help you pick a distance metric.

Assuming you pick e.g. Euclidean distance, one simple way to find inter-group distance would be to first calculate the group centroid, i.e. average location of each group by averaging the locations of each group together.  You can do this by just averaging all the 5x1 vectors belonging to a group, and repeat for each group.  Then calculate the Euclidean distance between the resulting 6 centroids, giving you a 6x6 distance matrix.


What algorithm to use for a specific 'Named Entity Recognition'/'Information extraction' problem

Assuming your financial documents have a consistent structure and format and despite the algorithm kind of becoming "unfashionable" as of late due to the prevalence of deep learning, I would suggest that you try using Conditional Random Fields (CRF).

CRFs offer very competative performance in this space and are often used for named entity recognition, part of speech tagging and variants thereof.

Another great thing about CRFs is that they takes into account the position of the current 'field' in a document and also the preceeding and succeeding fields in order to calculate what the current class should be so you don't have to spend too much time engineering structural features into your model.

I have had a great deal of success with the library  crfsuite in particular which is written in C but had a python wrapper.


Uploading images folder from my system into Google Colab

Method 1 :

zip the file
Upload the zipped file, there is an Upload button under the Files Section.
Unzip it using the command on colab :
!unzip level_1_test.zip

Method 2 :

upload the zip file to the google drive account.
The only difference is in step 2 where in place of the GUI upload option you can run the google code_snippets to upload download your zip file from the google drive to Colab account .
Unzip it using the command on colab :
!unzip level_1_test.zip


CNN implementation low accuracy on MINST data

Try by adding dropout to the network to avoid overfitting.
read the docs for more information
https://keras.io/layers/core/

and try these things as well

since the targets are integers,its better to use sparse_categorical_crossentropy than categorical_crossentropy and optimizer as Adam

model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizers.Adam(lr=0.001), metrics=['accuracy'])

and try by using sigmoid activation function for output layer

model.add(Dense(10, activation='sigmoid'))


Python Machine Learning/Data Science Project Structure

I don't think there are best practices in this developing area yet but, in addition to cookiecutter, there were some interesting ideas shown in a tutorial at the SciPy 2016 conference.
Personally, I try to minimize the number of sub-folders in a project unless I have a really good way to distinguish them and have good reason to keep them separate. Bad organization is nearly as bad as no organization. I think best practice might depend on use case -- not every project needs the same amount of boilerplate.


I suspect the problem is that your discrete values are non-integer floats.  The classifier's fit method runs type_of_target, which returns (in part):

  'continuous': y is an array-like of floats that are not all
            integers, and is 1d or a column vector.

(source)

(Tracing the source code wasn't straightforward; a traceback of the error from you would have been helpful.  For reference here, RandomForestClassifier inherits from BaseForest, whose fit method calls _validate_y_class_weight which (in the ForestClassifier child class) calls check_classification_targets which calls type_of_target.)


Loaded model predicts well in colab but gives same label and accuracy when downloaded

As far as I am aware you also need to save and load the tokenizer you used. The tokenizer is not fitted/trained and therefore is outputting nothing sensible for the model to predict on.


Generate new sentences based on keywords

This is area of NLG . You can use template based text generation techniques, wherein you have defined structure of output text and fill in required blank areas based on keywords. This technique is used in reports generation. An example is narrative science company.

Other approach can be to use OpenAI GPT . Example is Generate Text using OpenAIGPT2 in Python
 . You may have to tweak the code as per your requirement.

Paraphrasing can be another technique. An example of paraphrasing is - https://github.com/vsuthichai/paraphraser


Question mark on Correlation Matrix with RapidMiner

All your data is non-numeric, so there is no straightforward method to compute a correlation value. It appears that the software does compute a correlation value between 2-valued fields (breast and irradiat).

You will have to work on your data if you want to compute the full correlation matrix:

Make fields with numeric values numeric. For instance, the 6th field seems to be numeric data encoded as strings (make sure those are not numbered categories that cannot be ordered, otherwise the correlation will mean nothing)
Give numeric values to ordered categories. For instance, for the age field, you could certainly use a centered numeric value for each category (45 instead of 40-49, 55 instead of 50-59, etc.), or just a rank.
Ignore rows with no data. For instance, in the 5th field, you have binary results (yes/no) and appear to have a few question marks. Removing the lines with question marks will make the field 2-valued, something that appears to allow the software to compute the correlation.

For the rest of the fields, there is no magic trick, unfortunately. You may want to one-hot-encode them, but interpreting the results will be more complicated. There are also ways to test categorical fields independence, but that's a different measure: see this post.


Evaluation metric for Information retrieval system

I want to know how to come up with ground truth(relevancy label) if it's not available?

There's simply no way to properly evaluate a system if nobody knows what the output is supposed to be. However there are ways to work around a lack of annotated data:

Ask a panel of annotators to grade the quality of the output on a sample. Disadvantage: if a relevant instance is never predicted, the annotators are unlikely to notice it.
Compare the output to a state of the art system. Disadvantage: the evaluated system can only be as good as the reference system, any error by the reference system is considered correct.
Generate artificial data with an automatic method. Disadvantage: the evaluation relies on the quality of the artificial data, so in theory one has to prove that the artificial data is as good as real data... which is usually harder than actually collecting real data.

In that case, how researchers calculate precision and recall? or how do they generate it?

They can't. It would be like grading an exam paper without knowing the correct answers.
This is why benchmark datasets are so important for the research community and are published as proper scientific contributions.


What is the difference between continual learning and active learning?

Active Learning:

The main hypothesis in active learning is that if a learning algorithm can choose the data it wants to learn from, it can perform better than traditional methods with substantially less data for training.

Active learning (AL) augments the learning process with a model that guides and extends the learning process. Using AL, the agent can ‘actively’ gather labels from multiple sources, or curate its labels better (including rejecting/refining incorrect or incomplete labels), or embed supplementary information on causality or complications among the data that require added effort in resolving difficulties within the process of labeling or in applying existing learning to similar problems.

Continual learning

Continual learning is the ability of a model to learn continually from a stream of data. In practice, this means supporting the ability of a model to autonomously learn and adapt in production as new data comes in.


Is regularization included in loss history Keras returns?

the regularisation values are by default computed in the loss and so you cannot see the regularisaiton values separately (as far as I know). For a brief explanation of an example using the cross-entropy loss with regularisation, have a look at the example in Michael Nielsen's book.

You can see what is returned and available by saving the results from calling model.fit().

You can train a model then also check out:

print(history.history.keys())

which will show you all available metrics for analysis after training the model.

You could perhaps define a custom function that compute the regularisation terms you are using, and execute that as either your own metric or as a callback function in a Keras model. If you use your own metric, those are by default recorded separately from the training/validation loss value.


Layer notation for convolutional neural networks

One paper referenced by the first paper you linked to is here. It explains in section 3 (experiments) the following notation:

  2x48x48-100C5-MP2-100C5-MP2-100C4-MP2-300N-100N-6N represents a net
  with:

  2 input images of size 48x48
  a convolutional layer with 100 maps and 5x5 filters
  a max-pooling layer over non-overlapping regions of size 2x2
  a convolutional layer with 100 maps and 4x4 filters
  a max-pooling layer over non overlapping regions of size 2x2
  a fully connected layer with 300 hidden units,
  a fully connected layer with 100 hidden units
  a fully connected layer with 6 neurons (one per class)

From this, the answer to your question is:

100C3 means a convolutional layer with 100 maps and 3x3 filters
MP2 means a max-pooling layer with non overlapping regions of size 2x2
200C2 means a convolutional layer with 200 maps and 2x2 filters
etc with the "C" layers (means convolutional, preceding integer is the number of features maps; final integer is the filter size)

According to the second paper you linked, the subscript _5 indicates five pairs of 300nC2−300nC2−MP2 connected layers (see section 3), and the n indicates "the number of filters in the nth convolutional layer is [300]n".  According to the accompanying model diagram (figure 3 in the linked paper), the C2 and C1 layers produce 1x1 output, meaning a scalar value. This would mean C2 is a convolutional layer with 1 map and a 2x2 filter and C1 is a convolutional layer with 1 map and 1x1 filter (though I don't fully understand what this adds).


Duplicating is RandomOverSampling, not help much in OverSampling.
I quickly did a RandomUnderSampling.The score looks good for a baseline to improve.
I have not done anything for the model improvement

 Code as it is from my Google Colab -

url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip'

from urllib.request import urlretrieve
urlretrieve(url, "data.zip")

from zipfile import ZipFile
file_name = "/content/data.zip"
with ZipFile(file_name, 'r') as zip:
    zip.extractall()

import numpy as np,pandas as pd
data = pd.read_csv("/content/bank-additional/bank-additional-full.csv",delimiter=";")
X = data.iloc[:,:-1]
y = data.iloc[:,-1]
y.value_counts()

X_cat = X.select_dtypes(include='object')
from sklearn.preprocessing import LabelEncoder
lbe = LabelEncoder()
for colname in X_cat.columns:
    X_cat[colname] = lbe.fit_transform(X_cat[colname])
    X[colname] = X_cat[colname]
y = lbe.fit_transform(y)

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X,y,test_size=0.20,random_state=201, stratify=y)

from imblearn.under_sampling import RandomUnderSampler
rand = RandomUnderSampler(sampling_strategy=.6)
x_train, y_train = rand.fit_resample(x_train, y_train)

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=200,max_samples=0.05)
model.fit(x_train, y_train)
from sklearn.metrics import accuracy_score
y_pred_train = model.predict(x_train)

####Metrics on train
from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(y_train, y_pred_train).ravel() # (55148, 1715, 8,90)
print("Training",fp/(tn+fp),fn/(fn+tp), accuracy_score(y_train, y_pred_train), tn, fp, fn, tp)

####Metrics on test
y_pred = model.predict(x_test)
tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel() # (55148, 1715, 8,90)
print("Test",fp/(tn+fp),fn/(fn+tp), accuracy_score(y_test, y_pred), tn, fp, fn, tp)

from sklearn.metrics import recall_score
print("Test-recall",recall_score(y_test, y_pred))

Next -
- Try SMOTE and Combo of Over and Under
- Work on Feature Engg and Dimensionality reduction
- Check other models


why does transform from tfidf vectorizer (sklearn) not work

As far as I can tell it's interpreting your new word_set word_set = ['dog', 'cat', 'foo'] as three separate documents containing one word each, whereas if you did word_set = ['dog cat foo'] it would interpret this as a single new document containing those words.

What behavior are you expecting from this function? Is corpus = words a list of document strings, or a list of single words? If it's the latter, this is likely not doing what you think it is doing, and you should instead make corpus a list of document strings.


How do I use rnn to forecast to n periods with limited data?

there are 3 ways you can do this in NN -

1) take forecasted values ( horizon 1) and pass this again to built model to forecast next values ( horizon 2) and so on.

2) build a model with lagged values and have multiple output values( t, t+1, T+2... t+n). You need to decide window for lagged values.

3) use LSTM.


First steps with Python and scikit-learn

In python 3 the print function must have parenthesis, so print(clf.predict([[150, 0]])) will work


It's an economic calculation, really.  When you have a computing "problem" (in the most general possible sense) that you can't solve with one computer, it makes sense to use a cluster of commodity machines when doing so A. allows you to solve the problem, and B. is cheaper than a forklift upgrade to a bigger computer, or upgrading to specialized hardware.

When those things are true, and you are going the "commodity cluster" route, Hadoop makes a lot of sense, especially if the nature of the problem maps (no pun intended) well to MapReduce.  If it doesn't, one shouldn't be scared to consider "older" cluster approaches like a Beowulf cluster using MPI or OpenMP.

That said, the newer YARN based Hadoop does support a form of MPI, so those worlds are starting to move closer together.


A Neural Network That Learns Bitwise XOR

I think there might be a few things going on.

You might have a reason but I don't know why you have shaped your input data into three dimensions: size=(50000,size,size).

Also, you might have a reason but I don't know why you ran each feature separately through a different layer (each with a single hidden unit), and then merged the outputs before running the merged output through another series of layers:

a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])

Also, I suspect that running the features through a single hidden unit reduces the information sent through the rest of the network, so the network cannot learn the XOR function.

Here is some code that works for me:

from keras import models

from keras.layers import Dense

import numpy as np

Simulate data:

X_1 = np.random.choice([0, 1], size = (50000, 1))
X_2 = np.random.choice([0, 1], size = (50000, 1))

X = np.concatenate((X_1, X_2), axis = 1)

Y = np.bitwise_xor(X[:, 0], X[:, 1])

FNN Model:

# Define model.

network_fnn = models.Sequential()
network_fnn.add(Dense(4, activation = 'relu', input_shape = (X.shape[1],)))
network_fnn.add(Dense(4, activation = 'relu'))
network_fnn.add(Dense(1, activation = 'sigmoid'))

# Compile model.

network_fnn.compile(optimizer = 'rmsprop', loss = 'binary_crossentropy', metrics = ['acc'])

# Fit model.

history_fnn = network_fnn.fit(X, Y, epochs = 5, batch_size = 32, verbose = True)


How to deal with memory insufficient read by pandas in python

I would suggest you to use Dask. I used it successfully when I had to read large data with my 4GB RAM. You can get more details here.

To read a CSV, you can do the following:

import dask.dataframe as dd

csv_file = 'data.csv'
df = dd.read_csv(csv_file)


How to detect medicine name from the medicine wrapper

The Python Library
Python-tesseract is an optical character recognition (OCR) tool for python. That is, it will recognize and “read” the text embedded in images. Python-tesseract is a wrapper for Google's Tesseract-OCR Engin
and please tell me where you find the medicine wrapper ( Packaging ) of different medicines.


Group_by 2 variables and pivot_wider distribution based on 2 others

I cannot get pivot_wider to work here either, but I can replicate your described operation without it.

df %>% group_by(Name) %>%                                      # Group only by Name first
  mutate(Total = sum(Amount)) %>%                              # Total Amount by Name
  group_by(Name, Rating) %>%                                   # Now perform your calculations
  summarize(sum_rating = sum(Amount),
        count = n(),
        wm_Price = weighted.mean(Price, Amount),
        wm_Rate = weighted.mean(Rate, Amount),
        mode_Type = mode(Type),
        POS = sum(Amount * (Direction == "POS")) / max(Total), # This is % POS
        NEU = sum(Amount * (Direction == "NEU")) / max(Total), # This is % NEU
        NEG = sum(Amount * (Direction == "NEG")) / max(Total)  # This is % NEG
        ) %>%
   mutate(pct_rating = sum_rating / sum(sum_rating))

Here is the output

# A tibble: 9 x 11
# Groups:   Name [3]
  Name  Rating sum_rating count wm_Price wm_Rate mode_Type   POS   NEU   NEG
  <fct> <fct>       <int> <int>    <dbl>   <dbl> <fct>     <dbl> <dbl> <dbl>
1 Drew  A            1110     2     93.0    4.97 white     0.153 0     0.220
2 Drew  B            1108     2     42.1    4.58 blue      0.220 0.153 0
3 Drew  C             754     1     54.4    4.3  green     0.254 0     0
4 Jon   A             454     1     65.2    3.4  blue      0     0     0.154
5 Jon   B            1499     2     67.5    4.34 blue      0     0     0.508
6 Jon   C             998     2     65.5    3.95 green     0.154 0.184 0
7 Nick  A            1240     2     59.3    3.4  white     0.428 0     0
8 Nick  B             343     1     54.4    6.4  blue      0.119 0     0
9 Nick  C            1311     2     79.9    3.80 green     0     0.226 0.227
# ... with 1 more variable: pct_rating <dbl>

This method works because r can do math on logical values, treating them as TRUE = 1 and FALSE = 0. Thus, if you want to count the number of occurrences of POS in the Direction column, you could sum a logical vector:

sum(Direction == 'POS')

If you want to know the percent of the rows that contain POS, then use the mean:

mean(Direction == 'POS')

You can even weight that mean by another vector:

weighted.mean(Direction == 'POS', Amount)

The results of my calculation do not match your sample output, so I want to check to make sure I understand what you want the column POS to contain for each row. I understand your desired calculation to be as follows: the Amount of POS for Drew and A (455) divided by the total Amount for Drew (455 + 655 + 454 + 654 + 754 = 2972). Thus, 455 / 2972 = 0.153. I cannot replicate the 0.3735 value you have in your sample output.


How does decision tree for regression makes a prediction?

Depends on the implementation but commonly used is a certain cursive partitioning method called CART.  The algorithm works as following:  It searches for every distinct values for your predictors and chooses to the split based on what minimize the SSE for two groups of dependent variables.  The difference is within the SSE is usually the difference between the actual value and the average of the sample or the difference between the actual value and the output of a linear regression.  For each group, the method will recursively split the predictor values within the groups.

In practice, the method stops when a certain sample size threshold is met.  The node with the lowest SSE becomes the root node.

Reference: You can read more about tree based regressions here: Tree-based regressions


Interpreting the new dimensions after PCA

To give meaning to the Axis of the PCA, you can study the scalar product between the two new axes and all your original axes (you have to normalise the vectors before doing that).

Those axes with a high scalar product will be highly associated the with the new axes and viceversa. This may help you interpret them.


Using LSTM for binary text Classification, getting almost same accuracy at each epoch

First of all, this is a binary classification problem (positive sentiment / negative sentiment), correct? And the dataset is roughly balanced? What were you trying to achieve by sorting them by length? Why are you using a generator and not just running fit on the dataset?

Don't worry about local minima. What you're looking at is a model/optimizer combination that converges to around 74% on this dataset. The one being fed sorted data has a higher accuracy at the start of the epoch because short tweets are easier to classify than long ones. And I wouldn't even worry about the batch-by-batch accuracy, just the epoch-by-epoch curve.


Calculate average Intersection over Union

In my experience, the two approaches can give quite different results. It doesn't really show in the example you provided because the sizes are similar. However, in some cases in object detection, you can have the same object appear with very different sizes in two images. Your first approach will weigh the IoUs equally but the second will give more weight to the larger object. The scripts I find online to calculate the average IoU seem to do the second approach but in my opinion, each instance of a class should be weighed the same regardless of its size.
Edit:
The first approach is: $\frac{1}{n}\sum\frac{i_i}{u_i}$
The second is: $\frac{\sum i_i}{\sum u_i}$.
Let's assume that we are detecting two objects the first object is very small and is well detected. It has an intersection of 20 pixels and a union of 21 pixels. The second object is very large and is poorly detected. It has an intersection of 20 pixels and union of 200. The first method will give an average IoU of 0.52 the whereas the second will give an IoU of 0.18. The first approach treats every object detection individually and averages the IoUs effectively giving each object detection the same weight. The second approach is biased toward the larger object.


Confused with the derivation of the gradient descent update rule

Upon writing this I have realised the answer to the question. I am still going to post so that anyone else who wants to learn where the update rule comes from can do so. I have come to this by studying the equation carefully. $∇C$ is the gradient vector of the cost function. The definition of the gradient vector is a collection of partial derivatives that point in the direction of steepest ascent. Since we are performing gradient 'descent', we take the negative of this, as we hope to descend towards the minimum point. The issue for me was how this relates to the weights. It does so because we want to 'take'/'travel' along this vector towards the minimum, so we add this onto the weights. Finally, we use neta which is a small constant. It is small so that the inequality $∆C>0$ is obeyed, because we want to always decrease the cost, not increase it. However, too small, and the algorithm will take a long time to converge. This means the value for eta must be experimented with.


It's very likely that the authors assume that the spam class is positive, whereas you intuitively associated the ham class with positive. Both options make sense in my opinion:

the former interpretation is based on the idea that the goal of the task is to detect the spam emails, seen as the class of interest.
the later interpretation considers that the ham emails are the "good ones", the ones that we want, hence the "positive" class.

There's no error when one reads the paragraph with the authors' interpretation in mind. This confusion illustrates why one should always clearly define which class is defined as positive in a binary classification problem :)


How to deal with unbalanced class in biological datasets?

There can be two things for an imbalanced dataset:

. For eg, in a cancer detection dataset, the number of positive(cancer present) samples, will be quite less than the negative( cancer absent) samples. In this case, you should not apply sampling technqiues. You should rather focus on minimizing False Negatives , as cancer should not be missed while diagnosing a patient at the expense of a few false positives.

.Second case, for eg, The popular  Iris Datset where the classes should be balanced. But suppose you get an imbalanced version of the dataset, then you should apply sampling technqiues to the dataset. Because, the class proportion does not match the actual proportion , which was not the case in the Cancer Example.


How to convert a mel spectrogram to log-scaled mel spectrogram

I think you're wrongly interpreting what the authors meant by log-scaled. When the authors mention log-scaled, they are not referring to the frequency (y) axis, although spectrograms are typically log-scaled here. They are instead referring to the scale of the 3rd dimension in the spectrogram. In your case, the raw spectrogram is displaying power in color. What you want is instead decibels, which are log-scaled.

In your case, the code would look like this:

y, sr = librosa.load('audio/100263-2-0-117.wav',duration=3)
ps = librosa.feature.melspectrogram(y=y, sr=sr)
ps_db= librosa.power_to_db(ps, ref=np.max)

lr.display.specshow(ps_db, x_axis='time', y_axis='mel')

Note: Each spectrogram will be scaled based off of the ref within librosa.power_to_db. If you do not supply anything, librosa just shoves a 1 in there, which may or may not be what you're looking for. You can also try out np.median.


Can we add other features along with Text in sentiment analysis

Yes you definitely can.
Here's an example: Using Convolutional Neural Networks to Classify Hate-Speech

The authors used classic embeddings concatenated with a vector of size 28 representing the presence or not (in a tweet) of each letter of the alphabet (26) plus any digit (1) plus any other symbol (1).
So basically for a tweet like 'I love NLP!' the representation would be:

(embeddings)(00001000100110110000000)(0)(1)

where you have the classic embeddings for each word, then a one hot vector for the alphabet letters, another hot vector (just a binary variable actually) for digits and one last that check for the presence of any other symbol (! in this case). The resulting concatenated vector is then passed to a LSTM, convolution layer or whatever architecture you want to use.

Consider that there are infinite other variables that you can concatenate to the embeddings. For example you could concatenate counters for each alphabet letter or any other symbol in a sentence (in this case I suggest to separately normalise these variables between 0 and 1 before concatenating them to the embeddings), one-hot representation of n-grams or dependencies, the total number of words, presence or not and counters of capital letters, presence of entities, sentiment scores obtained through external resources like Vader (see nltk) and so on.


Given data that is labeled as outliers, how can I classify data as outliers?

If you are sure that your data are actually normally distributed and that your outliers actually form a high-dimensional ring around your "good" data, you simply require a distance-metric, e.g. Mahalanobis distance is suitable for normal data, and determine the threshold value for distance from the mean to consider all points beyond the high-dimensional ball or ellipsoid depending on the variance structure outliers according to your description.


Yes, this is feasible.
One-class classification is a thing, but it is usually used in a context where it is hard or impossible to get negative samples. In your case, I would argue, you can quite easily get tweets that are not about activism, therefore you can render it as a binary classification, because you have data points of two classes or labels: 1 for tweets that are part of your class and another 1 for tweets that are not.
There are many ways to build a classifier, SVM is only one of them. You could also use a Naive Bayes algorithm, or as @Kasra mentioned a neural network model.
No matter what you use, you will have to organise your data such that you have samples of both classes: activism and non-activism within your set. This means that you should randomly pick tweets from your big dataset and manually check if they relate to activism, even if they don't have the hashtags in them that you used for identifying the activism tweets in the beginning.
Further, you have to think about the features that your classifier will use. The simplest might be the bag of words within the tweets, but you might also pre-process the tweet to exclude stop-words.
Depending on which algorithm you use, you might find that your classifier relies a lot on the presence of your particular hashtags as features for predicting the class. In this case it might struggle to identify other tweets without this hashtags as activism, even if they are activism. I would experiment with pre-processing the tweets in your entire dataset to remove those hashtags from the the tweets.


Difference between rand index and adjusted rand index?

The adjustment is simply

(Rand index - Expected value)/(Optimal value - Expected value)

The purpose is to scale it in an interpretable way. 0 is "as good as random", less than 0 is worse, and close to 1 is good.

The problem with the non adjusted Rand index is that a random result on certain data sets can achieve a high score otherwise.


Pearson correlation coefficient - is correlation estimator acceptable?

If the discrete variable has a lot of discrete values then it is almost the same as a continuous variable, because continuous variables are technically discrete due to the way how numbers are represented in computers (float64 for Python).

The worst case is binary, but, in my experience, Pearson coefficient work well with binary and continuous data together. I know that asymptotic distributions could lead to biased estimators of linear regression. But Pearson coefficient is the way to calculate, it is not an estimation of something, so I cannot say if it is biased or non-biased. I know that if the relationship is linear then you get a strong Pearson coefficient, and if not then you get a small number.

For continuous data and binary data it means that you need to have a strong threshold. If everything below the threshold means 0 and everything above means 1 for the binary variable, then the correlation is strong. Though if I remember correctly it will still not be 1, because you cannot explain 100% of variance of a continuous variable with one binary variable. But you probably can get a number about 80%. This all depends on where the threshold is, and how the variables are distributed. If you have two distinct clouds of continuous variable that are separated by a big distance, then you will get a number close to 100%. Similar logic could be used for discrete variables that have several values.

Pearson coefficient gives a stronger weight to where the most points are, due to the way it is calculated. So if you do not have points in particular regions then there is no way for the coefficient to reflect the correlation there. In practice it means that you usually have points in a limited range, and you calculate the linear correlation in this range.


Identifying top predictors from a mix of categorical and ordinal data

Ricky,

Loose thoughts:

Depending on the algorithm you intend to use, centering might not be a good idea (e.g. if you go for SVM, centering will destroy sparsity)
I would suggest not to handle ordered / unordered separately, as you are likely to miss interactions that way. If the categorical ones don't have too many possible values, randomForest in R can handle factors.
if that is an issue (as you seem to hint), I think you have two possibilities: binary indicators or response rates
if it's feasible in terms of computational cost, i would convert all factors to binaries (use sparse matrices if necessary) and then try a greedy feature selection. caret, if memory serves, has rfe or somesuch.
if that's too much trouble, try calculating response rates / average values per factor level (I don't see any info whether your problem is classification or regression): you split your set into folds, and then for each fold fit a mixed effects model (e.g. via lme4) on the remainder, using the factor of interest as the main variable. It's a bit of a pain to setup all the cv correctly, but it's the only way to avoid leaking information.

Hope this helps,
K


cosine similarity between items (purchase data) and normalisation

My question is, do I need to normalize each product's vector before
  using columnSimilarities()?

No, you do not need to normalize each product's vector before using columnSimilarities() since it is performed within the operation already. I think your confusion comes from the fact that your considering dot product and cosine similarity to be the same. They are not.

Dot Product is only a component of the Cosine Similarity Function Denoted as "A (dot) B"

With regards to the answer you referenced, the solution suggests using the dot product as an alternative to cosine similarity because a dot product calculation is not affected by the magnitude. The dot product doesn't use the mean as part of its calculation.

  when it's good to use normalization before calculating cosine
  similarity and when it's not?

I would advise against using normalization before cosine similarity. However, there are other methods you can consider if cosine similarity is not returning desired results. You can use adjusted cosine similarity or dot product(as referenced in the answer you linked). Both of these measures take into account differences in magnitude. The adjusted cosine similarity subtracts the mean before calculating cosine similarity. Dot product doesn't use the mean in its calculation.  Which is important in your context. For example, Users who always rate products 5 stars and users who always rate products 1 star are not equivalent.

  what are the different ways to normalize?

There are a lot of ways to normalize a vector. You can use Z-score standardization, the standard min-max scaling, l1 or l2-normalization,etc etc

Further Information: Is feature normalization needed prior to computing Cosine distance


why transpose convolution is called "transpose"

Transpose in mathematics means to change the order of matrix in an opposite way, the same notion carries here but not the exact sense, you are talking about.
The same problem exists with the word 'convolution', it means something else in mathematics. What is done in deep learning in name of convolution is cross-correlation in mathematics.


LSTM fot text classification always returns the same results

You are not using the same Tokenizer in testing which you used in training, so texts_to_sequences is not outputting the required result.


df = df[~df['country_code'].isin(country_list)]

https://pandas.pydata.org/pandas-docs/stable/gotchas.html#using-the-in-operator


Discrepancy between training set and real-world data set: domain adaptation?

You may be looking for sampling bias. Also the other case (where training set does in fact represent the real world data-set well) is generally known as representative sample.

Hope this helps.


I believe you are correct, that the paper and Wikipedia disagree. The paper's formula suggests you apply no discount at $r <= b$, which means both of the first two elements are not discounted.

The Wikipedia formula discounts the second element onward.

There's an impassioned statement in Talk about why the Wikipedia formula is right: https://en.wikipedia.org/wiki/Talk:Discounted_cumulative_gain

But I can't see why; it offers no reference other than observing that "it seems plainly wrong to not discount from the second element." I'll comment there.


Dealing with feature vectors of variable length

First of all welcome to the community!

About the question I would say there are misunderstandings. You say you calculate 4 features "per objects". It means that every data point (object) is described with 4 features. So the length of features are not different. Please note that I say this because "you calculate features for your samples/data points/objects" so every object here is a point in 4-dimensional space. If you mean something else please correct me.

In other words, you have a data matrix with some rows (number of objects) and 4 columns.

More general about the whole concept. No, this is not a valid approach to create standard data. In standard data the number of columns are always the same because you can not study objects if they are defined in different spaces. Many ML algorithms also work with standard data which means you can not use them if the size of feature vectors vary. More conceptual so to speak, the size of features you calculate for objects can not vary. How? I want to extract features from faces. I say eye color, distance between eyes and distance between ears. I have 3 features and for any person who comes to my study I calculate the same set of features. How can the size be different if the feature extraction process is standard?

But in non-standard data you may end up with such a case. In this case you may standardize the features. E.g. as a baby example, imagine different graphs with different number of nodes and edges. You can describe each graph with its average degree, average path length and number of nodes. Then each graph is described with 3 features and you can feed it into an algorithm.

But if you don't want to standardize the data (because it usually has loss of information) you need to find tailored analytic methods for which you need to explain more about your data.

I hope it helped!

Good Luck!


Issue with using Sparse Data Frame in Mlxtend Apriori function

This question is now closed as it has been accepted as an enhancement needed by the developer.

https://github.com/rasbt/mlxtend/issues/501


How to measure accuracy of a route prediction

You need to define what a "quality route" is in the context of your problem and score based on that. This will likely be something along the lines of the total distance of the route, the expected time to traverse the route, the expected fuel/resource consumption required to complete the route, or some combination of these. You presumably want to predict routes that are as good or better than the routes given in your set R: you need to define what it means for a route to be "better."


How to build a model where multiple data points contribute to a result

My intuition is if I arranged the data as individual interactions, with output of final score, that’ll give me what I want. Is that true?

If I understand correctly, your goal is to predict the final score that a customer gives at the end of their dinner based on all the interactions they had during the dinner, right? If yes then I don't think you can keep individual interactions as instances, because then (1) the model would only be able to use one interaction to predict the score and (2) the model would predict a different score for each interaction.

So each instance should represent the full meal and somehow contain as features all the possible information extracted from the interactions. For example an instance could contain the number of interactions, total length, number of refills, etc.

(side note: I wouldn't like eating in a restaurant where customers and staff are monitored so closely, but maybe that's just me)


How to download a Jupyter Notebook from GitHub?

You can do :

Open the jupyter notebook you want to run.
Click on: Raw

Save Ctrl+S
Remove:  .txt

Now navigate to the directory where notebook is downloaded in jupyter notebook and open it.


Depends on the context it may mean differently. One common usage of the term is when not all the data points have valid/reliable values for a specific feature. For example for a feature set of $x=\{x_1, x_2, \dots, x_p, \dots, x_n\}$ not all data points may have specific feature $p$.

It is possible that some data points lack a value for feature $p$ either because of data corruption, noise or etc. In this case the percentage of data point which actually have a reliable value for this feature is called the feature coverage of feature $p$ for this specific data set.


Comparing multi-class vs. binary classifiers in predicting a single class

In general we can't compare the performance of a multiclass classifier with the performance of a binary classifier since the former expresses how good the classifier is at classifying any instance of any class. So if there are $n_A$ samples labelled A, there's only a proportion of $n_A/n$ of the global accuracy of the multiclass classifier which is about A. In particular a multiclass classifier usually tends to favor the largest classes, so if class A happens to be a small proportion of the data then the global performance will not reflect how good it is at classifying A: for example it might have 90% accuracy simply because class B is 90% of the data, this doesn't prove anything about class A. By contrast the performance of the binary classifier is by definition solely about class A.
However if one has access to the detailed evaluation of the multiclass classifier, typically the confusion matrix, then it becomes possible to calculate the performance of the classifier for a single class, say class A. Actually by merging all the B,C,D rows together and all the B,C,D columns together in the confusion matrix one obtains exactly a binary classification confusion matrix, and from that one can calculate a performance which can be compared against another binary classifier. But in this setting the multiclass classifier is at a disadvantage for the reason mentioned above: it also has to deal with the other classes and this can cause it to "sacrifice" a class, whereas the binary classifier doesn't have this issue.


Replace all numeric values in a pyspark dataframe by a constant value

As per your problem, I think it might be easier to use lit. Try this-

from pyspark.sql.functions import lit
new_df = df.withColumn('column_name', lit(1))

Hope it helps!


what is BIO Tags for creating custom NER Named entity recognization?

It is easy. You need to tag a phrase using B (Begin), I (Interior), and E (End). For example, you want to tag "United States of America" as the name of a country. You will tag likes:

United(B_Country) States(I_Country) of(I_Country) America(E_Country)

In the same text if you find "Islamic Republic of Iran", you will tag likes:

Islamic(B_Country) Republic(I_Country) of(I_Country) Iran(E_Country)

Also, you will tag "United Kingdom" likes:

United(B_Country) Kingdom(E_Country)

Therefore, for every label that you have in the label set, you will have three labels in the tagging that is B_LabelName, I_LabelName, and E_LabelName.

Notice that In some tagging systems, also uses from I_LabelName instead of E_LabelName.

Now, what is "O"? In some cases the all part of the phrase is not in the label but comming in the middle of the phrase and we need to say it is out ("O") of the label. For example, in the text analysis we want to take out "of" from country names. Hence we will tag "United States of America" like the following:

United(B_Country) States(I_Country) of(O_Country) America(E_Country/I_Country)


a data point - too little to spend resources during data preparation?

Practically everything related to statistics (including Machine Learning) has to do with studying chance, i.e. trying to determine to what extent an observation is due to chance or not.

For example one might want to know whether a drug actually helps with a particular disease or not. If we observe that one patient improves after taking the drug, there's not enough evidence to conclude since many other factors could have caused the improvement. This is why one needs a very strict protocol in order to obtain a statistically meaningful observation (two groups of patients, placebo etc.).
A reasonably high number of observations is needed, otherwise it's impossible to distinguish the effect of "chance" (any other factor) and the real effect of the drug.

  a) Is there any systematic/mathematical/theoretical approach that can tell me anything less than N% is too little to impact/influence the output?

The standard method for knowing whether an observation is due to chance or not is to use the appropriate statistical significance test. There are many of them and they depend on what exactly is being tested.

  b) How do you decide which items are too little to impact output. Do you go with your judgement which is a subjective approach?

In ML it's common to take a more experimental approach, for example trying with/without an observation or feature and then evaluate which versions works better. Of course it helps to have an intuition of what is more likely to work. In general including extremely rare observations is a bad idea because it's likely to cause overfit, i.e. when the model "learns" something which is actually due to chance.

[added following OP's update]

In this case this is a resource allocation problem, I don't think  statistical significance is relevant here. Assuming that you want to optimize the use of manual labor based on how often a drug is used, i.e. the only thing to maximize is the sum of the frequencies of the drugs being labelled, then it's simple: rank all the drugs by their frequency in descending order, then proceed with manual annotation following this order. This way you're sure that the drugs which account for more patients are done first, so whenever manual annotation stops the largest possible amount has been labelled.


Does training of neural networks follow the same order in each epoch?

All the parameters are updated after a batch, there is no notion of order of update.
The batch can or cannot be in the same order. It could lead to overfitting in the sense of the network learn the order of the dataset. An easy turnaround is to shuffle the batch between each epochs.


Are CNNs best approaches to find and count objects in images?

For your situation, I would recommend using an object detection network to both locate and classify your components.  Although you do not need the location or classification data, you can still count the number of objects the network detects.  Moreover, this approach does not require any data science background because there are many tutorials on this subject and tons of out-of-the-box tools, such as tensorflow's object detection API.

You will need to annotate your data, but there are tons of programs to help you with that too.  Here's one I found quickly.

I suggest using faster-RCNN with the NAS feature extractor because, although it is slower than other CNN architectures, it is the most accurate.  Also, the network would probably perform better if you did not combine all the components into a single class.

The steps necessary to accomplish this task are essentially the following:

Install and setup tensorflow
Download faster RCNN NAS from the tensorflow detection model zoo
Annotate your data by drawing bounding boxes around the components and labeling each bounding box.
Train your model
After saving the trained model, you just need to setup a pipeline that takes an input image, feeds it to your network, and returns the total number of detections ignoring the bounding box and classification data.  Here's a great tutorial for deploying a trained model.


How Orange makes decision about declaration of a meta attribute

As far as I know, Orange supports advanced annotation with a three-line header in both CSV and Excel files. So if you have a discrete attribute that gets marked as string, you should annote it as discrete on the second header line, or prefix the attribute name on the first line with D#.

When no advanced annotation is provided, it seems Orange's heuristic treats attributes containing general strings (as opposed to numbers) as discrete only if they contain fewer than approximately $\sqrt{|data|}$ unique values.


How to write a LSTM model with 3 dimensional X_train and Y_trains?

In all the above cases, you are not setting the output shape properly. The last layer should create 3 features.
Try this for example:

model = Sequential()
model.add(LSTM(24, return_sequences = True))
model.add(Dropout(dp))
model.add(Dense(3))


Detecting address labels using Tensorflow Object Detection API

It is difficult to say whether the algorithm will detect full box of text or not. This is kind of difficult problem because object to detect does not have proper structure. I might be wrong here!

You are labeling correctly.

You have very few images to learn from. More images are always better for learning tasks.

Don't look at loss value which appears in the terminal, instated check total loss in TensorBoard (I am sure this loss will be more).
For how long to train? in the example config file from TensorFlow API they say on pets dataset (which has only 7349 images) you need to train model for 200000 steps to get satisfactory results. If this problem is learnable, training for more steps should work.


Dimension Reduction - After or Before Train-Test Split

It depends on the type of algortithm you use for dimensionality reductions. In case you use PCA, you should build your PCA on your train set. Then you need to set your principal components to transform your points in test set into the same space. This way you can then use train and test set in the same reducted space.


L2 regularization increase the loss rate of the deep learning model

Suppose a neural network with a regular loss function.

$$
\sum_{i=1}^N L \left( y_i, \; \hat y_i \right)
$$

Here, $y_i$ is label for the $i$-th example, while $\hat y_i$ is the model's prediction for the same. The loss function $L$ is a function that compares the actual and the predicted output and outputs a value indicating how close the prediction was to the actual output.

L2-regularization adds a norm penalty this loss function and as a result to each weight update.

$$
\sum_{i=1}^N L\left( y_i, \; \hat y_i \right) + λ \cdot \|W\|_2^2
$$

This penalty counters the actual update, meaning that it makes the weight updates harder. This has the effect of actually increasing the output your loss function.

What you should be looking for, by adding regularization to a model, isn't a reduction in training loss but a reduction in the validation loss. This would indicate that the regularization is successful in reducing your model's overfitting, which was its goal.


Old question, but in case anyone is still interested in an answer...
In the perceptron algorithm a point $x$ has a label $y$ equal to 1 or -1. The predicted label for a point is $w\cdot x$. The goal is to separate the points with an hyperplane orthogonal to w in such a way that the points with label 1 are on one side and the points with label -1 are on the other side. Mathematically, the "sides" of the hyperplane are characterised by the equations $w\cdot x>0$ and $w\cdot x<0$, respectively. ($w\cdot x=0$ is the equation of the hyperplane itself).
Therefore $x$ point is correctly labelled (it is on the right side of the hyperplane orthogonal to $w$) if

$y=1$ and $w\cdot x>0$, or
$y=-1$ and $w\cdot x<0$

In both cases, a correctly labelled point corresponds to $yw\cdot x > 0$, while an incorrectly labelled points corresponds to $yw\cdot x \leq 0$.
The whole algorithm and the proof of its convergence are based on this simple observation.
If we change the condition for update to $yw\cdot x < 1$, the algorithm would not be able to find a separating hyperplane. For example, consider the simple dataset $x_1 = (2,0), y_1= 1$ , $x_2 = (-2,1), y_2= -1$.

Start: $w=(0, 0)$.
Step 1: $y_1w\cdot x_1 = 0 < 1$ therefore $w \leftarrow w + y_1x_1 = (2, 0)$
Step 2: $y_2w\cdot x_2 = 4 > 1$, do not update $w$.
Step 3: $y_1w\cdot x_1 = 4 > 1$, do not update $w$. We went over all points without updating $w$, therefore exit the loop.
Result: the hyperplane orthogonal to $w=(2, 0)$ is the y-axis, which clearly does not separate $x_1$ from $x_2$.

TLDR;  Update conditions different from $yw\cdot x \leq 0$ do not offer any guarantees for finding a separating hyperplane.


Difference between AWS amazonei_tensorflow and tensorflow conda environments

The amazonei environments are for use with the Amazon Elastic Inference accelerators. You can see this mentioned in the DLAMI README file:

for TensorFlow(+Keras2) with Python3 (CUDA 9.0 and Intel MKL-DNN) ______________
____ source activate tensorflow_p36

for Tensorflow(+Amazon Elastic Inference) with Python3 _________________________
____ source activate amazonei_tensorflow_p36


Is there any package in python similar to darch in R?

A simple google with keyword    gives me:

A tutorial about implement DBN from scratch
http://deeplearning.net/tutorial/DBN.html
A python package about the DBN
https://github.com/albertbup/deep-belief-network
Some interesting discussion about the DBN (and little history)
https://github.com/keras-team/keras/issues/748
SO discussion
https://stackoverflow.com/questions/23188131/deep-belief-network-with-scikit


How should one deal with null values in time stamp data types?

Leave those time stamps null in order to indicate that you do not possess the time of those events. Everything else depends on the logic of your application.


Why the invariant reward helps training?

Well, some inefficient agents will need more steps to reach the goal. Others will have a more target-oriented, efficient way to reach the goal. The efficient agents will need less steps and have a better/bigger score.


Well, I started studying in Datacamp and I really recommend this Machine learning course it's very complete for new learners.

Udacity has this Deep-learning course and it's a complement for the Coursera Machine learning course.

It's very important to research a lot in other sources but there you can paint a path for your Data Science skills through these ones.

I'm working and studying in Machine learning and I store the test projects in this repository, but is very important to have a little bit of knowledge of python and octave

Already exist other languages and tools like R for test your data sets


Minimum number of samples to train XGBoost without overfitting

It's completely true that the number of examples should be related to features. But it is not only the number of features because the range of a number(max-min and count of different number) is also important. On the other hand, if you have noise you need more examples, so it's related to your dataset.


Remove all columns where the entire column is null

[Updated]: Just realized it is about pyspark!
It is still simple! A concrete example (idea heavily borrowed from this answer):
Creating a dummy dataset
import pandas as pd
import numpy as np
import pyspark.sql.functions as sqlf

main= pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))

main["E"]= np.NAN
main["F"]= np.NAN

df = sqlContext.createDataFrame(main)

Function to drop Null columns
def drop_null_columns(df):

    """
    This function drops columns containing all null values.
    :param df: A PySpark DataFrame
    """

    null_counts = df.select([sqlf.count(sqlf.when(sqlf.col(c).isNull(), c)).alias(c) for c in df.columns]).collect()[0].asDict()
    to_drop = [k for k, v in null_counts.items() if v >= df.count()]
    df = df.drop(*to_drop)

    return df

Outcome
df_dropped = drop_null_columns(df)


Does Gradient Boosting perform n-ary splits where n > 2?

Gradient boosting can be applied to any base model, so doing it with a Quinlan-family decision tree (which allow for such higher-arity splits for categorical features) should make this possible.  However, all implementations of gradient boosted trees that I know of (and certainly XGBoost, CatBoost, LightGBM) all use CART as their tree model, so you won't get anything but binary trees.  (These GBMs do modify CART a little, e.g. in using histogram binning to reduce the split searches, but nothing as drastic as n-ary splits for categoricals.)


How to read file from user in Shiny and assign it to a variable in global.r?

I have added the following code to global.R file

data <- reactiveValues()

I used assign function in server.R to assign values to data in global.r

output$contents <- renderText({
      if(is.null(input$file1))return()
      inFile <- input$file1
      data2<-read.csv(inFile$datapath)
  assign('data',data2,envir=.GlobalEnv)
  print(summary(data))
 })


Why does keras.Model subclass behave differently to the sequential API?

I create two new objects in call(): layers.Flatten and layers.Dense. When I create the objects in the constructor, the model works as expected.
class CriticModel(Model):

    def __init__(self, channel):
        super(CriticModel, self).__init__()
        self.cnn1 = CNNBlock(channel[0])
        self.cnn2 = CNNBlock(channel[1])
        self.flat = layers.Flatten()
        self.dense = layers.Dense(1)

    def call(self, input_tensor, training=False, mask=None):
        x = self.cnn1(input_tensor, training=training)
        x = self.cnn2(x, training=training)
        # x = layers.Flatten()(x) <-- This creates a new obj at eacht forward call
        # x = layers.Dense(1)(x)  <-- This creates a new obj at eacht forward call
        x = self.flat(x)
        x = self.dense(x)
        return x

    def get_config(self):
        super(CriticModel, self).get_config()
```


pytorch convolution with 0-stride along one dimension

Setting stride to 0 is not necessary, torch will simply compute with respect to the input tensor sizes, so you can set stride to (1,1).
For x of size (batch_size, 3, max_dim_0, max_dim_0) (square image) the tensor output will be of size (batch_size, 32, 1, max_dim_0).


Why to exclude features used for label generation during modeling?

What you are doing is right. You can build a ML model using this. In your case your input and output are correlated. Consider the salary of an employee wrt his experience. These both are related and sometimes used to derive salary based on experience.

What you might have read is if one feature is used to derive other feature and dont use both features, as both will yield same results. You can point to that writing for further clarification.

Additionally, what is the significance of variables T1,T2 and T3? If these are nowhere correlated to the output label then actually you would have to eliminate them.