Linearly increasing data with manual reset

I thought this was an interesting problem, so I wrote a sample data set and a linear slope estimator in R.  I hope it helps you with your problem.  I'm going to make some assumptions, the biggest is that you want to estimate a constant slope, given by some segments in your data.  Another assumption to separate the blocks of linear data is that the natural 'reset' will be found by comparing consecutive differences and finding ones that are X-standard deviations below the mean. (I chose 4 sd's, but this can be changed)

Here is a plot of the data, and the code to generating it is at the bottom.

For starters, we find the breaks and fit each set of y-values and record the slopes.

# Find the differences between adjacent points
diffs = y_data[-1] - y_data[-length(y_data)]
# Find the break points (here I use 4 s.d.'s)
break_points = c(0,which(diffs < (mean(diffs) - 4*sd(diffs))),length(y_data))
# Create the lists of y-values
y_lists = sapply(1:(length(break_points)-1),function(x){
  y_data[(break_points[x]+1):(break_points[x+1])]
})
# Create the lists of x-values
x_lists = lapply(y_lists,function(x) 1:length(x))
#Find all the slopes for the lists of points
slopes = unlist(lapply(1:length(y_lists), function(x) lm(y_lists[[x]] ~ x_lists[[x]])$coefficients[2]))

Here are the slopes:
(3.309110, 4.419178, 3.292029, 4.531126, 3.675178, 4.294389)

And we can just take the mean to find the expected slope (3.920168).

Edit: Predicting when series reaches 120

I realized I didn't finish predicted when series reaches 120.  If we estimate the slope to be m and we see a reset at time t to a value x (x<120), we can predict how much longer it would take to reach 120 by some simple algebra.

Here, t is the time it would take to reach 120 after a reset, x is what it resets to, and m is the estimated slope.  I'm not going to even touch the subject of units here, but it's good practice to work them out and make sure everything makes sense.

Edit: Creating The Sample Data

The sample data will consist of 100 points, random noise with a slope of 4 (Hopefully we will estimate this).  When the y-values reach a cutoff, they reset to 50.  The cutoff is randomly chosen between 115 and 120 for each reset.  Here is the R code to create the data set.

# Create Sample Data
set.seed(1001)
x_data = 1:100 # x-data
y_data = rep(0,length(x_data)) # Initialize y-data
y_data[1] = 50
reset_level = sample(115:120,1) # Select initial cutoff
for (i in x_data[-1]){ # Loop through rest of x-data
  if(y_data[i-1]>reset_level){ # check if y-value is above cutoff
    y_data[i] = 50             # Reset if it is and
    reset_level = sample(115:120,1) # rechoose cutoff
  }else {
    y_data[i] = y_data[i-1] + 4 + (10*runif(1)-5) # Or just increment y with random noise
  }
}
plot(x_data,y_data) # Plot data


Tools to perform SQL analytics on 350TB of csv data

There are many tools that are able to support such queries (as you mentioned Hive or Spark), and it is really up to your requirements in terms of number of queries, number of people who are going to query the data, what kind of BI or reporting tools you might want to use with it, etc.

More than that, your requirements are probably going to change in the future. If you provide easier ways to query the data, the more people around you are going to use it.

For example, if you choose to load that data into Amazon Redshift (https://aws.amazon.com/redshift/), it will give you a full SQL flexibility, very fast performance and the ability to connect to several BI, visualization and reporting tools. When you COPY the data into Redshift it is being compressed (column based encoding) 5-8 times, and you can put all your data in a cluster of 6 nodes of DS2.8XL of Redshift, for a couple of weeks for your POC.

The concept of compression can be applied in different solutions as well. For example, instead of using CSV format, you can convert it to Parquet (https://parquet.apache.org/) format, and save on your storage and increase the performance as you need to move around and scan less bytes of data. Once you have your data in Parquet, you can query it for example with Presto, like Netflix are doing - http://techblog.netflix.com/2014/10/using-presto-in-our-big-data-platform.html

Regarding Spark, it can help even if you don't fit all the data into memory. Spark can also help you with the usage of Zeppelin (https://zeppelin.incubator.apache.org/) as an interactive notebook, with nice visualization features, as well as write code with Scala, Python or even R with SparkR.


Differences and similarities between nonnegative PCA and nonnegative matrix factorization

I came across this paper by Asteris et. al. entitled "Orthogonal NMF through Subspace Exploration". You can google it as links posted here might break in the future.

The basic idea boils down to if we can construct non-negative principal components of the matrix of interest, we can use these principal components to do an orthogonal NMF. Since the matrix of interest is non-negative (this is what NMF assumes) and the non-negative principal components are non-negative, then by construction the transformed matrix is also non-negative. The paper details how exactly the construction can be achieved and guarantees convergence.

To answer my own question above: There are many ways to do NMF. In the case where the transforming vectors are orthogonal, i.e. Orthogonal NMF, it is identical to Non-negative PCA. Details can be found in the paper cited above.


One option is Kullback–Leibler divergence which is a measure of how one probability distribution is different from a second probability distribution.

Kullback–Leibler divergence compares the entire distribution (not just peaks) and does not make any assumptions of the functional form of the distribution (which is useful since your data is not very Gaussian).

In fact, Kullback–Leibler divergence can compare discrete distributions so you do not need to fit a Kernel Density Estimation (KDE) curve.


LDA topic model has 0-weight topics, is that normal?

It's normal: LDA tries to maximize the likelihood of the data according to the parameters by finding the right probabilities for the parameters. Usually at the beginning increasing the number of topics allows the model to separate topics more precisely and therefore obtain a higher likelihood. But at some point (depending on the data), increasing the number of topics cannot help the model anymore because the topics are already separated to the maximum and using all the topics would actually decrease the likelihood.
So it's a sign that you don't need that many topics. Note that it doesn't mean that the number of "used" topics is optimal for the application, it's often a balance to find.


Best or recommended R package for logit and probit regression

Unless you have some very specific or exotic requirements, in order to perform logistic (logit and probit) regression analysis in R, you can use standard (built-in and loaded by default) stats package. In particular, you can use glm() function, as shown in the following nice tutorials from UCLA: logit in R tutorial and probit in R tutorial.

If you are interested in multinomial logistic regression, this UCLA tutorial might be helpful (you can use glm() or packages, such as glmnet or mlogit). For the above-mentioned very specific or exotic requirements, many other R packages are available, for example logistf (http://cran.r-project.org/web/packages/logistf) or elrm (http://cran.r-project.org/web/packages/elrm).

I also recommend another nice tutorial on GLMs from Princeton University (by Germán Rodríguez), which discusses some modeling aspects, not addressed in the UCLA materials, in particular updating models and model selection.


I would suggest instead of trying to get many sources, get one source that goes through concepts first (fairly robustly), then seek out sources to refine or deepen your knowledge. One source comes from Stanford's NLP group, and is Introduction to Information Retrieval.  The only thing I don't like about this books is that it tends to orient documents as columns (where data science has more or less agreed that they are rows), but that's a pretty trivial concern (as long as you can take a matrix transpose).  Aside from that, this book has excellent explanations, and the proper depth and breadth to be considered an exhaustive base for NLP.


is gradient descent also used during feed forward propagation in Neural Network?

The feed-forward pass computes the outputs of each layer, and hence the output of the network, given a current state, i.e. the values for all the parameters for this pass or iteration. The parameters are the weights and biases of the neurons.
Once the feed-forward pass has completed the next step is to update all the parameters in order that in the next feed-forward pass the cost function is smaller. To update the parameters, the parameters are moved in the opposite direction of the gradients of the cost function w.r.t them. This is called Gradient Descent Algorithm, and basically, taking into account that the gradients indicates the direction of steepest increase of the cost function, if the parameters are moved towards the opposite direction, the next step the cost function is supposed to be smaller.
The problem with neural networks is that we don't know the gradients of the cost function because the number of the parameters are massive (billions) and the cost function themselves are quite complex (not convex, with pathological curvatures...) To overpass this issue, backpropagation has invented as a means of efficiently compute the gradients of the cost function (it uses the chain rule and propagates the errors from the lat layer to the first one)

To the best of my understanding , weights are updated during back propagation only using gradient descent and no gradient descent is used during feed forward propagation. is it correct ?

Yes and no...

Yes if you think of gradient descent as only the updating stage of the parameters
No if you understand Gradient Descent as a whole, in which you need the results obtained in the forward pass to compute the values of the gradients in the backwards pass

I'm personally prone to the second option, in which Gradient Descent comprises two stages: a gradient computation and parameter updating


Logistic regression program does not give statistically correct result

If you initialized the weights randomly and predicted immediately, you should probably expect 1/num_classes accuracy.

But you are training first. The data may be random, but it is still possible to overfit it. With a sample that small and wide, you could imagine some degree of separability of the classes by chance. In that scenario, your network can learn weights that are better than random (far from perfect in only one iteration).

The effect would likely be lessened by using much more data or holding out/using new random data to test accuracy.


How to include categorical fields to enhance a text classification

Scikit-learn has compose.ColumnTransformer which

allows different columns or column subsets of the input to be
transformed separately and the features generated by each transformer
will be concatenated to form a single feature space. This is useful
for heterogeneous or columnar data, to combine several feature
extraction mechanisms or transformations into a single transformer.

A demo of mixing numeric and categorical types is here. In your example, CountVectorizer is numeric and Label is categorical.


setting class weights for imbalanced dataset, how using EarlyStopping?

Early stopping is not directly affected by imbalanced data.
Also comput_class_weight effects the training, not the evaluation.
As far as picking a metric for evaluating imbalanced data, it depends on the specific problem. The most common choices are F-score, precision, and recall. It appears that you are using scikit-learn which offers a weighted option, finding average weighted by class support.


How to train and evaluate machine learning models with growing/changing datasets over time

The solution you described is definitely an option.
The other motivation for retraining is that the distribution of new data has drifted away from that of the original dataset and that warrants retraining to maintain the same level of model performance.
In such situations, It is possible that you may actually throw away some of your older training data because they may have become stale and include a part of newly available data for training. The remainder new data can then be used for out of sample testing and validation.
You also want to be able to figure out when to trigger the retraining. This is usually done by an independent drift detection process, which keeps track of some metric related to your Business and triggers retraining if it dips below a predefined threshold.


How can I get the diameter of each community

The method diameter doesn't appear to take vertex sets as a parameter.  Instead, call diameter on the appropriate induced subgraphs:

for ic in s_List:
    y = data_graph.induced_subgraph(ic).diameter()
    print(y)

Edit: Erm, I guess this depends on exactly what you want to measure.  While it's rather against the purpose of "communities," it is possible that the distance between a pair of vertices in your community is longer if the paths are restricted to the community than if they can pass through the rest of the graph.  Inducing the subgraphs as above would miss that.


Following up on the comment about deep learning, with high dimensional time series data you would be much better served with a recurrent-type of deep model. For example, an LSTM is a very good starting point with high-dimensional data. This may be a good place to start: Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras

Although CNNs are very useful for high-dimensional data, when you have a time series, it's best to start with a model that is designed for a time series. A CNN may do well, and you should compare your results to a CNN, but it is not a time series model.


How to recreate T-SNE dimensions deterministically?

Do not use tSNE visualizations for clustering. The results are misleading.

See this great answer: https://stats.stackexchange.com/a/264647/7828

Apart from that, you just need to fix the initial positions. For examples by fixing the random generator seed. But the fact thar it does not work every time should already warn you that it is not too reliable...


is there any way to plot ROC curves from weka

In the Weka explorer, go to the classify tab and train/test your algorithm.
The result buffer appears in the bottom left box under the section labeled "result list"
Right click the result buffer and click visualize threshold curve, then select the class you want to analyze
to save the ROC curve as an image, hold shift + alt and left click on the graph


Should estimated probabilities from multi class classification sum to 1

Summing up to one or not both can have their special meaning that I'll try to explain them. If you have classes that are not mutually exclusive, say you have dog and cat classes and they both can exist in an image. In such cases, you should use sigmoid nonlinearity as the output of each class and interpret each one separately. Each one that has a value greater than half can explain the existence of the corresponding label. On the contrary, your inputs may be mutually exclusive which means in each input you may just have a cat or a dog. In this case, you should employ softmax nonlinearity and yes, it sums up to one. The winner would be the one with the highest value.


In Line 207, generated pictures are written to a directory images, which must exist. In your case it does not, that's why it throws that error.


How to set limits of Y-axes in countplot?

Countplot from seaborn will not work as you expect. When you calculate the frequencies, you want to plot the values in p.values as they appear. Countplot will take a dataframe where labels are not aggregated and then count each one of them, as it did in the first case.

So countplot will be appropriate for the case where your dataframe looks like:

index | reason |
0        EMS
1        EMS
2        Traffic
3        Fire
4        Fire
5        EMS
6        Traffic
...

In the second case you already have your frequencies:

index | reason |
EMS       10
Traffic   21
Fire      15

Then count plot will just count the lines and it will be one for each, that is why your plot looks like that.

To solve your problem you could just plot using .plot from pandas:

df['reason'].value_counts(normalize=True).plot(kind='bar')

Where the parameter normalize=True will show normalized frequencies instead of raw count values.


At 3:30 in video, he mentions this switching. What he is referring to by 'switch around' is that he is specifying the size of the LSTM being used. The 1024 refers to number of cells (or size) of the LSTM, and the 3 is the number of layers.

In the tutorial page being referenced I the snippet he is referencing is:

python translate.py
  --data_dir [your_data_directory] --train_dir [checkpoints_directory]
  --size=256 --num_layers=2 --steps_per_checkpoint=50

For details on why LSTMs are useful for this task, check out the paper.


Sklearn has different objects dealing with mutual information score.
What you are looking for is the normalized_mutual_info_score.
The mutual_info_score and the mutual_info_classif they both take into account (even if in a different way, the first as a denominator, the second as a numerator) the integration volume over the space of samples.
Your code becomes
import pandas as pd
from sklearn.metrics import confusion_matrix
from sklearn.metrics import normalized_mutual_info_score, mutual_info_score

y = [1,1,1,1,1,0,0,0,0,0,1,1,1,1,1,0,0,0,0,0]
print("Confusion matrix:")
print(confusion_matrix(y,y))

print("Mutual information:")
result = normalized_mutual_info_score(y, y)
result_not_normalised = mutual_info_score(y,y)
print('norm: ', result)
print('not-norm: ', result_not_normalised)

giving,
Confusion matrix:
[[10  0]
 [ 0 10]]
Mutual information:
norm:  1.0
not-norm:  0.6920129648318737

Note also that $ln(2) \simeq 0.6931$.


Reduce multiclass classification targets to binary classification targets in scikit-learn

Of the three stated purposes of pipelines, you'd get the "convenience and encapsulation" one, but not the others:

Joint parameter selection: you don't have any parameters for this transformation.
Safety (from data leak): your transformation is context-specific, so there is no data leakage in applying it to the entire dataset up front.

This feels like something that is the definition of the targets, and is best considered a part of the data retrieval.

sklearn expects transform methods to have input just X and not y.  For the most part, you can work around that by overriding fit_transform from TransformerMixin.  However, nothing downstream will expect to get two return values (transformed X and y), so this won't work.
You can make a little more headway with the imbalanced-learn package, which provides its own Pipeline with more flexible transformation syntax.  The purpose there is to implement resamplers, and that throws a major issue: resamplers do not apply at prediction time.


Tensorflow predicting same value for every row

I think I found the problem. I need my input to have shape of (1, 256, 256, 1), thats why I performed image = image[...,0][...,None]/255 but doing so changed all the values to 0


Difference between business and financial and data analyst

Business Analyst
A business analyst is one who understands the specific domain of the project (ex. retail, merchandising to be specific, supply chain etc.). His/her role is to understand the business problem, analyze the current state and capture requirements using various tools like surveys, interviews, group discussions and then provide recommendations and create a Requirements document for sign off.
Business Analyst Skillset-

Communication,  Analytical thinking, Domain Knowledge, Generic
technical Knowledge, Problem solving skills,Decision solving skills,
Managerial Skills,Negotiation and Persuasion Skills.

Financial Analyst
Financial analysts use financial data to spot trends and extrapolate into the future, helping their employers and clients make the best investing decisions. Businesses rely on financial analysts to determine when it is an auspicious time to buy or sell specific securities and, in some cases, companies use reports put together by financial analysts to determine if the entire business should be sold.
Financial Analyst Skillset-

Financial analysts should be great problem-solvers, excel at the use
of logic and possess strong skills in quantitative analysis. In
addition, successful financial analysts have an in-depth understanding
of various financial markets and investment products.

Data Analyst
A data analyst's role is one that works with lots of data to derive meaningful insights to either address business problems or discover hidden trends and patterns that can be leveraged to meet the business objectives.
Data Analyst Skillset-

Data analyst also needs similar skills with some additional skills
like to analyze data like SQL, DATA mining, OLAP, Reports etc.

Note: Strong people skills, leadership ability, and teamwork are beneficial for each type career.
For more info, you can visit these resources-

Career Advice: Financial Analyst Vs. Data Analyst
What is the difference between data analyst and business analyst?
The Differences Between a Business Analyst & a Data Analyst

Hope it helps, cheers! :)


Dummy vectors and performance measurement for vector search Face Recognition

No, creating dummy unknowns is not the best way to do it.
A better approach can be, if a new face comes in, we calculate distance between vector of the new face and all of the vectors of known faces already present with us. And to identify the correct face, the minimum distance is considered. But this minimum distance should also be below a threshold value. If the minimum distance is above the threshold value, then it is considered as an unknown face. The threshold value is set manually.
Just to give you the idea, here is pseudo example -
Let's say you have 5 registered (known) faces. Their vectors are [1] [2] [3] [4] [5]. Assume that your model represents each face by a vector of shape 1 x 1.
You set the threshold value as 0.4
Your distance function is defined as abs(vector1 - vector2)
Scenario 1 -
A new face comes in, and your model generates a vector [2.2]
So you calculate the distance between new face and known faces as - [1.2] [0.2] [1.2] [2.2] [3.2]
Your minimum distance is [0.2] which is less than your threshold 0.4. Hence, this new face is identified as "[2] 2nd face" face.
Scenario 2 -
A new face comes in, and your model generates a vector [7.5]
So you calculate the distance between new face and known faces as - [6.5] [5.5] [4.5] [3.5] [2.5]
Your minimum distance is [2.5] which is more than your threshold 0.4. Hence, this new face is identified as an "unknown" face.
Lastly,
As you mentioned, you can use KNN based approach. Although it would produce very accurate results, it might not scale well. As number of faces in your database grows, KNN approach will slow down.


Yes, your logic and what you are thinking is excellent.

There is only a flaw in your thinking: The variables you run the model with must not necesarily be "correlated" in a linear sense of the word, just don't discard any variable because any of them could explain your binary output, and not have a linear relationship with it.

Is a common solution to binarise an output to detect anomalies, but you will lose the ability to predict "how much" outlier is an outlier, make sure you don't need this information after.


What is the output of multivariate LSTM model?

If you want to predict "Avg CPU Load", your y_train should be only the "Avg CPU Load" column and your x_train should be rest of the columns (you might skip the time column if all timestamps are equally spaced).

The network will then learn a function mapping rest of the columns to "Avg CPU Load" and it shall be predicting the same.

Hope it helps.

edit: in testing phase, also you have to follow the same structure for x_test (and y_test i.e. ground truth).

edit2: In this case, for RNN, you are right by generating each training sample as all columns over 60 timestamps. You can check if the prediction is indeed  what you want by plotting against  ground truth. Although, I think you should  predict all the columns (new_y_train is all columns instead of avg CPU load only) i.e. make the output dimension same as input dimension. Make the network predict the whole thing and pick your column of interest for your consideration. This is probably the more natural way to do it.


Regression yields much smaller standard deviation and the mean is off, what could be wrong?

The difference in standard deviation is nothing suspicious.  It is only to be expected, if you have a weak correlation.

Suppose the regression is y ~ x, i.e., y = ax + b.  Suppose that x only explains a small fraction of the variability in y, i.e., y is scattered all over the place and the least regression line only weakly fits the data.  Suppose also that the line is nearly horizontal (i.e., a is small).  Then the standard deviation of the predicted y-values will be small (since the line is nearly horizontal) but the standard deviation of the actual y-values might be large.

But really, the way to figure out what is going on is to visualize the data.  You should always start by visualizing the data.  Plot a scatterplot, and superimpose the least squares fit line on top of it.  I bet you'll immediately have a better sense of what might be going on.


Managing NaN in target variables (testing)

Well, you can't do supervised learning on unlabeled data. Either you discard those records or you have to go semi-supervised way.


Can setting of different thresholds help in model performance in case of handling class imbalances?

If changing the threshold improves the performance of the model, it would be better to change it and use an optimal value. Balancing the dataset does not mean that you can not change the discrimination threshold. Also, in some cases when the data is originally balanced, changing the threshold could be very useful and it is a very smart move.


Building a search tool and classifying text using NLP and ML

Here are a few general thoughts about your project:

As far as I understand, you're trying to extract very specific information from a semi-structured database using free-form natural language queries, correct? If yes it's important that you realize that this is a quite ambitious project, reaching a decent stage of quality is probably going to take a lot of work, and the performance is unlikely to be perfect.

Apparently numerical values and units are important pieces of information for matching the query. In this case you should probably implement a special process for those, because standard text processing is not going to work very well.

Detection: if there is only a small number of possible ways in which these values are written, it's probably more efficient to use ad-hoc regular expressions. If not, you could try to train a custom NER model.
Representation: that's the tricky part imho. For general text vectors are fine, but with vectors it's difficult/impossible to represent special values like those. Given that you have a semi-structured database, you might want to try a more semantic representation adapted to your data: that could involve techniques from semantic role labeling, relationship extraction, etc.
Matching: the advantage of a semantic representation is that you could convert the query to a semantic representation and then apply a detailed matching procedure suited to your data (in particular you can compare corresponding numerical values and use a threshold, or return the difference to represent how far they are).


two different attention methods for seq2seq

(a) is Luong's attention mechanism (link) while (b) is  Bahdanau's mechanism (link)


How to choose between Tensorflow and Pytorch?

As of October 2020...
In terms of basic neural network functionality, they are pretty equivalent.
Some differences:

Stability: tensorflow 2.0 underwent a lot of changes from tensorflow 1.x, specifically in the very way it worked (they changed from a computational graph paradigm to an imperative paradigm). This caused a lot of friction and left many underlying problems in tensorflow.
API coherence: tensorflow API has evolved and changed a lot and, through its evolution, many pieces of functionality were duplicated, and later where deprecated and removed. Also, there is a framework called Keras, which, at the beginning was a separate piece of code, but now is integrated in tensorflow. The co-existence of tensorflow and keras brought tension (see here). The Pytorch API has been more consistent over time.
Documentation: the API of tensorflow has changed a lot over the time that many tutorials and stackoverflow questions are outdated. Pytorch has changed less and has kept good backward compatibility so, while there are some tutorials that may include outated practices, most of them should work.
Deployment: tensorflow is known to be better suited for "production scenarios", e.g. it has tensorflow serving for exposing trained models through a service. This, however, is changing and now there are official alternatives for model deployment for pytorch.
Mobile support: tensorflow lite is the official way of having deep learning models use the GPU of Android mobile devices, so it's easier to deploy on Android if you use tensorflow. Note that, while there is a mobile version of Pytorch, it does not support GPU yet.

So, how to choose? This is my opinion:

Use tensorflow if you have a specific reason for using it, e.g. you want to deploy on Android easily, you want to use tensorflow serving, you want to reuse models that are only available for tensorflow.
Use pytorch in any other case.


How to know which loss function is suitable for image classification?

If you have two classes (i.e. binary classification), you should use a binary crossentropy loss.

If you have more than two you should use a categorical crossentropy loss.


Print the prediction of convolutional neural network

You can replace these lines:

opt = sess.run(optimizer, feed_dict={x: batch_x,
                                     y: batch_y})
loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
                                                  y: batch_y})

by:

opt_, loss_, acc_, out_ = sess.run([optimizer, cost, accuracy, out], feed_dict={x:batch_x,
                                                  y: batch_y})

In this way, you can evaluate all the parameters given in the list (optimizer, cost, accuracy, out) in a single sess.run(). After, if it is classification task, you will probably need to apply softmax and argmin to out_


Text-Classification-Problem, what is the right approach?

I think perhaps the first thing to decide that will help clarify some of your other questions is whether you want to perform binary classification or multi-class classification. If you're interested in classifying each instance in your dataset into more than one class, then this brings up a set of new concerns regarding setting up your data set, the experiments you want to run, and how you plan to evaluate your classifier(s). My hunch is that you could formulate your task as a binary one where you train and test one classifier for each class you want to predict, and simply set up the data matrix so that there are two classes to predict - (1) the one you're interested in classifying and (2) everything else.

In that case, instead of your training set looking like this (where each row is a document and columns 1-3 contain features for that document, and the class column is the class to be predicted):

1           2           3           class
feature1    feature2    feature3    politics
feature1    feature2    feature3    law
feature1    feature2    feature3    president
feature1    feature2    feature3    politics

it would look like the following in the case where you're interested in detecting the politics class against everything else:

1           2           3           class
feature1    feature2    feature3    politics
feature1    feature2    feature3    non-politics
feature1    feature2    feature3    non-politics
feature1    feature2    feature3    politics

You would need to do this process for each class you're interested in predicting, and then train and test one classifier per class and evaluate each classifier according to your chosen metrics (usually accuracy, precision, or recall or some variation thereof).

As far as choosing features, this requires quite a bit of thinking. Features can be highly dependent on the type of text you're trying to classify, so be sure to explore your dataset and get a sense for how people are writing in each domain. Qualitative investigation isn't enough to decide once and for all what are good features, but it is a good way to get ideas. Also, look into TF-IDF weighting of terms instead of just using their frequency within each instance of your dataset. This will help you pick up on (a) terms that are prevalent within a document (and possibly a target class) and (b) terms that distinguish a given document from other documents. I hope this helps a little.


How to recognize product based on image using neural network?

Is it possible to train new model with this kind of data

Yes, you need Convolutional Neural Networks (CNN) for image classification. If you only have one image per product in your dataset, I suggest you to use a lot of image augmentation, a technique that is meant to artificially increase the size of an image dataset by applying combinations of distortions to image data.

  or build it using existing model?

You can re-train existing models. On the TensorFlow website you can find a huge list of pretrained CNN architectures that you can download ad use for your needs. It's a powerful alternative to training a new model from scratch.


Will reinforcement learning work if states wont get repeated again?

I am not convinced that framing a document retrieval success as reinforcement learning problem will be easy to get working.

However, your core question is answerable. This problem occurs in other problems that are too large to explore all possible states. For instance, in the game of Go, the AlphaZero system cannot possibly explore all game positions. Similarly, in Atari games, many of which have also had reinforcement learning applied successfully, it is very unlikely that the agent has seen all possible states of the system - typically it will have trained on a million video frames.

The answer to this issue is to use some form of function approximation that can generalise to new unseen data. That is precisely what supervised machine learning models do, so the RL agent will use one internally, typically a linear regression or neural network, so that values and or policy learned from states it has explored will be associated with similar unseen states (for some interpretation of similar).


Mathematical way of identifying wrong suggestions or outliers

what is the best mathematical aproach to dynamically find the best possible threshold value above which i can say that the definition is correct and below which you need to have a look if the classes are correct

First, in general it's clear that there's no way to be certain that an answer is correct or incorrect: for instance if for some reason all the classifiers return the same wrong answer, then the threshold condition is satisfied but the answer is wrong.
Now the only way to find an optimal threshold to decide whether an answer is acceptable is to use a set of labelled examples in order to evaluate the answers of the classifiers.
Why? Because the correctness of the answers cannot be determined by any mathematical formula.
So the goal of the game is to assess to what extent this set of classifiers can be trusted to give a correct answer, and that depends on the threshold. So the threshold is a parameter of the predicting system, and this parameter can be estimated (tuned) based on some validation data. The basic method consists in trying all the possible values for the parameter, evaluate the performance in every case and pick the value which obtains the highest performance. Note that there are many  possible evaluation measures for such a scenario: the measure should be chosen carefully depending on the goal of the system (e.g. is it better to have one false positive answer or false negative?).


Can Keras be used to build clustering models

Yes, you can use Keras to build clustering algorithms. Recalling that Keras is a high-level api for Tensorflow or Theano, and that you can build a K-means algorithm (very well known clustering technique) in Tensorflow, you should be able to build it in Keras. Example (in tensorflow):https://learningtensorflow.com/lesson6/.

Nevertheless, sklearn has in-built clustering algorithms like K-means or Gaussian Mixture Models, so at first, I think a better approach is using these sklearn funcionalities.


Label shift is the opposite of a covariate shift.
 In this case, the assumption is that even though the feature distribution remains the same, the Label distribution might changes.
 e.g. Symptoms --> Diseases
 It can be different for different country (based on medical education of the Country/Doctor)
 It can change with time also based on advancement in Medical knowledge
  Similar logic can be built for "Words --> Slang". It can change with time due to the acceptance of these words.
Read the references for a formal explanation.
References-
Dive into Deep Learning
Detecting and Correcting for Label Shift with Black Box Predictors


Naive Bayes doesn't assume independence of attributes ... It assumes conditional independence (or what you call independence within a class). This allows us to write the likelihood in the bayes rule P(X | Y) as the product of all P(Xi | Y), where X = (X1, ... Xi, ... Xn) and n is the number of attributes.


How to Convert a Pandas Column having duration details in string format (ex:1hr 50m) into a integer column with value in minutes

One liner:

df['Duration']=  df['Duration'].str.replace("h", '*60').str.replace(' ','+').str.replace('m','*1').apply(eval)

Basically, it converts your string into an equation in string format and evaluates it.

Hope it helps!


Just having 200 unique labels and using MultiLabelBinarizer does not automatically mean overfitting. Overfitting is an empirical question. The number features and observations relative to the complexity of the algorithm effect the chance of overfitting.


How to convert binary classifier to multiclass classifier?

Yes it is.

For multiclass classification problems, you can use 2 strategies: transformation to binary and extension from binary.

In approaches based on transformation to binary, you have:

OVA (one versus all), which is based on training k binary classifiers (k = #classes), where the i-th classifier is specialized on distinguishing the i-th class from all the other k-1 classes.
OVO (ove versus one), which is based on training k * (k-1) / 2 classifiers, where each classifier learns to distinguish 2 classes only. When a prediction is required, each clasisfier votes on the class it thinks it's correct, and the class with more votes is selected as the output.

On the other hand, you have extension from binary approaches: some classification algorithms are already capable of dealing with these multiclass problems. Some examples: kNN, decision trees, naive bayes...

You can find a bunch of resources on this.
For more practical purposes, please check out the following resource: https://scikit-learn.org/stable/modules/multiclass.html


There are many ways to calculate similarity between articles. I have not seen anybody doing a vector conversion and running comparisons. However, there is a text mining strategy called "Term-Frequency / Inverse-Document-Frequency" which is a clever way to find unique words and phrases in documents. You can run this on multiple documents, and compare the extracted keywords to match them for recommendations.

Check out my ebook for more details: https://lizrush.gitbooks.io/algorithms-for-webdevs-ebook/content/chapters/tf-idf.html

If you want to leverage this technique on web documents (like a blog), there is a free service to do so: https://algorithmia.com/recommends


The evaluation of an imbalanced dataset should be done with more than one single metric since you need to evaluate the performance on both the majority and the minority classes.

In the case that is mentioned the performance is evaluated only with recall (the true-positive rate). It could be that the model is now focusing on the oversampled minority class (especially if it's the new majority class) and having a weaker performance on the original majority class, and I assume that the precision or the false-negative rate were decreased.

There are several evaluation metrics that are considering the performance of both classes that are not sensitive to the classes imbalance - F1 score, Geometric mean, and the most common the ROC/PR AUC


What kind of clustering would work better on such data? Would k-means work on such data?

Theoretically, Gaussian Mixture Model could identify [\, /] clusters.


How to improve accuracy in the following code?

Adding more images in the training set is one way to increase accuracy. You can also do transfer learning, i.e. using layers trained by a larger dataset if you are classifying common things such as animals which have networks that have already been trained such as the ResNet50.

When you have enough images that the accuracy does not increase anymore, you can try adding more layers, with convolutional and ReLu layers alternating. If you train for many epochs and the model starts to overfit i.e. training set accuracy still trends up but test set accuracy trends down, you can add layers that reduce overfitting, such as the dropout layer. You should first get the model to overfit and then try reducing overfitting.

If you don't have enough training data you could also do augmentation, e.g. adding rotated, shifted and horizontally flipped images to the training set.

The thing with traffic signs is that if you are classifying images with also the landscape instead of just the cropped traffic signs, you are going to have problems. You should then also detect the traffic signs (YOLO v3 could be used as transfer learning) instead of just classifying them.

Here's also a good example of traffic sign classification using transfer learning on Kaggle: https://www.kaggle.com/valentynsichkar/traffic-signs-classification-with-cnn


How to save prediction values for the whole data in Keras

First, you could always just wrap you code with a loop.

files = [file1, file2, ...]
predictions = []
for file in files:
    original = load_img(file, target_size=(224, 224))
    numpy_image = img_to_array(original)
    image_batch = np.expand_dims(numpy_image, axis=0)
    processed_image = vgg16.preprocess_input(image_batch)  # you dont need to copy :)      predictions_cur = vgg_model.predict(processed_image)
    print(predictions_cur)
    predictions.append(predictions_cur)
    #label = decode_predictions(predictions_cur)

Secondly, you could improve the efficiency of this code by using the tf.data api.
For example:

from tensorflow.data import Dataset

batch_size = 32
num_threads = 4
target_size=(224, 224)

def preprocess_image(file):
    # file is not a tensor, so we have to use functions that work on tensors
    img_string = tf.read_file(file)
    img = tf.image.decode_png(img_string)
    img = tf.image.resize_images(target_size)
    img = vgg16.preprocess_input(img)  # this function can handle tensors as well.
    return img

files = [file1, file2, ...]
dataset = Dataset.from_tensor_slices(tf.constant(files))
dataset = dataset.map(preprocess_image, num_threads)
dataset = dataset.batch(batch_size)
dataset = dataset.prefetch(1)
predictions = vgg_model.predict(dataset, steps=len(files) // batch_size)
# You could decode the predictions from here :)

Explanation

Let me elaborate on each step :)

dataset.from_tensor_slices(files) Will just build a dataset object, where the samples it will use are your files.

dataset.map(preprocess_image, num_threads)
This indicates to preprocess each sample before we use it.
This will also be made in num_threads parallel calls, which will haste the process in case we batch samples together for prediction.

dataset = dataset.batch(batch_size)
This will batch samples together for prediction. This is useful to make predictions in parallel and haste the process.

dataset.prefetch(1)
This will prefatch the next batch while the previous one is being predicted upon. So this will also make the process faster because you save time this way.

and finally predictions = vgg_model.predict(dataset, steps=len(files) // batch_size)
This will predict over the all the files we've provided to the dataset above.
Do notice that we indicate that the number of steps is len(files) // batch_size so if len(files) isn't a multiply of batch_size, some images will not be predicted upon. If you want all the images to be predicted upon, you can use a batch_size that divides len(files), or just use batch_size = 1.


computer science student - asking for some machine learning guiding (voice cloning)

Try to look into existing text-to-speech/speech synthesis. This link list multiple different models/approaches in this domain. When trying to look at state-of-the-art (SOTA) methods in this domain I find this GitHub page on which a lot of papers can be found on audio and speech. A very good (SOTA) model/architecture seems to be Tacotron 2/WaveNet. Hope this helps you with your project.


You've given all four regions a dummy variable, so these are perfectly multicollinear, and the (unpenalized?) regression doesn't have a unique solution.  R automatically drops a column in this situation and reports the NA.

https://stats.stackexchange.com/q/212903/232706
https://stackoverflow.com/q/7337761/10495893
https://stats.stackexchange.com/q/25804/232706

(You can see where R calls C and subsequently FORTRAN functions, then inserts the NAs at
https://github.com/wch/r-source/blob/0f07757ad10ca31251b28a2c332812e63c0acf38/src/library/stats/R/lm.R#L117
A nice article that helped me find that: http://madrury.github.io/jekyll/update/statistics/2016/07/20/lm-in-R.html )


How to interpret Logistic regression coefficients using scikit learn

Significance: As far as I know, sklearn does not come with a module to get p-values. You could go for statsmodels:

https://stackoverflow.com/a/55655645/9524424

Interpretation: Remember that Logit uses a logistic link function, so there is no easy on-the-go interpretation of the coefficients. You need to calculate marginal effects, so the change in $y$ when you alter some $x$, all other things equal. Stata has such module(s). However, I think sklearn has no such implementation. Python statsmodels seems to come with an option to derive marginal effects as well, but I never tried it. Look for .get_margeff.

https://www.statsmodels.org/stable/generated/statsmodels.discrete.discrete_model.DiscreteResults.get_margeff.html


Fast AI Lesson 4 - MNIST. Confused about multiplying weights by pixels?

The input images contain a total of 784 (28 * 28) pixels, meaning that the model has 784 inputs. If we then use one node in the next layer which is connected to all inputs we also get 784 weights as each pixel is connected to the node in the next layer. See also the following image:

The input layer contains all the different inputs to our model (i.e. all pixels) and as you can see all those inputs are connected to all nodes in the next layer. Each connection/line has a specific weight assigned to it, the number of weights for a fully connected layer is equal to n_input * n_output. In the fast ai course this means that there are 784 weights (28 * 28 * 1), while in the example below there are 12 weights (3 * 4) for the first and second layer and 8 (4 * 2) for the second and third layer.
The reason for the bias already seems explained in the notebook itself, without the bias the function is not flexible enough. If all inputs are equal to zero the output would also be zero, whereas it needs to be a number different from zero. The bias allows for this by shifting the function up and down, see also the b term for a linear regression.


Why does adding random pixels stop my model learning in cnn?

I think the problem here is that you are trying to relate what you know about the input to what the model is trying to predict. I suggest trying to interpret why your model is giving certain outputs for images.

I came across SHAP a while back when reading about model interpretation. The image plot of SHAP should be able to narrow in on why your model is interpreting a label for a specific Image.

This should lead you to figure out why your model is making the prediction.


Proper derivation of dz[1] expression for backpropagation algorithm

For backpropagation algorithm, is it true to have weight, $w$ transposed in the expression of $dz^{[1]} = w^{[2]T}dz^{[2]} * g^{[1]'}(z^{[1]}) $ ?

Yes it is correct. You can show it is with some different arguments:

Checking correct dimensions

Although this is not a robust theoretical argument, checking dimensions is actually what I do in practice when I am confused when implementing NNs.

Here's the rough logic of a dimension check on the equation in your question:

$W^{[2]}$ is a $n^{[2]} \times n^{[1]}$ matrix for the forward propagation to work.
$dz^{[2]}$ is a $n^{[2]} \times 1$ column vector (because it is the gradient of cost function with respect to $z^{[2]}$, so it has to have the same dimension as it)
$dz^{[1]}$ is a $n^{[1]} \times 1$ column vector
It is $dz^{[2]}$ that is being multiplied - so the multiplying matrix must be $\text{something} \times n^{[2]}$.
The output needs to be same dimensions as $dz^{[1]}$ - so the matrix must also be $n^{[1]} \times \text{something}$.
That means the multiplying matrix must have dimensions $n^{[1]} \times n^{[2]}$ - which $W^{[2]T}$ has (and no other component of the network could match)

Theory from item-by-item calculation, converted to matrix math

But why is it $W^{[2]T}$ and not some other matrix with the desired dimensions? For that we need to go back to the chain rule. Note that the expression you have written is actually two steps of the chain rule combined - first we derive $da^{[1]}$ from $dz^{[2]}$, then we derive $dz^{[1]}$ from $da^{[1]}$. It is that first step that introduces terms from $W^{[2]}$, so I will just show that. Also, deriving this directly using matrix notation is more complex, so we'll just drop into using the index values, and use the "official" partial derivative notation.

To calculate $da^{[1]}$ for a specific $i^{th}$ neuron, you have to sum over the gradients $dz^{[2]}$ for all the neurons it links to:

$$ \frac{\partial J}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} \frac{\partial z_j^{[2]}}{\partial a_i^{[1]}} = \sum_j \frac{\partial J}{\partial z_j^{[2]}} W_{ij}^{[2]} $$

Doing this for each $i$ value in turn gets your the full $da^{[1]}$ vector.

Compare this to how the forward propagation works in the same network using the same notation and indexing:

$$z^{[2]}_j = b^{[2]}_j + \sum_i W_{ij}^{[2]} a_i^{[1]}$$

And you can see that the index used to drive the sum is different between forward and backward calculations:

When we write $W^{[2]}_{ij}$, then $i$ is index of matrix row, $j$ is index of matrix column from $W^{[2]}$.
When multiplying forward, we sum over $i$, i.e. each row of $W^{[2]}$ multiplied by each matching element of $a^{[1]}$, and this is summed up.
When you look at the gradient, you can see the equivalent sum is over $j$.
To turn that gradient calculation into a normal matrix multiply (just so the matrix notation works, and you don't have to write out the sum) you have to swap rows and columns - which is what a transpose is.

This is partly a definitions and notation issue. We could define a different kind of matrix multiply that worked with sums column-by-column. But this is not usually done, because multiplying by the transpose does the same thing.


Is AUC a good metric for evaluating the performance of a multi-class classification?

No - AUC (Area Under Curve) can not be used directly to assess the performance of multi-class classification.
If you want to use AUC, it is necessary to binarize the output. Either each class as to be compared against each other class or 1 class versus the rest.


# make a month column to preserve the order
df['month'] = pd.to_datetime(df['date']).dt.strftime('%m')

# create the pivot table with this numeric month column
df_pivot = df.pivot_table(index='month',columns=['type','text'],aggfunc=sum, fill_value=0).T

# create a mapping between numeric months and the English version
mapping = pd.Series(df.transdate.tolist(),index=df.month.values).drop_duplicates()

# replace the columns according to the mapping
df_pivot.columns = [i for i in map(mapping.get, df_pivot.columns)]


How do I use the data I have to make predictions?

I'm quite pessimistic about this: ideally one would run the actual task (or a proxy for the task, like a simplified version of the task) with the 100 CPUs. Here the only option I can think of is to:

use the noisy measurements of the 10-20 CPUs as the basis of the data for the actual task
try to map these measurements to the 8 benchmarks, so that we can predict the performance of the 100 CPUs based on their results with the 8 benchmarks.

I'm pessimistic because:

the measurements on the tasks are noisy,
there's no guarantee that these measurements can be reliably linked with the 8 benchmarks,
and no guarantee either that the 8 benchmarks would provide a good estimate on the task.

That being said, technically you could imagine a design with supervised learning, the idea being to predict performance on the task (target variable) from the results of the 8 benchmarks (features). The training data would consist of the results of the 10-20 CPUs for which you have both results (8 benchmarks + performance on the task). I would suggest trying this with decision tree regression because it provides you with an interpretable model (white box solution).


Bagging main goal is to minimize variance of your model.  Basically, if you have a model that is on average pretty accurate but inconsistent (meaning, it does well for a given data set, poorly generalizations) then bagging may be a way to produce a more consistent estimators.  Decision trees are the common example of this because they are the canonical high variance machine learning algorithm.

As for your last question, the size of each new training set needs to be the same size as the original training set.  The way you achieve this is by random sampling of the original dataset with replacement (meaning the new dataset may have duplicates).  The number of new training sets is dependent on the problem.  Sometimes 100 is fine other times you need 1000 or so.  There isn't a way to just know how many sets you need.  It is a parameter that needs to be tuned.


XGBoost is it possible to prevent a feature from being used twice in the same tree?

If you mean "how many times the same feature can appear in an [individual] tree", then
you can use max_depth to indirectly limit the number of features included in a single tree, even down to one feature. Since XGBoost is designed to use weak learners, having a lower depth value is ok.
model = XGBClassifier(max_depth=n)

However, I think the problem is not that XGBoost is getting stuck on a single feature. For example, maybe the other columns contain little or no correlation to the label. How does your model perform when you remove that column entirely. That should tell you if you only have one good feature in your dataset.


As You said, the most common situation for recommender system is to predict rating. Then RMSE/MAE is used. For results of a ranked item list different measures are used, e.g. Prec@K, Rec@K, AUC, NDCG, MRR, ERR. However, the are many algorithms for recommendation with its own hyper-parameters and specific use cases. Depending what You are trying to achieve You should benchmark Your solution with suitable method. The best source for this can be publication, where as You noticed, many times results for ML dataset are given. What's more about benchmarking recommender systems, sometimes the best prediction results are not so important as other aspect e.g. novelty or diversity.

For most common methods try this benchmark results for librec project:
http://www.librec.net/example.html
There are no results for ML-10M, only ML-1M for rating and ML-100K for ranking. But You can always run chosen algorithm by yourself. There are other projects of this kind: http://www.mymedialite.net/, http://lenskit.org/.

Saying about evaluation of recommender systems this project should be mention:
https://github.com/recommenders/rival
It's dedicated for data preparation (splitting) and evaluation for recommender systems. It was presented at last RecSys'2014 [1]. And also look at this [2]. There are some benchmark results too.

[1] A. Said and A. Bellogín, “RiVal – A Toolkit to Foster Reproducibility in Recommender System Evaluation,” pp. 371–372, 2014.

[2] A. Said and A. Bellogín, “Comparative Recommender System Evaluation : Benchmarking Recommendation Frameworks,” pp. 129–136.


How to transpose rows into single column based on time-stamp index in python?


What model is recommended: I am using text features in a regression and want to interpret coefficients

I suppose you have a binary outcome (upvote: yes/no). In this case you could use simple linear (ols) regression (with lasso penalty). Each word (in a bag of words) is a „dummy“ here. If you look at the predicted coefficients, you can directly interpret them as „marginal effects“. Higher value means higher chance of getting an upvote (if the word is present). You can also directly see the magnitude of exp. increase in upvote probability.

One problem is that OLS is unconstrained wrt to y. So you can end up with predicted probability of upvote > 1. Use logistic regression if this bothers you. Under logit you will have similar results. Positive/negative coefficients indicate if a word increases/decreases the probability of an upvote. But because logit uses a transformation to squeeze y into a interval from zero to one, the coefficients are the log-odds and you cannot directly infer marginal effects. You would need to calculate this separately.

I tend to say: use OLS and get quick and dirty results if you are not interested in super precise estimates but if you merely look for robust estimates for which words are important.

However, if you really want to do this in a sound way, you would also need to think about „interaction“ of words (positive or negative effect on y), which are „masked“ on approaches as described above.


Moving from macbook (without GPU) to linux system with Titan V, only getting a 4x speedup, what am I doing wrong?

# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""A script for testing that TensorFlow is installed correctly on Windows.

The script will attempt to verify your TensorFlow installation, and print
suggestions for how to fix your installation.
"""

import ctypes
import imp
import sys

def main():
  try:
    import tensorflow as tf
    print("TensorFlow successfully installed.")
    if tf.test.is_built_with_cuda():
      print("The installed version of TensorFlow includes GPU support.")
    else:
      print("The installed version of TensorFlow does not include GPU support.")
    sys.exit(0)
  except ImportError:
    print("ERROR: Failed to import the TensorFlow module.")

  candidate_explanation = False

  python_version = sys.version_info.major, sys.version_info.minor
  print("\n- Python version is %d.%d." % python_version)
  if not (python_version == (3, 5) or python_version == (3, 6)):
    candidate_explanation = True
    print("- The official distribution of TensorFlow for Windows requires "
          "Python version 3.5 or 3.6.")

  try:
    _, pathname, _ = imp.find_module("tensorflow")
    print("\n- TensorFlow is installed at: %s" % pathname)
  except ImportError:
    candidate_explanation = False
    print("""
- No module named TensorFlow is installed in this Python environment. You may
  install it using the command `pip install tensorflow`.""")

  try:
    msvcp140 = ctypes.WinDLL("msvcp140.dll")
  except OSError:
    candidate_explanation = True
    print("""
- Could not load 'msvcp140.dll'. TensorFlow requires that this DLL be
  installed in a directory that is named in your %PATH% environment
  variable. You may install this DLL by downloading Microsoft Visual
  C++ 2015 Redistributable Update 3 from this URL:
  https://www.microsoft.com/en-us/download/details.aspx?id=53587""")

  try:
    cudart64_80 = ctypes.WinDLL("cudart64_80.dll")
  except OSError:
    candidate_explanation = True
    print("""
- Could not load 'cudart64_80.dll'. The GPU version of TensorFlow
  requires that this DLL be installed in a directory that is named in
  your %PATH% environment variable. Download and install CUDA 8.0 from
  this URL: https://developer.nvidia.com/cuda-toolkit""")

  try:
    nvcuda = ctypes.WinDLL("nvcuda.dll")
  except OSError:
    candidate_explanation = True
    print("""
- Could not load 'nvcuda.dll'. The GPU version of TensorFlow requires that
  this DLL be installed in a directory that is named in your %PATH%
  environment variable. Typically it is installed in 'C:\Windows\System32'.
  If it is not present, ensure that you have a CUDA-capable GPU with the
  correct driver installed.""")

  cudnn5_found = False
  try:
    cudnn5 = ctypes.WinDLL("cudnn64_5.dll")
    cudnn5_found = True
  except OSError:
    candidate_explanation = True
    print("""
- Could not load 'cudnn64_5.dll'. The GPU version of TensorFlow
  requires that this DLL be installed in a directory that is named in
  your %PATH% environment variable. Note that installing cuDNN is a
  separate step from installing CUDA, and it is often found in a
  different directory from the CUDA DLLs. You may install the
  necessary DLL by downloading cuDNN 5.1 from this URL:
  https://developer.nvidia.com/cudnn""")

  cudnn6_found = False
  try:
    cudnn = ctypes.WinDLL("cudnn64_6.dll")
    cudnn6_found = True
  except OSError:
    candidate_explanation = True

  if not cudnn5_found or not cudnn6_found:
    print()
    if not cudnn5_found and not cudnn6_found:
      print("- Could not find cuDNN.")
    elif not cudnn5_found:
      print("- Could not find cuDNN 5.1.")
    else:
      print("- Could not find cuDNN 6.")
      print("""
  The GPU version of TensorFlow requires that the correct cuDNN DLL be installed
  in a directory that is named in your %PATH% environment variable. Note that
  installing cuDNN is a separate step from installing CUDA, and it is often
  found in a different directory from the CUDA DLLs. The correct version of
  cuDNN depends on your version of TensorFlow:

  * TensorFlow 1.2.1 or earlier requires cuDNN 5.1. ('cudnn64_5.dll')
  * TensorFlow 1.3 or later requires cuDNN 6. ('cudnn64_6.dll')

  You may install the necessary DLL by downloading cuDNN from this URL:
  https://developer.nvidia.com/cudnn""")

  if not candidate_explanation:
    print("""
- All required DLLs appear to be present. Please open an issue on the
  TensorFlow GitHub page: https://github.com/tensorflow/tensorflow/issues""")

  sys.exit(-1)

if __name__ == "__main__":
  main()

Copy paste, run this code. Will tell you if you have tensorflow GPU support.
For example, on my MacBook Pro it says

TensorFlow successfully installed.
The installed version of TensorFlow does not include GPU support.


What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering?

How would you know you have to do cluster analysis before looking at your data ?

Setting aside data quality questions (which you should never do), a bare minimum of EDA will help you :

Know if it's relevant to do a clustering analysis (rarely imo)
Know if K-means is the best clustering tool (rarely imo)
Get an idea of the number of the clusters

Then you should do some EDA after, to understand what are the clusters you have selected.

Edit: Basically, it will help you answers this kind of question : How do I interpret my result of clustering?


This seems intuitive.
When we have a single-label response (Multi-Class setting), our response prior to encoding looks like:
[1, 4, 2, 3, 5]

After encoding,it becomes
[[1,0,0,0,0], [0,0,0,1,0], [0,1,0,0,0], [0,0,1,0,0], [0,0,0,0,1]]

Indicating the presence of ith Class. But only one.
Similarly, in a multi-label setting-
Input should be - [[1,2,5], [4,1], [2,3] [3], [5,4]]
Encoded value   - [[1,1,0,0,1], [1,0,0,1,0], [0,1,1,0,0], [0,0,1,0,0], [0,0,0,1,1]]
Indicating the presence of (i,j,k)th Classes. Can be many
There is no need to keep duplicate values in the data before encoding.
The inverse function will work in the same way.

The set of labels for each sample such that y[i] consists of classes_[j] for each yt[i, j] == 1.

But if you try to transform again the o/p of inverse, you will get the same encoding.
mlb.transform(zet)

array([
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0],
[1, 1, 0, 0, 0, 0, 1, 0, 0, 0],
   [1, 1, 0, 1, 0, 1, 0, 0, 0, 0],
[1, 1, 0, 0, 1, 0, 0, 0, 0, 0],
[1, 1, 0, 0, 0, 0, 0, 0, 1, 0],
[1, 1, 0, 0, 0, 0, 1, 0, 0, 0],
 [1, 0, 1, 0, 0, 0, 0, 1, 0, 0],
 [1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 1, 0, 1, 0, 0, 0]])


Could you tell us a little bit more about those 2 models ?
If your small network is recurrent, the computation graph of the model is not very parallelizable. It could explain the latency.


Keras Conv1D for simple data target prediction

Your error is coming from the Keras framework not working with strings as the output labels. You will want to transform these to 1-hot encoded vectors to train your model. Here is some code to do this.
Getting the data
import pandas as pd
df = pd.read_csv('iris.csv', header=None, names=['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species'])

This will assign a class label, we will one-hot encode them later
df['labels'] =df['species'].astype('category').cat.codes

Splitting the data and reshaping the data
First we will split the data into a training and testing set. Then we will one-hot encode the labels. And finally we will structure the inputs to match what is expected from Keras. To use a 1D convolution we need to add a spatial dimension.
from sklearn.model_selection import train_test_split
import keras

X = df[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']]
Y = df['labels']
x_train, x_test, y_train, y_test = train_test_split(np.asarray(X), np.asarray(Y), test_size=0.33, shuffle= True)

# The known number of output classes.
num_classes = 3

# Input image dimensions
input_shape = (4,)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

x_train = x_train.reshape(100, 4,1)
x_test = x_test.reshape(50, 4,1)

The model
Your model was insufficient to get good results so I added an additional hidden layer into the mix to get acceptable results.
from __future__ import print_function
from keras.models import Sequential
import keras
from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv1D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

model = Sequential()
model.add(Conv1D(32, (3), input_shape=(1,4), activation='relu'))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.summary()

Now let's train the model
batch_size = 128
epochs = 10
model = model.fit(x_train, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test_binary))

100/100 [==============================] - 0s 50us/step - loss: 1.0906 - acc: 0.6400 - val_loss: 1.0893 - val_acc: 0.7000

We get 70% accuracy, that's not so bad. But it can be improved by changing the model to better suit the data source.
Plot the convergence
plt.plot(model.history['loss'])
plt.plot(model.history['val_loss'])
plt.title('model train vs validation loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper right')
plt.show()


Is there a way to cluster words based on how similarly they sound?

I haven't tried it myself, but you could try the IPA (International Phonetic Alphabet) version of your words and then calculate the Levenshtein distance.

There is a Python library called panphon. I have no affiliate with it. Just found it with Google Search.

import panphon.distance
dst = panphon.distance.Distance()
dst.dogol_prime_distance(u'pops', u'bobz')
>> 0
dst.dogol_prime_distance(u'father', u'mother')
>> 1
dst.dogol_prime_distance(u'school', u'fool')
>> 3


how to determine percentage below which a feature can be removed from a model

Why do people say that?

when you have that much missing values (95%) than that could be encoded as special value, and it will look quantitatively as a value in itself. Now when you start using this feature as "usable" what will most likely happen is that you get noise, i.e. conflicting information/non-discriminatory one. In other words this feature will be saying the same information for all the classes (since there is 95% of it, its most likely). Hence you can do this encoding, fit a random forest and look at gini coefficients (for example!) and you quantify mathematically how important this feature is relative to the problem and the dataset.

If you are looking for quantifying mathematically this percentage generally, thats impossible since its all relative and problem specific. From experience 50%+ missing values is extremely hard to work with.

CAVEAT you dont even have to encode it as value in itself, just fit lgbm model, there missing values will be assigned with values that minimises the loss, and you may even find out that its not that useless after all.


How to find the filename associated with a prediction in Keras?

The order of the files that populate file_list, is the same order X_test appears in, by row.

So just match the indices to correlate filename with prediction.

X_test[0] ~ prediction[0] ~ file_list[0]


What is the form of data used for prediction with generalized stacking ensemble?

Q1.  This can be done either way.  You may use only the base model predictions, or those and all the original features, or anywhere in between.  Passing along the original features may be known as "feature-weighted stacking," the idea being that the meta-estimator can learn that some of the base models are better on certain subsets of the original data (but AIUI the original approach there is to select/engineer more useful subsetting features, not to simply pass along all the original features; I assume this depends on context).

For example, the parameter passthrough in sklearn's StackingClassifier passes the original dataset along, as does use.feat in mlr's makeStackedLearner.

Q2.  Absolutely, making predictions requires each of the base models making their predictions (this can be done in parallel), then passing those off to the meta-estimator.

Q3.  Yes, a simple split suffices to prevent the data leak that occurs in more naive stacking, where the meta-estimator is fitted using the "predictions" of the base models on their own train set.  And yes, you will need another test set for evaluating the performance of the ensemble.  k-fold cross-validation is often recommended for the base models so that more data is available for all the models to train on.


Definition

A inner product (AKA dot product and scalar product) can be define on two vectors $\mathbf{x}$ and $\mathbf{y}$ $\in \mathcal{R^n} $ as

$$ \mathbf{x.x^T} = <\mathbf{x},\mathbf{y}>_\mathcal{R^n}=<\mathbf{y},\mathbf{x}>_\mathcal{R^n} = \sum_{i=1}^{n} x_i \times y_i $$

The inner product can be seem as the length of the projection of a vector into another and it is widely used as a similarity measure between two vectors.

Also the inner product have the following properties:

Commutative or symmetric
Distributive (over vector addition)
Bilinear
Positive-definite: i.e $\mathbf{x.x^T}  > 0,\forall \mathbf{x} $

The covariance of two random variables $X$ and $Y$ can be defined as

$$ E[(X-E[X]) \times (Y - E[Y])] $$

the covariance holds the properties of been commutative, bilinear and positive-definite.

These properties imply that the covariance is an Inner Product in a vector space, more specifically the Quotient Space.

Association with the kernel trick

If you are familiar with Support Vector Machines you probably familiar with the Kernel Trick where you implicitly compute the inner product of two vectors into a mapped space, called feature space. Without performing the mapping you can compute the inner product into even a possibly infinite dimensional space given that this mapping.

To perform that inner product, you need to find a function, known as kernel functions, that can perform this inner product without explicitly mapping the vectors.

For a kernel function to exist it needs to have the following atributes:

It needs to be symmetric
It needs to be positive-definite

That is sufficient and necessary to a function $\kappa(\mathbf{x,y})$ to be considered a inner product in an arbitrary vector space $\mathcal{H}$.

As the covariance, comply to this definition it is a Kernel Function and consequentially it is an Inner Product in a Vector Space.


Keras Callback example for saving a model after every epoch?

Setting 'save_weights_only' to False in the Keras callback 'ModelCheckpoint' will save the full model; this example taken from the link above will save a full model every epoch, regardless of performance:

keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)

Some more examples are found here, including saving only improved models and loading the saved models.


Trying to figure out which the training set is

To clarify the proposed classifier (because I think it depends on some nonstandard notation): classify any point that is in the (finite) training set according to its true label.  (Clearly this gives you an error rate of 0 on the training set.)  For any point not in the training set, just predict label 0.
Now, consider a random new point in the gray square: it's the same as a point from the training set with probability 0, so assume it's not.  Then our classifier predicts label 0.  But the point has probability 1/2 (area of blue square over area of gray square) of being label 1.  So the error rate of our classifier, on test data, is 1/2.


StratifiedKFold: ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead

I had the same problem, you can find my detailed answer here.

Basically, KFold does not recognize your target as multi-class because it relies on these definitions:

  'binary': y contains <= 2 discrete values and is 1d or a column
  vector.
  'multiclass': y contains more than two discrete values, is not a
  sequence of sequences, and is 1d or a column vector.
  'multiclass-multioutput': y is a 2d array that contains more
  than two discrete values, is not a sequence of sequences, and both
  dimensions are of size > 1.
  'multilabel-indicator': y is a label indicator matrix, an array
  of two dimensions with at least two columns, and at most 2 unique
  values.


Understanding The Vertical and Horizontal stack in conditional gated Pixelcnn paper

I also had the same problem when reading the paper, I believe it is not explicitly mentioned.
I think the whole image is padded with zeros on the top row when you are using the vertical stack. In this way, you ensure that the output of any pixel of row i will depend only on the pixels at row i-1 till row 0. But at the same time, you have the advantage of depth of field compositionality and remove the blind spot (given you are stacking enough convolutional layers).


Why is the logistic regression decision boundary linear in X?

The decision boundary $p(X)=a, a\in(0,1)$ is

$$\beta_0+\beta_1X=\log\frac{a}{1-a}$$

which is linear in $X$.


How do I convert a pandas dataframe to a 1d array?

You can try this

import pandas as pd
import numpy as np
filename = 'data.csv'
df1 = pd.read_csv(filename)
#convert dataframe to matrix
conv_arr= df1.values

#split matrix into 3 columns each into 1d array

arr1 = np.delete(conv_arr,[1,2],axis=1)
arr2 = np.delete(conv_arr,[0,2],axis=1)
arr3 = np.delete(conv_arr,[0,1],axis=1)

#converting into 1D array
arr1 = arr1.ravel()
arr2 = arr2.ravel()
arr3 = arr3.ravel()

This should mostly do the job. Use the arr1 ,arr2,arr3 in the function you mentioned. They are the 1d array of the columns you split


How to approach mapping families of vectors on a lattice and forecast resulting value

It really seems like a problem of formulating your probkem. On your 2D exemple you can split your $X_i$ along your axes and juxtapose them to build a data set. Each $X_i$ has two components : along axis 1 ($X_{i1}$) and 2 ($X_{i2}$).

Basically you would just build your data set by juxtaposing the 18 columns :

$X_{11}$ , $X_{12}$ , ..., $X_{i1}$ , $X_{i2}$ ,..., $X_{91}$ ,$X_{92}$

To try to predict your y.

You can augment the dimension of you problem by similar juxstaposition of your features.
It would get a bit more complex if you augment the dimension of y, as you would need multiple models.

Trying to predict $y_{t+1}$ would lead to other considerations, like considering $y_t$ as a feature of your model.


custom tensorflow layer that inherits another class

Although I'm not an export of RNN's, the following should work given that RNN is inherited from Layer:

from keras.layers import RNN
class custom_LSTM(RNN):
   # implementation of class
   ...

It might be good to provide the smallest piece of code that can reproduce your error.


Is generalizing a model, then removing the generalization good for FFNNs?

L1 and L2 regularizations matter only during training, they are a way to update the Network's weights in (hopefully) the right way. Once you use your model for prediction that doesn't matter anymore.

Dropout is active only during training. Once it's done, the Network usese all the trained nodes to make a prediction.

In other words, no need to remove regularization techniques manually.


Automomatic typographical error correction

I will reference two sources:

Wikipedia has its own lists: Lists of common misspellings
Aspell is certainly a place to visit. You can find a list of misspellings and corrections here.

Still, it is a good idea to think of using a spell checker, such as Aspell, since it gives you the ability to deal with no-so-common misspellings (of course, at the cost of false positives). A good starting point is Peter Norvig's essay on creating a spelling corrector.


Using Machine Learning to create marketing segments

Wow, what a great question! This is a critical task for anyone who works with data in a marketing function.

First, I'll address your question about pairing the prediction to email. Your email is serving as a unique identifier. When you split your data set, you'll have to keep the email and suppress it from the prediction, and use it to match/compare to the validation data.

If these are new predictions, you could create a new column in your data that contains the "predicted" value. Specifics on this process would be better suited for another community, probably StackOverflow.

Second, I'd like to talk about what you're trying to achieve.

I think there are two questions ways you could approach this with the work you've already done:

Identify who is most likely to subscribe, and focus your efforts on creating nurturing campaigns to "warm" these prospects. The final goal is to get a subscription.

Here you're going to be using those "events" that you mentioned. What exactly do you have? in the past. I've just dumped web activity data and started working with that. You'll have to engineer a lot of features to capture information that your web activity won't report on (these could be # of logins, # of checkouts, time since last visit, this is where you need to get creative).

Take this data and try to make some predicte who is likely to subscribe. In some sales industries, this is refereed to as a lead score.

Since you're using R, you could try using XGBoost -https://cran.r-project.org/web/packages/xgboost/vignettes/xgboostPresentation.html. Since you're trying to predict a binary variable, you could use their decision tree.

This is going to take a while, so don't get frustrated if you don't this to production in the first week (or month). Here's some reading:

https://datastories.com/gallery/predictive-user-scoring-example

https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/

https://rdatascientist.wordpress.com/2015/01/26/predictive-lead-scoring-using-r-first-of-a-two-part-blog-series/
Identify which types of customers you want to acquire, and focus on building campaigns to engage these types of customers. The final goal here is to acquire quality customers.

This is more marketing and strategizing than it is machine learning. Don't get me wrong, there is maching learning in identifying the clusters. Here's a great example I used: http://www.kimberlycoffey.com/blog/2016/8/k-means-clustering-for-customer-segmentation.

Once you identify what the different segments and what behavior they exhibit, you have to team up with the other stakeholders and try to decide what kind of customers do you want? Whats the benefit is getting a customer who makes one large purchase a year vs. a customer who makes five smaller purchases a year? You'll have to take your time identifying customers/prospects who are the most valuable, and plan a strategy around that.


How tensorflow understand accuracy and loss of training data?

You are still able to calculate metrics such as loss and accuracy on training data (or any data for that matter), however the important thing to keep in mind is that it is by definition training data. Therefore the metrics from the training data are not how you would expect the model to perform on new unseen data as the model has been training on these specific samples (and can possibly remember them).


Early stopping on validation loss or on accuracy?

TLDR; Monitor the loss rather than the accuracy

I will answer my own question since I think that the answers received missed the point and someone might have the same problem one day.

First, let me quickly clarify that using early stopping is perfectly normal when training neural networks (see the relevant sections in Goodfellow et al's Deep Learning book, most DL papers, and the documentation for keras' EarlyStopping callback).

Now, regarding the quantity to monitor: prefer the loss to the accuracy. Why?
The loss quantify how certain the model is about a prediction (basically having a value close to 1 in the right class and close to 0 in the other classes). The accuracy merely account for the number of correct predictions. Similarly, any metrics using hard predictions rather than probabilities have the same problem.

Obviously, whatever metrics you end up choosing, it has to be calculated on a validation set and not a training set (otherwise, you are completely missing the point of using EarlyStopping in the first place)


Yes. It is also a function, but not an affine transformation of the input but a relatively complex sum of products of indicator functions of the input. Usually, this function is represented by the fitted tree and not as a formula.

So e.g. if you learn a tree of depth one and the split is at age 40 with mean response of 80 if age < 40 and mean response of 100 if age $\ge$ 40, then the function could look like
$$
  \hat f(\text{age}, \text{income}) = 80 \cdot {\mathbf 1}\{\text{age} < 40\} + 100 \cdot {\mathbf 1}\{\text{age} \ge 40\}
$$
You can maybe imagine how long the formula is if the depth is 7...


First idea is to use, instead of an invalid data point, the last valid value. This is done in finance where some stocks are not quoted everyday (and when you want to publish the value of your fund you use the last available value for the stocks).

However, this can be an issue when you have too many missing values (especially for prediction).

You can try Bayesian data augmentation (works somewhat well with time series) but if you say that you have a lot of missing data you have to accept that anything that you do on rather poor quality data will be of limited quality.


How does the Nearest Centroid method work?

You are making a mistake regarding what is given: during training, you don't have a radius $R$. You have the coordinates $\vec{x}$ and the label $y$ for each point:
$$ \vec{x}_1 = [-2;1] \quad \vec{x}_2 = [1;2] \quad \vec{x}_3 = [0;-1] \quad \vec{x}_4 = [1;0] \quad \vec{x}_5 = [1;1] $$
and
$$ y_1 = T \quad y_2=T \quad y_3=T \quad y_4=S \quad y_5=S $$
with that, your "trained" centroids are
$$ \mu_T = \frac 13 [-2 + 1 + 0; 1 + 2 - 1] = \left[-\frac 13, \frac 23\right] $$
$$ \mu_S = \frac 12 [1 + 1; 0 + 1] =\left[1, \frac 12\right]$$
You calculate these centroids before you get any test values.

Then, for testing, for your observed point $\vec{x} = [1,1]$, you calculate the Euclidean distance between the point $\vec{x}$ and the centroids $\mu_T$ and $\mu_S$:
$$ \| \vec{x} - \mu_T \| = \left\| [1;1] - \left[-\frac 13; \frac 23\right] \right\| = \left\| \left[\frac 43; \frac 13\right] \right\| = 1.37 $$
$$ \| \vec{x} - \mu_S \| = \left\| [1;1] - \left[1; \frac 12\right] \right\| = \left\| \left[0; \frac 12 \right] \right\| = 0.5$$
Finally, the term $\hat{y} = \arg\min_{l \in \mathbf{Y}} \|\vec{x}-\mu_l\|$ is used to find the estimated class $\hat{y}$ (the hat symbol is to denote that this is an estimated $y$, not one we knew before.). $\arg\min$, means that you find the minimum value - which is 0.5 in our case, and chose which "argument", i.e. which class, leads to that minimum value. In our case, the class which leads to the minimal distance is $S$, so the result is $\hat{y} = S$, and our test point is a square.


Which is a better method Lasso regression or sequential Foward selection?

I think there is no obvious advantage of one algorithm over another: I'd suggest running cross-validation to investigate which works best for the data at hand. You've suggested forward stepwise and Lasso, which both performs feature selection, you could also investigate backward stepwise and best-subset selection. Linear regression models are fast to train, so maybe you could investigate the cross-validation error for all 4 algorithms.

Hope this helps.


Tools and protocol for reproducible data science using Python

The topic of reproducible research (RR) is very popular today and, consequently, is huge, but I hope that my answer will be comprehensive enough as an answer and will provide enough information for further research, should you decide to do so.

While Python-specific tools for RR certainly exist out there, I think it makes more sense to focus on more universal tools (you never know for sure what programming languages and computing environments you will be working with in the future). Having said that, let's take a look what tools are available per your list.

1) Tools for data version control. Unless you plan to work with (very) big data, I guess, it would make sense to use the same git, which you use for source code version control. The infrastructure is already there. Even if your files are binary and big, this advice might be helpful: https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git.

2) Tools for managing RR workflows and experiments. Here's a list of most popular tools in this category, to the best of my knowledge (in the descending order of popularity):

Taverna Workflow Management System (http://www.taverna.org.uk) - very solid, if a little too complex, set of tools. The major tool is a Java-based desktop software. However, it is compatible with online workflow repository portal myExperiment (http://www.myexperiment.org), where user can store and share their RR workflows. Web-based RR portal, fully compatible with Taverna is called Taverna Online, but it is being developed and maintained by totally different organization in Russia (referred there to as OnlineHPC: http://onlinehpc.com).
The Kepler Project (https://kepler-project.org)
VisTrails (http://vistrails.org)
Madagascar (http://www.reproducibility.org)

EXAMPLE. Here's an interesting article on scientific workflows with an example of the real workflow design and data analysis, based on using Kepler and myExperiment projects: http://f1000research.com/articles/3-110/v1.

There are many RR tools that implement literate programming paradigm, exemplified by LaTeX software family. Tools that help in report generation and presentation is also a large category, where Sweave and knitr are probably the most well-known ones. Sweave is a tool, focused on R, but it can be integrated with Python-based projects, albeit with some additional effort (https://stackoverflow.com/questions/2161152/sweave-for-python). I think that knitr might be a better option, as it's modern, has extensive support by popular tools (such as RStudio) and is language-neutral (http://yihui.name/knitr/demo/engines).

3) Protocol and suggested directory structure. If I understood correctly what you implied by using term protocol (workflow), generally I think that standard RR data analysis workflow consists of the following sequential phases: data collection => data preparation (cleaning, transformation, merging, sampling) => data analysis => presentation of results (generating reports and/or presentations). Nevertheless, every workflow is project-specific and, thus, some specific tasks might require adding additional steps.

For sample directory structure, you may take a look at documentation for R package ProjectTemplate (http://projecttemplate.net), as an attempt to automate data analysis workflows and projects:

4) Automated build/run tools. Since my answer is focused on universal (language-neutral) RR tools, the most popular tools is make. Read the following article for some reasons to use make as the preferred RR workflow automation tool: http://bost.ocks.org/mike/make. Certainly, there are other similar tools, which either improve some aspects of make, or add some additional features. For example: ant (officially, Apache Ant: http://ant.apache.org), Maven ("next generation ant": http://maven.apache.org), rake (https://github.com/ruby/rake), Makepp (http://makepp.sourceforge.net). For a comprehensive list of such tools, see Wikipedia: http://en.wikipedia.org/wiki/List_of_build_automation_software.


How does a CBoW model convert a word to a vector?

CBoW (Continuous bag-of-words) is a theoretical architecture, not exactly a saved model or a library like gensim. Gensim might be an implementation of CBoW to which you feed a one-hot vector and get your word vector output.
CBoW as a theoretical model gives 'representational meaning' to a word based on the 'representational meaning' of the word surrounding it. 'Representational meaning' may just be a fancy term for word vectors. So basically the model builds word vectors out of surrounding word vectors possibly using negative sampling and the NCE loss, but the main point is that it fine-tunes these vectors until their 'representational meaning' becomes well-honed.
These well honed vectors are then probably callable from gensim through one-hot inputs.


LSTM for time series forecasting with H20.ai

Active development on H2O's Deep Water project has stopped, see here: https://github.com/h2oai/deepwater

I don't think LSTM was included in the last release, and I would use Keras instead.

A list of Jupyter notebook demos is found here as well: https://github.com/h2oai/h2o-3/tree/master/examples/deeplearning/notebooks


How to assess nested cross validation results in comparison to non-nested results?

That is not a regression problem. Thus those results can not be interpreted.
It should be framed as probabilistic binary classification. The target is binary because there are two outcomes - disease or not. Probabilistic because it on a scale from 0 (not possible) to 1 (completely certain).


DBPedia as Table not having all the properties

The official answer:

Date: Sun, 23 Nov 2014 03:08:19 +0100
From: Petar Ristoski <petar.ristoski@informatik.uni-mannheim.de>
To: 'Barry Carter' <carter.barry@gmail.com>
Subject: RE: CSV tables don't have all properties?

Hi Carter,

The question was already answered on the DBpedia mailing list, but I will
try to clarify it again. On the DBpedia as Tables web page says that "For
each class in the DBpedia ontology (such as Person, Radio Station, Ice
Hockey Player, or Band) we provide a single CSV/JSON file which contains all
instances of this class. Each instance is described by its URI, an English
label and a short abstract, the MAPPING-BASED INFOBOX data describing the
instance (extracted from the English edition of Wikipedia), geo-coordinates,
and external links." As you can see we only provide the mapping-based
infobox properties (dbpedia-owl namespace), while the properties from the
dbpprop (raw infobox properties) namespace are completely ignored.
Therefore, dbpprop:bestBowling is missing from the file.

Also, there is a section "Generating your own Custom Tables" [1], where we
explain how to generate your own tables that will contain the properties you
need.

Regards,

Petar

[1] http://wiki.dbpedia.org/DBpediaAsTables#h347-4


How to visualise multidimensional categorical data with additional time dimension

This is something you can get in Tableau in 10 minutes.

What you need is year binning, skills weights and length(year_end - year_start) for Gannt chart:


Difference between rulefit & skope-rules Python Packages

In the Skope rules poster: http://2018.ds3-datascience-polytechnique.fr/wp-content/uploads/2018/06/DS3-309.pdf
They made themselves a differentiation

However, our approach mainly differs in the way that decision rules are chosen: semantic deduplication based on variables composing each rule as opposed to L1-based feature selection (RuleFit).

In my personal opinion and after seeing the semantic deduplication package, I will be quite careful as it made some strong modifications of logical rules by default.


Like most function in r, droplevels operates on your data but does not change it. The droplevels function returns a factor vector with the levels dropped, but it does not change the original vector. To do that, you need to assign the result of droplevels back to the original original vector. Here is an example:

# Building an example data.frame
set.seed(1234)
rating <- factor(as.character(sample(1:5, 100, replace = T)))
label <- sample(LETTERS, 100, replace = T)
example <- data.frame(label = label, Rat = rating)
summary(example$Rat)
  1  2  3  4  5
 25 26 17 17 15

# Subsetting the data
example234 <- subset(example, Rat != "1")
example234 <- subset(example234, Rat != "5")
summary(example234$Rat)
  1  2  3  4  5
  0 26 17 17  0

# Dropping unused levels
droplevels(example234$Rat)
  [1] 4 4 4 4 2 4 3 4 3 2 2 2 2 2 2 2 2 3 3 2 2 3 4 2 2 3 4 2 4 2 3 4 3 2 4 2 4
  [38] 3 3 3 4 2 2 4 2 3 3 4 3 2 2 4 3 3 2 2 3 2 2 4
  Levels: 2 3 4
summary(example234$Rat)
  1  2  3  4  5
  0 26 17 17  0

# Reassigning the data with dropped levels
example234$Rat <- droplevels(example234$Rat)
summary(example234$Rat)
  2  3  4
 26 17 17

Notice in my example, like your attempt, if you just run droplevels on the data, it does not change the underlying data. However, if you assign the results of droplevels(example234$Rat) to example234$Rat, your underlying data has been properly changed.


When does cache get expired for a RDD in pyspark?

It will not expire until Spark is out of memory, at which point it will remove RDDs from cache which are used least often. When you ask for something that has been uncached it will recalculate the pipeline and put it in cache again. If this would be too expensive, unpersist other RDDs, don't cache them in the first place or persist them on your file system.


Understanding `get_combination_wise_output_matrix` when investigating a multi-label classification problem

So order here means how many possible combinations of labels you want to compare (e.g., order=1 would by how often does each label appear, order=2 would be how often any two combinations of labels appear with values like 5,5 meaning "rows that only have a label in index 5 and no other label, and where the max order should be the number of labels you have --> 23 in your case. If you set order=23 and if you had a row that only had a label in index position 4, one of our results would be a vector of 4 repeated 23 times.

I think what's weird for me is that I would expect to see the "only one label in index 4" presented as [0, 0, 0, 0, 1, 0, ,,, 0, 0] (== len 24) instead of [4, 4, 4, ... 4, 4] (== len 24).

Odd presentation for sure. Kind of like sparse matrix notation.