What is the 1 Unit in the contraint of SVM: $y_i(wx_i+b) \geq1$

It is a unit of distance, I would usually assume euclidean distance.

In more detail:

The data point $x_i$ is projected onto the vector $w$, which defines the orientation of the discriminating linear hyperplane as it is orthogonal to $w$. Where the discriminating hyperplane is "fixed" along the orientation of $w$ is decided by the bias Term $b$.

So for classifying a positive data point $x_{+}$ we would want $wx_{+} + b$ to a positive number.

For a negative $x_{-}$ datapoint $wx_{-}+b$ should be negative.

We can combine these two disired outcomes into a single constraint by using the labels, so we want $y_{+}(wx_{+}+b) \geq 0$ and $y_{-}(wx_{-}+b) \geq 0$.

By replacing the $0$ with a positive value $d \geq 0$, we can alter the constraint to encode our desire that the points should be $d$ far away from from the discriminating hyperplane, giving the constraint is $y_i(wx_i +b) \geq d$.

What kind of unit of distance depends on the vector space. Usually orthogonal projections in euclidean vector spaces are used, in that case it would be euclidean distance.


What to replace outliers with? (supermarket transaction data)

Simplest approach
First a very fast and perhaps practical approach: just remove them without replacing them!
From your bar chart, it seems you have a lot of transactions - several hundred thousand. Removing a few hundred (I can't even see a bar for the > $600 transactions) and not replacing them wouldn't mean the remaining data is unusable. Replacing those transaction e.g. using the mean of the distribution, is basically trying to make sure they don't have any significant effect on the model; so what is the point in replacing them?
Looking at your frequency chart, I would be interested to know what appears in the final large peak. Could you just remove those transaction?
You could give this a go and move onto the modelling, see if you get sensible results - if not, come back to investigate further.

Back to statistical methods
Using 3 standard deviations isn't a bad approach - assuming your data is normally distributed, it means you only remove 0.3% of the data. The issue is perhaps that your distribution of transaction amounts does not look normally distributed - it looks more like a beta distribution(the orange line):

You could therefore try your approach of removing data outside the distribution, by computing the parameters of the beta distribution, and using those instead of the normal mean/std.
To do this, check out scipy.stats.beta, and be aware that you should probably normalise your transactions to the range [0, 1.0] (see Scikit-Learn's StandardScaler). Here is a thread on understanding the output of your beta distribution.

Impute
If you simple treat your outliers as missing data, there are some nice ideas and explanations about filling missing gaps in your data (a.k.a. data imputation) in this well-known book: Elements of Statistical Learning (pdf)$^1$ - see section 9.6.
$^1$Authors: Trevor Hastie, Robert Tibshirani, Jerome Friedman


You can use the ELI5 library to explain the feature contributions to individual predictions for XGBoost models.

See Explaining Predictions in the docs, copied below:

  To get a better idea of how our classifier works, let’s examine
  individual predictions with eli5.show_prediction():

from eli5 import show_prediction
show_prediction(clf, valid_xs[1], vec=vec, show_feature_values=True)


If your definition of points belonging to a cluster is simply the points closest the cluster centroid, then the boundaries cannot overlap. The point assignments are a Voronoi map like:

(Source: https://www.quora.com/What-is-the-difference-between-K-Means-and-Voronoi/answer/Ethan-Brooks-3)

The closest centroid, and thus assignment, is unambiguous.


How exactly do I extract the important features from strings for machine learning?

(1) How exactly do I go about extracting information from the Ad topic line?

The best way to deal with the test string is to Convert it into a term-document matrix and calculate the TFIDF Scores for every word in the string.

(2) How should I approach the categorical variables Country and City? ( I've heard about encoding but won't a large number of countries/cities be a problem?)

Yes you said the right large number of cities will be a problem i suggest still you can go with that, Rather there is a way find out the distribution of all your cities and cities and convert them into integers.
Even another way could be give weights to the cities which are repeated most,let me give you an example:

Suppose you have 10 observations and 5 cities A(5 times),B(2 times),C(once),D(Once),E(Once):

so A will get a weight of 5/10=0.5 which B will get a weight of 2/10=0.2 and so on.

Hope this helps!


Explainable AI and unsupervised algorithms

PCA isn't a great example, as it's inherently explainable, in the same way that linear regression is inherently explainable.

In supervised learning, at a high level, the Shapley value for each feature tells you the extent to which including that feature in an ML model allows you to make better predictions about the target variable than if you did not have access to that feature. One way of implementing this is to calculate the difference in test cross entropy between a model which includes that feature and one which does not (and it's much more complicated than that in reality, as there are many models which do not include that feature, you have to average over all coalitions of other features, but that's another story).

So in short, you need some sort of cost function, which including features tends to improve.

Now let's turn our example to the simplest of all clustering algorithms, KMeans. In KMeans, the most sensible cost function is probably to split your data into train and test, fit on a training set, and then take the sum over your test set, of each point's euclidean distance from its nearest centroid.

Now, what happens if you remove a feature? Well, in general, post fitting, the sum described above will run over fewer features, so the cost will be lower. This in turn will mean that every feature has a negative Shapley value...not good

More generally, in unsupervised learning, the cost function tends to be related to the probability of seeing a datapoint, according to the model, i.e. the likelihood. The key however, is that this is not just the likelihood of the target, it's the entire, multidimensional likelihood of all features (indeed, there is no target in unsupervised learning).

What you would need, in order to calculate Shapley values for unsupervised learning problems, is some way to describe "how good the clustering is" with one number. The "goodness of clustering" would need to be comparable when you changed the dimensionality of the problem...and standard cost functions used in inferential clustering (which KMeans is kind of a simplified form), do not satisfy this.

So I've only answered a subset of your problem, I've said that Shapley values don't really make sense for inferential clustering problems, rather than "explainability doesn't make sense for unsupervised algorithms". I don't know enough about all unsupervised algorithms to claim Shapley values don't make sense for any unsupervised algorithm, but it certainly seems to me to be the case, that you'll need an algorithm which lends itself to determining for any coalition of features included in the clustering, "how good the clustering is", in a way which somehow "objectively measures whether similar instances are grouped", rather than which calculates full data likelihood in a way which is inversely proportional to the number of features in the model. This seems to go against the idea of unsupervised learning, namely that there's no objective truth.


ML technique to predict next year output based on text quantities

I'm not sure machine learning is the right method to count the number of arguments in a list. You can just write a function that does that and it's all you need...

Now if you want to predict the number of total components, assuming it holds some kind of temporal relation, there are a few techniques possible. But remember: Garbage in, Garbage out...

The easiest is to build a regression model by using past numbers as features, this gives an equation like this:

$$Y_t = a_1 \times Y_{t-1} + a_2 \times Y_{t-2} + ... + b$$

Next you can look at time series (ARIMA, SARIMA), which are a bit more advanced: additional to the time steps, they can take moving average and seasonal components into account.

And finally, you can also look at neural networks. If you have enough, and more importantly meaningful data, you can look at multilayer perceptron with a time window, 1d-convolutions and RNN/LSTM... Plenty resources exist if you want to go with this.


Deep learning technique to recognize a person's silhouette

For this purpose you have to have a data-set that can be interpreted by the human. I mean an expert should label the data samples without hesitating. Then you can make a typical neural network and train it. In your case you have 10 classes and all 60 thousand images are labeled by an expert. Consequently, you are able to train a network. Your task is supervised and you need labeled data-set.


Categorical and ordinal feature data representation in regression analysis?

The distinction between ordinal and categorical does matter. If in truth the difference between white and red was drastically different from red and black, your (10,20,30) ordinal model would not have performed well.

One hot encoding can learn the relationship between the ordinal values more finely, but throws out the information that the variables are related. Similarly, with insufficient data it is more likely to overfit.

Ordinal variables lessen those  problems but at the cost of forcing you to define the interval. There are a number of methods for defining the values of your ordinal variables, like rologit.


TensorSpec is mostly used by tf.function to specify input signature. tf.function will create a graph for different input shapes and datatypes, but it is possible that your function graph is compatible with different shapes. As a performance optimization, You can optionally provide a signature so that no unnecessary graphs are created.


The name of the model you are looking for is: Constrained Estimation of Ordinary Least Squares, this model allows you to introduce linear constrains to your estimation by just modifying the matrix expression of the OLS estimation.

The resulting model should satisfy the expression

$$Q^T\beta = c$$

It is solved with the equation:
$$\hat{\beta^c} = \hat{\beta}-(X^TX)^{-1}Q(Q^T(X^TX)^{-1}Q)^{-1}(Q^T\hat{\beta}-c)$$

For your problem, $Q=[1,1,1]$, $c=1$, $X$ and $Y$ are the matrixes for your data.


what is difference between Logistic regression and SGDClassifier with log loss OR SVM and SGDClassifer with hinge loss?

From a practical point view, yes you could use both options seamlesly with in general similar results (check this scikit learn functionality), but:

SGD (Stochastic Gradient Descent) is an optimization algorithm among others
log loss/hinge loss... are the loss functions used in the selected optimization strategy (SGD in your case) to find the optimal weights to fit such linear models, but another optimization solvers  could be used with those algorithms. See this post for logistic regression with sklearn, where you have the solver parameter, which can be, among others, 'sag' (stochastic average gradient descent), but others like 'lfbs' are also accepted.


The exact meaning of cost function ? (Machine Learning)

The cost function is the judge for your model. It judges how well your model perfoms. By choosing a loss function you choose which properties of your model outputs the loss function will judge. Mathematical convenience usually is desired for the loss function to be applicable

The MSE will punish outputs that are further away from the desired value more severely than those that are closer because of the quadratic. Therefore, the outputs that are the furthest away from the desired value impact the cost function and thus the optimization greatly. If your dataset has many outliers, then these can influence your cost function a lot. In this case you could use techniques like dimensionality reduction or just choose another loss function like the $L_1$ loss, which is more robust to outliers. Furthermore, if you choose MSE you implicitly assume that the noise of your data is guassianly distributed, since the MSE loss will minimize the crossentropy between the empirical distribution of your outputs and the gaussian distribution. See https://stats.stackexchange.com/questions/288451/why-is-mean-squared-error-the-cross-entropy-between-the-empirical-distribution-a

The logistic regression cost function will judge the outputs of your model by other characteristics. The logistic regression intends to classify your outputs correctly into two classes. The output of your model lies between $[0, 1]$ the respective classes. Once outputs lie on the correct side of the decision boundary their impact on the lossfunction decrease rapidly with the distance to the decision boundary. Look at the logistic curve and observe where points are closest to the classes ($0$ or $1$) and then check out where the loss function is the steepest for each class. So the cost function of the logistic regression will punish those outputs severly that are missclassified, and those, although a bit less, that are correctly classified but lie very close to the decision boundary. As a result it is a decent loss function for binary classification. There are probably a few further implicit assumptions with logistic regression, however I have not dug into that in the past to be able to say what they are.

Since one usually assumes that the data is i.i.d (identically, idependently distributed: each datapoint is to be treated equally), the loss for each output is summed up.

In summary: Loss functions are chosen to judge the desired properties of the model outputs, and usually exhibit favorable mathematical qualities for the optimization. By choosing the loss function you can alter the criteria your optimization should follow and often times thus introduce prior knowledge.


Best way to handle padding in time series data such as text

Time series Analysis refers to the type of problems where we have to analyse an outcome based on time dependent inputs. Time series data is basically a sequence of data, hence time series problems are often referred to as sequence problems. Recurrent Neural Networks (RNN) have been proven to efficiently solve sequence problems. Particularly, Long Short Term Memory Network (LSTM), which is a variation of RNN, is currently being used in a variety of domains to solve sequence problems.

If we are trying to obtain more features, it is a good idea to utilise the idea of distributed representation of words, in which each input is represented by many features and each feature is involved in many possible inputs. In distributional semantics, word embedding models are used to map from the one-hot vector space to a continuous vector space in a much lower dimension than conventional bag-of-words model. we can learn the dependencies between characters and the conditional probabilities of characters in sequences so that we can in turn generate wholly new and original sequences of characters.

In such a perspective, Word2Vec word embedding model would be a good choice. It could be a good idea to convert the words into vectors using word embedding models. Then the word sequences in sentences can be feed as input to the LSTM to learn the long distance contextual dependency among the words.

Typically, sequence problems can be broadly classified into one-to-one, many-to-one, one-to-many, and many-to-many types. As per the descriptions, it seems the requirement is to analyse many sequences of textual letters as different many textual sequences itself. In these sequence to sequence analysis problems, you can either choose a single feature or with multiple features.

Long Short Term Memory (LSTM) neural networks are generally suitable for sequence prediction and sequence classification problems. If we are using LSTM for this problem, we need to model the text sequence accordingly. First we need to vectorise the text by turning each text into a sequence of integers or into a vector. It is important to limit the dataset to a top word limit. Also we need to have a maximum sequence length for each text.

Then we can tokenise the text applying this model. We can truncate and pad the input sequence such that they are all same length for modelling. The sequence will not be of same length in terms of content, but same length vectors need to be constructed.


Filling missing values for Embedded List in Python3

According to the suggestion, @bkshi gave to me I come up with a solution here below:

Also since texts_to_sequences() function convert my list to sequences starting from 1, I could use pad_sequence() and use 0 instead of a string value.

This solution satisfies my requirements so I used a number as padding instead of a string value.

import pandas as pd
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

my_list = [['the first line'],
       ['the 2nd line'],
       ['the 3r line'],
       ['the 4th line'],
       ['the 5th line'],
       ['the'],
       ['the 5th line, this is']]
max_features = 10 #how many unique words you're using

tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(my_list)
my_list = tokenizer.texts_to_sequences(my_list)
my_list = pad_sequences(train_X, maxlen=None, dtype='int32', padding='post', truncating='post', value=0.0)


Can a Gradient Boosting Regressor be tuned on a subset of the data and achieve the same result?

You should never train or do grid search on your entire data set, since it will lead to overfitting and reduce the accuracy of your model on new data. What you have described is actually the ideal approach: do grid search / training on a subset of your data. Yes, your model will get different results vs if you used the entire set of data, but your model will be much stronger because of it.

For more details on why you would want to split up / sample your data, see this quesiton: https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set


Multivariate Time series analysis: When is a CNN vs. LSTM appropriate?

Is it reasonable to use a CNN instead of an LSTM, even though it is a time series?
Yes, it is. Convolutional Neural Networks are applied to any kind of data in which neighboring information is supposedly relevant for the analysis of the data.
CNN are very popular with images, where data is correlated in space, and in video, where correlation happens both in space and time.
Are there any indicators for when you should never switch to a CNN?
CNNs are limited in a sense: they have a rigid, forward structure.
If you are trying to perform:

classification in sequences that have varying length ($N$ to $1$);
trying to output another sequence which has no fixed proportion between their length and the length of the input ($N$ to $M$).

Simple feed-forward neural networks will fail (due to dimension inconsistency). So, in that case you should use a recursive neural network such as a LSTM.


Which openCv function can be used to compute BEV perspective transformation given a point coordinates and the camera extrinsics/intrinsics?

After a deep investigation, I found out a good solution:
The projection matrix is a multiplication between theextrinsic and the intrinsic camera matrices

https://medium.com/analytics-vidhya/using-homography-for-pose-estimation-in-opencv-a7215f260fdd
Since the extrinsic is a 4x3 matrix and the intrinsec is a 3x3 matrix, but we need the projection matrix to be a 3x3 matrix, then we need to convert the extrinsic into 3x3 before performing the multiplication.

cv2.getPerspectiveTransform() gives us the Projection Matrix when we don't have the camera params:

https://towardsdatascience.com/a-hands-on-application-of-homography-ipm-18d9e47c152f

cv2.warpPerspective() transforms the image itsef.
For the problem above we don't need these two functions since we already have the extrinsics, the intrinsecs and the coordinates of the points in the image.
Considering the presented above, I wrote a function to transform into BEV a list o points list_x_y given the intrinsics and the extrinsics:
    def compute_point_perspective_transformation(intrinsics, extrinsics, point_x_y):
    """Auxiliary function to project a specific point to BEV

        Parameters
        ----------
        intrinsics (array)     : The camera intrinsics matrix
        extrinsics (array)     : The camera extrinsics matrix
        point_x_y (tuple[x,y]) : The coordinates of the point to be projected to BEV

        Returns
        ----------
        tuple[x,y] : the projection of the point
    """

        # Using the camera calibration for Bird Eye View
        intrinsics_matrix = np.array(intrinsics, dtype='float32')
        #In the intrinsics we have parameters such as focal length and the principal point

        extrinsics_matrix = np.array(extrinsics, dtype='float32')
        #The extrinsic matrix stores the position of the camera in global space
        #The 1st 3 columns represents the rotation matrix and the last is a translation vector
        extrinsics = extrinsics[:, [0, 1, 3]]

        #We removed the 3rd column of the extrinsics because it represents the z coordinate (0)
        projection_matrix = np.matmul(intrinsics_matrix, extrinsics_matrix)

        # Compute the new coordinates of our points - cv2.perspectiveTransform expects shape 3
        list_points_to_detect = np.array([[point_x_y]], dtype=np.float32)
        transformed_points = cv2.perspectiveTransform(list_points_to_detect, projection_matrix)
        return transformed_points[0][0][0], transformed_points[0][0][1]


What is the difference between bootstrapping and sampling in reinforcement learning?

I will try to answer this question conceptually and not technically so you get a grasp of the mechanisms in RL.

Bootstrapping: When you estimate something based on another estimation. In the case of Q-learning for example this is what is happening when you modify your current reward estimation $r_t$ by adding the correction term $\max_a' Q(s',a')$ which is the maximum of the action value over all actions of the next state. Essentially you are estimating your current action value Q by using an estimation of the future Q. Neil has answered that in detail here.
Sampling: Imagine samples as realizations (different values) of a function. Many times it is really difficult to estimate, or come up with an analytical expression, of the underlying process that generated your observations. However sampling values can help you determine lots of characteristics of the underlying generative mechanism and even make assumptions of its properties. Sampling can come in many forms in RL. For example, Q learning is a sampled-based point estimation of the optimal action value function (Bellman equation). In a world that your agent knows nothing you cannot use Dynamic Programming to determine the expected reward form every state. Thus, you need to sample "experience" from your world and estimate the expected reward from any state. In Policy Gradient methods in order to determine the gradient of your expected reward you need to sample trajectories over states and actions from a probability distribution as you cannot determine it analytically.

Hope this helps!


What are the possible ways to detect skin while classifying diseases?

You have to have sample tissues from skins with Melanoma, Psoriasis and Healthy Skins.

As is understood in the question, you developed a solution which can distinguish with 96% accuracy between melanoma and psoriasis. You are missing from the possibility of distinguish between sick and healthy skins.

The model you are looking for is a multiclass classification. Where CLASS3: Healthy skin.

You are doing ok so far, you just need to include this category and images of healthy skins in your model.


Good books or resources for better understanding mathematical notation used in Machine Learning / Deep Learning Papers

For a pragmatic introduction to Deep Learning with many Tensorflow code examples I'd recommend
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition
or
Deep Learning with Python.
The latter is from 2017 and therefore a bit outdated but a new edition is expected to be published later this year (the author has just finished a draft version).
Both books explain, support or define all concepts with code examples. While there is no literal translation from math to code, I do think they provide the required understanding by directly using code as a notation.


You can consider string similarity based on edit distance. eg: Levenshtein distance

However, you also need to compute this with respect to prefixes of the strings, and not only the full strings. (eg: to compare "Facebook" with "Facebook Inc."). Also normalize the capitalization of the strings before comparison (make them all lowercase, etc.), and maybe remove punctuation as well.

Experiment with a few of the above, see if the results make sense, and fine tune the method accordingly. Also using other metadata will help. URL and context/topic words/entities occurring around the company name will be associated with the company, and can help find the right groupings. This will be more involved though, so you may want to attempt this later on, if the simpler approaches do not suffice.


Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative?

An imbalance of this magnitude is definitely a problem. Because machine learning algorithms penalize model complexity and try to maximize fit, a model that always predicts "False" will often outperform any other model because it predicts the right outcome for 99% of the data and is super simple!

What one should do in this case is downsample (http://www.simafore.com/blog/handling-unbalanced-data-machine-learning-models), train the model, and then scale the resulting predictions back up to reflect the imbalance in the original data.


What is a good way to start Data Analysis of unknown dataset (JSON data)

@Emre makes a great point. You should probably ask the business what they're trying to accomplish because only then will you know which data matters. You don't want to waste time cleaning up data that has minimal importance to the business.

Regardless of the business objective though, it's worth pointing out that Mongo DB is a pretty poor choice for storing analytical data. There is no enforced schema, so each record might be in a completely different and unexpected format. You might have entire columns missing, maybe no columns at all, wrong data types, duplicates, and no way to get easy summary statistics like you would in a relational database. You're pretty much going to have to do a database dump of your MongoDB data and probably comb through your records semi-manually, looking for commonly occurring schemas and going from there.

Summary: MongoDB is great for operational data when all you want to do a quick lookup. It's a poor choice when you want to do any sort of in depth analysis on it though. Relational databases are better for analytical queries because the data is structured and also because you can more easily enforce data quality.


How does the seed value work in Weka for clustering?

Pseudo-random number generators are initialized with a seed.

There is little real randomness available. Most just looks random to us, because the pattern is too complex. Read up some examples on pseudo random number generation such ad Mersenne Twisters.


There is an application of tf-idf on the sklearn website.

sklearn handles sparse matrices for you, so I wouldn't worry about it too much:

  Fortunately, most values in X will be zeros since for a given document less than a couple thousands of distinct words will be used. For this reason we say that bags of words are typically high-dimensional sparse datasets. We can save a lot of memory by only storing the non-zero parts of the feature vectors in memory.
  scipy.sparse matrices are data structures that do exactly this, and scikit-learn has built-in support for these structures.

Regarding your point about inserting the weight, I guess you have already performed tf-idf on your training corpus, but you don't know how to apply it to your test corpus? If so you could so as follows (taken from above link)

from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import CountVectorizer

count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(data)

# Perform tf-idf
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

docs_new = ['God is love', 'OpenGL on the GPU is fast'] # New test documents
X_test_counts = count_vect.transform(docs_new) # Count vectorise the new documents
X_test_tfidf = tfidf_transformer.transform(X_new_counts) # transfrom the test counts


Training stateful LSTM with different number of sequences

In the circumstance of stock market prediction, I think that when the sequence length reaches a certain point, the network will learn to forget the opening price 6 months ago or the volume 3 years back. That data is no longer relevant to the cell state, as recent events are more indicative of stock price changes.In terms of the LSTM method of storing both long and short-term memory, the LSTM will have the functionality to "forget" old and irrelevant data from earlier in the sequence, maintaining a stable and efficient cell state regardless of sequence length (of course, the sequence should be long enough for the LSTM to build up any cell state at all).
Hope this helps.


This is more of question how to select the correct machine learning algorithm, I would refer you to the following blog Which machine learning algorithm should I use?

Regression Algorithms models the relationship between variables that is
iteratively refined using a measure of error in the predictions. Most popular examples are:

Ordinary Least Squares Regression (OLSR)
Linear Regression
Logistic Regression
etc  ...

On the other hand, Artificial Neural Networks models are inspired by the structure and/or function of biological neural networks. "Neural networks currently provide the best solutions to many problems in image recognition, speech recognition, and natural language processing." Neural Networks and Deep Learning/.  Neural Networks are hard to train; thus my recommentation not to start with Neutral Network.


How to add attention mechanism to my sequence-to-sequence architecture in Keras?

By default an LSTM model returns only the output of last timestep.
model = Sequential()
model.add(LSTM(hidden_nodes, input_shape=(n_timesteps, n_features)))
##output shape is (n_features)

So the below step is needed to repeat the output vector 'n' number of times where 'n' should be the number of time-steps
model.add(RepeatVector(n_timesteps))
##now shape becomes (n_timesteps,n_features)

But when you specify 'return_sequences=True', LSTM returns a hidden state for ALL timesteps. The output shape from the LSTM directly is (n_timesteps,n_features). So you DONT need to to a 'Repeat Vector'
So to eliminate the error, just remove line 4
Edit - I would recommend using the 'return_sequences=true' option and NOT using Repeatvector option even though the latter may compile. This will lead to better results as you are passing far more data across timesteps to the next layer and this is the accepted approach for most situations


Naive bayes expectation maximization vs logistic regression for binary classification

On a very high level -
Naive Bayes is a probabilistic model and it is scale invariant. That means scaling the data won't affect your model's performance. It is mostly used for Natural Language Processing based use cases.
Logistic Regression is a geometric model based on decision boundary. This means scaling the data will affect your model's performance. It is mostly used for traditional row column based datasets.
In Depth explanation can be found here -
https://dataespresso.com/en/2017/10/24/comparison-between-naive-bayes-and-logistic-regression/


Make classification and prediction at the same time

I think the most intuitive solution would be to have two networks (i.e. one for predicting the next values in the time series and one for classifying if it is or isn't a seizure), because these are two very different tasks and there are different models that excel at each.
The classification network could even be on top of the time series predictor (include the predicted values to assess if it is a seizure or not).
However, what you want (one model to solve both tasks) is most certainly possible and is usually referred to as Multi-Task Learning. In this framework there usually is a large network that branches off to $M$ smaller ones (one for each task). The shared layers are supposed to learn from all $M$ tasks and outperform networks trained individually. If you would like to see some research done in the area I would suggest following MIT's affective computing group (example MTL applications: 1, 2, 3, 4, 5).
In your case you would have a network with two outputs, one for time series prediction and one for classification. Many machine learning libraries allow you to define two cost functions (one for each task) and even have a hyper-parameter you can tune to select the importance of each task. I've done a similar thing with keras' functional api.


scikit learn target variable reversed (DecisionTreeClassifier)

It might be because of the conflict of order between model.classes_ and series.unique()
For a binary labels,
model.classes_ = array([0, 1])
series.unique() = array([1, 0])
Try creating a constant value i.e. np.array[0,1] for labels and see.


Running a python script with heavy memory requirements

Python is by default single-threaded. Read about the Global Interpreter Lock (GIL). Although some modules with utilise multi-threading, asyncronous processing and multicore solutions, these will not take effect when you are simply loading modules and data.

Unless you are using some distributed system, or perhaps something like Tensorflow-serving, you will not be able to train/exectute your 20 LSTM models in parallel in a single Python script, so perhaps you could just load one at a time. Alternatively, just make 20 copies of the script - one for each model - then launch all 20 scripts in different terminals (or via a multi-threaded or multicore script).

Asking on StackOverflow would reach an audience that is more adept in the sorts of problems.


How do I combine predictions from classifiers for two different problem?

My intuition would be to try to integrate the information about the products directly in the original model. Typically the possible products in a shipment can be represented as boolean features (one hot encoding), but this part might need some feature engineering if there are too many different products:

simple option: only a small set of features representing types of products (I'm assuming that it's not the specific product which causes custom delays, it's the type of product)
advanced option: feature selection/extraction to reduce the number of features

Generally a joint model (a single model which deals with all the information at once) tends to perform better, in particular because in the other option errors in the first model propagate to the second one. Also the two models option doesn't allow the second model to leverage any specific feature from the first one.
Note that this is just my intuition, I could be wrong.
Side note: probably this is already taken into account but I guess that the value of the shipment is also an important factor for customs delays.


Which model to use for multiclass audio classification?

500 samples for 6 classes is not so much. You should put aside about 100 samples for validation and 100 for testing, leaving 300 samples for training.
I'm assuming that these drum loop are on the order of 1 second long (0.5-5 seconds). Then I would recommend trying a pretrained audio model, which typically have 1 second analysis windows. For example OpenL3, which is a powerful and easy to use audio embedding. It even has pretrained versions trained on audio.
On top of the audio embeddings try a simple linear classifier like LogisticRegression or a RandomForest.


How can I improve the results of my clustering

Always use a supervised algorithm when you have labeled data for your problem. Why would you ignore the labels, your most valuable bit of information?

To improve quality, you most likely need to improve your features.


What you are trying to do is called one hot encoding or dummy encoding. Even as this may be possible to accomplish in dplyr (together with tidyr), I would recommend using the function one_hot() from the library mltools. For an explanation, see the following code snippets:

# set up sample data
df <- data.frame(
  id = 1:4,
  media = factor(c("radio", NA, "tv", "tv"), levels=c("radio", "tv", "cinema"))
)

One Hot Encoding with tidyr:

The following code snippet shows how you would accomplish one hot encoding with dplyr and tidyr (copied from this comment):

library(dplyr)
library(tidyr)

tib <- as_tibble(df)
tib %>%
  mutate(i = 1) %>%
  spread(media, i, fill = 0)

However, this code will give you the following table:

# A tibble: 4 x 4
 id    radio   tv   `<NA>`
<int>  <dbl>  <dbl>  <dbl>
  1      1      0      0
  2      0      0      1
  3      0      1      0
  4      0      1      0

Note that there is no cinema column present, even though we defined a respective level in our sample data-frame. Also, the spread()-function created an own column for values with NA's, treating them as a separate category.

One Hot Encoding with mltools (recommended):

library(data.table)
library(mltools)

dt <- data.table(df)
one_hot(dt)

This code will give you the following table:

id  color_radio  color_tv  color_cinema
1          1          0            0
2         NA         NA           NA
3          0          1            0
4          0          1            0

As you can see, the column cinema is now present even though our sample data-frame did not contain any observations of it. Also, there isn't an NA-column present anymore. The NA from the second row is instead visible in all columns (which makes way more sense, since you don't know, what category this observation IS, so you cannot know what category it IS NOT).

In addition, this code is not only easier to understand it also runs about two to three times as fast.


How to implement Classification and Anomaly detection (C++)

You can try using SVM if your dataset is medium-sized or small (500-10000). You train the model with two classes: normal or anomaly (your output would be the probability of being of one class or the other).

A great library in C++ is libSVM (https://www.csie.ntu.edu.tw/~cjlin/libsvm/) which allows to perform classification or regression tasks.

There is a grest section of Dr NG's coursera ml class that focuses on anomaly detection. I would expect your classes to be imbalanced (you have much more normal cases than anomalies so you need to think about how to deal with this issue, also how do you choose the probability threshold to classify data as anomaly or not)

There are other choices of course: K-means can be used + some rule or another classifier. I would suggest you watch the Andrew NG class try his method (pdf kernel estimation) and once you got the hang of it try another model (such as SVM).


There are multiple possibilities.

The obvious one is to sum weights of all connections from input layer to the first hidden layer per input neuron. The neuron with highest sum of connection weights should be the most important. This has several problems.

Weights could be negative so you should use absolute value of them.
The weight depend on the scale of the input variable. You should normalize the input for example by subtracting mean and dividing by standard deviation of the input (z-normalization).

Another possibility is to selectively remove features, retrain the network and observe removal of which features caused the biggest decrease in model accuracy. These are potentially the most important features.

The problem of this approach is, that it changes the architecture of the network, retrains it and does not report the relevance of features entering the original network.

You can find an elaborate discussion on the problem here ftp://ftp.sas.com/pub/neural/importance.html

There are multiple other approaches, such as Garson's, using complex functions to correct for some problems. You can find several of them implemented in NeuralNetTools package in R.


One the key advantages that is I use a lot is that is compatible with pdb, so that

pdb.set_trace # To the rescue

Permits the use of python data structures

and let us to use pythonic control flow instead of using the main tf equivalents.

Also it allows to avoid metaprogramming issues such as "lazy loading" and adding a bunch of operations to my graph. Also autograd similarities


Why not just use quantile regressions?

Usually, in regressions, the coefficients are estimated for the average case: $\min_\beta \sum_i||X_i\cdot\beta -y_i||^2$.

If you want, you can estimate your coefficients so that they tell you things like $P(R=y)>p$. You give a probability $p$, and the model gives you $y$. You can even build probability density function out of these estimation techniques.

In quantile regressions, you solve this optimization problem for every quantile $\tau$ that you want, $\min_\beta \sum_{i|y_i\geq X_i\cdot\beta}\tau|X_i\cdot\beta-y_i| + \sum_{i|y_i< X_i\cdot\beta}(1-\tau)|X_i\cdot\beta-y_i|$.

This is a more robust approach to modeling the revenue distribution. Usually people use quantile regressions because software is readily available, but they can be implemented for other models as well. I work in an institute that does exactly that.


How to sum values grouped by two columns in pandas

pivot_table was made for this:

df.pivot_table(index='Date',columns='Groups',aggfunc=sum)

results in

         data
Groups    one  two
Date
2017-1-1  3.0  NaN
2017-1-2  3.0  4.0
2017-1-3  NaN  5.0

Personally I find this approach much easier to understand, and certainly more pythonic than a convoluted groupby operation.  Then if you want the format specified you can just tidy it up:

df.fillna(0,inplace=True)
df.columns = df.columns.droplevel()
df.columns.name = None
df.reset_index(inplace=True)

which gives you

       Date  one  two
0  2017-1-1  3.0  0.0
1  2017-1-2  3.0  4.0
2  2017-1-3  0.0  5.0


Extracting Products Name from Unstructured text

It's not a great dataset for NER because normally NER relies on trigger words close or inside the NE. For example "he went to X" would indicate that X is a location. Here there's no usable context and even the order of the lines can change.
What I would try here is to classify the lines, because apparently there's a kind of structure to each sample:

sender/recepient
email address
phone no
website
Instructions (e.g. "handle with care")

Once all the types of lines which are not a product are eliminated, the ones remaining are likely to describe the product.


Neural networks with not-fixed dimension for input and output

If youre searching for neural network architecture that have varying number of inputs and outputs, Recurrent Neural Networks, LSTM's .. etc are examples. They are used in Natural Language Processing where the main goal is to examine patterns in sentences.  But I highly doubt that they will work for your use case since no information about it is provided.
Another way would be to create multiple neural networks with a different input/ output sizes, such that the input/output sizes are averages of the groups of similar input/output sizes.


How to perform topic modelling on query search results

If you assume that the documents returned for a query already share a common "topic", then what's the point of using topic modeling? You already have a kind of "topical subset" of the documents.

However if the goal is to clean up the results so that documents which are "topically similar" don't appear together at the top, i.e. to favor "topical diversity" in the top resulting documents, then this could be done in the following way:

train a topic model for all the documents offline. You obtain a vector of posterior probabilities p(T|D) for every topic T and document D.
for every query, modify the scoring method so that there is for instance a minimal topic distance between documents ranked at the top (this part might be tricky to get right but it's feasible).

Incidentally, this approach is also more realistic because it would be quite inefficient to train a full topic model for every particular query.


Keras model to focus on different metrics?

Here are the docs for the Model.compile method.
The loss parameter is the objective function whereas the metrics parameter is a "list of metrics to be evaluated by the model during training and testing".
So your second interpretation is correct.


Feature Interactions vs Feature Importances

The most intuitive way to explain it I found it on this Nature: Explainable AI for Trees: From Local Explanations to Global Understanding:
Feature Importance: The contribution of a feature to the final prediction. If it was a linear regression, this will be the coefficient. For a decision tree you have mean decreased accuracy or as the paper says, shapley values. From the shapley github

Feature Interaction:  A model makes a prediction based on two features, we can decompose the prediction into four terms: a constant term, a term for the first feature, a term for the second feature and a term for the interaction between the two features. The interaction between two features is the change in the prediction that occurs by varying the features after considering the individual feature effects.
Lets say our model is like:
y = a + b·x_1 + c·x_2 + d·x_1·x_2

The feature importance terms will be b, and c.
The feature interaction will be the product x_1 · x_2
So one is the coefficient and the other one the relationship between the two. In the following graph, you can see the relationship between Age and Sex changes when the Age increases value.


Judging from the screenshot, you are currently looking at the data in the preproces tab from the explorer module. In the menu above the top menu in your screenshot should be an undo option (5th option from the left).


Why Gaussian mixture model uses Expectation maximization instead of Gradient descent?

Not all the parameters (e.g., the assignment parameters) for a Gaussian mixture model are smoothly differentiable, thus can not be fit with gradient descent.
Other use cases for the expectation–maximization (EM) algorithm are:

Clustering
Latent variable estimation
Missing data estimation


Catboost Categorical Features Handling Options (CTR settings)?

I found out that in order to set the ctr parameters and all the components one should pass a list of strings, each string should contain the ctrType and one of its component:

The first word of the string should be a ctrType for example Borders: (click here for catboost parameters)
Then one component of the ctrType should follow. For example TargetBorderType=5.
All together 'Borders:TargetBorderType=5'.
Repeat the procedure to set an other component and add the new string to the list.

Example with two components set:

simple_ctr = ['Borders:TargetBorderType=Uniform', 'Borders:TargetBorderCount=50']


First of all, sklearn.metrics.mutual_info_score implements mutual information for evaluating clustering results, not pure Kullback-Leibler divergence!

  This is equal to the Kullback-Leibler divergence of the joint distribution with the product distribution of the marginals.

KL divergence (and any other such measure) expects the input data to have a sum of 1. Otherwise, they are not proper probability distributions. If your data does not have a sum of 1, most likely it is usually not proper to use KL divergence! (In some cases, it may be admissible to have a sum of less than 1, e.g. in the case of missing data.)

Also note that it is common to use base 2 logarithms. This only yields a constant scaling factor in difference, but base 2 logarithms are easier to interpret and have a more intuitive scale (0 to 1 instead of 0 to log2=0.69314..., measuring the information in bits instead of nats).

> sklearn.metrics.mutual_info_score([0,1],[1,0])
0.69314718055994529

as we can clearly see, the MI result of sklearn is scaled using natural logarithms instead of log2. This is an unfortunate choice, as explained above.

Kullback-Leibler divergence is fragile, unfortunately. On above example it is not well-defined: KL([0,1],[1,0]) causes a division by zero, and tends to infinity. It is also asymmetric.


Simple Linear Regression-----How to make my model more efficient??

You cannot really do much: the fit of the regressor is optimum, therefore it is the best that the algorithm can do given these points. What you can do, is change the weights of individual points of the dataset using the sample weight parameter of

LinearRegression.fit(X, y, sample_weight)

method, to "attract" the line towards them and see how this affects the accuracy.

I would not expect notable difference because your data do not follow a linear pattern.


How do we predict what is in an image using unsupervised deep neural networks?

After a lot of reading, I think I now understand. We really need to build 2 models.

Model 1

Unsupervised
Lots of unlabelled images
Used to 'learn features' (i.e. better that we have done manually through years of research e.g. edge detection, colour features etc).

Model 2

Supervised
Few labelled images
Use model 1 as the 'feature extractor'. i.e. pass a training image through model 1 and use the output layer as the feature vector.
Use the same approach to test images e.g. model 1 to extract features, then use the second model to output label predictions


How to create model for selecting a set of categories with a set of attributes?

If I understand your question correctly, the problem you are trying to solve is a "multiclass classification" problem, so I'd suggest doing some research on that. Possible solutions range from Support Vector Classification (simple) to deep neural networks (harder). In most machine learning problems, it's best to start with a simple approach. This means that:

You learn about simple patterns in your data
You get a benchmark
If the result from your simple is good enough, you can stop there :)

I recommend checking out scipy's docs on classification models. Also if you're a beginner you might find this cheatsheet handy for similar questions in future.


What should return doc.ents if the doc have no entities, in spacy?

You could just test whether the tuples entities has any elements:

for sent in list(doc.sents):
    if len(sent.ents) > 0:
        nb = nb+1

Edit: For the purpose of evaluating boolean expressions (as in case of if statements), empty lists are evaluated as false, even though they don't explicitly equal False. Expanding your example with some print statements:

nb = 0
spacy_nlp = spacy.load("de_core_news_md")
text = "Nichts. Auch nichts."
doc = spacy_nlp(text)
for sent in list(doc.sents):
    print("type of ents: " + str(type(sent.ents)) + " with " + str(len(sent.ents)) + " elements.")
    if sent.ents:
        nb = nb+1

print(nb)

Results in:

type of ents: <class 'list'> with 0 elements.
type of ents: <class 'list'> with 0 elements.
0


GridSearchCV results are different to directly applied default model (SVM)

I think the problem might be due to the data, as this code:

from sklearn import svm
from sklearn.model_selection import GridSearchCV
import pandas as pd
import numpy as np

X_pca = np.random.rand(100, 2)
labels = X_pca[:, 0] + X_pca[:, 1] > 0.5

model = svm.SVC()
params = {'C' : [0.01, 0.1, 1, 10],
      'gamma' : [0., 0.1, 1, 'auto'],
      'probability' : [True]
     }
clf =  GridSearchCV(model, params, cv=2, return_train_score=False)
clf.fit(X_pca, labels)

print(pd.DataFrame(clf.cv_results_).loc[:, ['mean_test_score', 'rank_test_score']] \
        .sort_values(by='rank_test_score'))

gives better output:

    mean_test_score  rank_test_score
10             1.00                1
13             1.00                1
14             1.00                1
15             1.00                1
11             0.95                5
0              0.83                6
1              0.83                6
2              0.83                6
3              0.83                6
4              0.83                6
5              0.83                6
6              0.83                6
7              0.83                6
8              0.83                6
9              0.83                6
12             0.83                6

(consider the fact that the results will depend on the seed of the rand function, but with other seeds they are similar and the mean test score changes using different values of the parameters.


Detect constant (zero-slope) sections in a noisy step function

For what you're trying to do, first-order differences without interpolation should work just fine. Once you've done that, the problem reduces to a simple anomaly detection task. Here's a demonstration using topoligical anomaly detection. Briefly:

  The technique is essentially a density based outlier detection algorithm that, instead of calculating local densities, constructs a graph of the data using nearest-neighbors. The algorithm is different from other kNN outlier detection algorithms in that instead of setting ‘k’ as a parameter, you instead set a maximal inter-observation distance (called the graph “resolution” by Gartley and Basener). If the distance between two points is less than the graph resolution, add an edge between those two observations to the graph. Once the full graph is constructed, determine which connected components comprise the “background” of the data by setting some threshold percentage of observations ‘p’: any components with fewer than ‘p’ observations is considered an anomalous component, and all the observations (nodes) in this component are outliers.

First, let's generate some data:

n_obs = 100
spike_perc = .1

###################

set.seed(123)

n_spikes = floor(spike_perc * n_obs)
inter_arrival = rexp(n_obs)
xv = cumsum(inter_arrival)

mu_spike = 5
sd_spike = sqrt(mu_spike)
sd_noise = mu_spike/75
spike_events = sample(n_obs, n_spikes)

flat_noise = rnorm(n_obs, 0, sd_noise)
spike_ampl = abs(rnorm(n_spikes, mu_spike, sd_spike))

yv = rep(0, n_obs)
yv[spike_events] = yv[spike_events]  + spike_ampl
yv = cumsum(yv)
yv = yv + flat_noise

Here's what our data looks like (spike events in red):

Calculate first-order differences (appending 0 to preserve index mapping):

dat = c(0,diff(yv))

Find us some outliers:

library(igraph)

# Helper function. Equivalently, use mefa::stack
row_col_from_condensed_index = function(n, ix){
  nr=ceiling(n-(1+sqrt(1+4*(n^2-n-2*ix)))/2)
  nc=n-(2*n-nr+1)*nr/2+ix+nr
  cbind(nr,nc)
}

# Algorithm parameters, may require tuning
rq = .2 # distance quantile for thresholding (higher->fewer outliers)
p = .1  # percent of population necessary for a component to be flagged as an inlier
method='euclidean'

# calculate inter-observation distances and threshold to define adjacency matrix
n=NROW(dat)
d = dist(dat, method=method)
r = quantile(d, rq)
edges = t(sapply(which(d<=r), function(x) row_col_from_condensed_index(n,x)))

# build adjacency graph
g = graph.empty(n, directed=FALSE)
g[from=edges[,1], to=edges[,2]]=1
V(g)$name = 1:n

# Flag outliers based on component size
components = clusters(g, mode="strong")
csize = components$csize[components$membership]
outliers = V(g)[csize<=p*n]$name

And here's how we did (flagged outliers in red):

If you go this route, you may need to tweak the rq and p parameters a bit. If you'd rather not deal with the quantile thresholding parameter (rq), an alternative approach is to use a nearest neighbor graph, in which case you'll have to tinker with the 'k' parameter (of KNN). That approach would replace the distance calculations and graph instantiation above with:

library(cccd)

g = nng(data.frame(dat), k=5, use.fnn=TRUE)


Decision Tree Classifier with Majority vote (Is 50% a majority vote)

The goal of ID3 is to get the purest nodes possible ( ironically that is what contributes to its problem of overfitting), so 50% is not pure at all, the data under that node is equally likely to be in one of the classes which makes peedicition tricky, it would be better to grow the tree further and find nodes which are more pure than atleast 50%.


Does switching the positive and negative case make a difference?

For all of the algorithms/procedures that I know, it makes no difference between a positive or negative interpretation of a class.

The fact that one category has a lower frequency (by that I am assuming that you mean a low probability) is insignificant. It is just a matter of communication, by maintaining the general designations in a specific field so that others will be able to read your work more easily.

As an example, in anomaly detection, one designates the cases with lower frequency as anomaly, but even so, technically it does not matter how you label your cases, they are interchangeable.


saving the images to a folder with custom filenames

Most of the code you provided doesn't do much to help your actual problem of renaming the files. I will just focus on tha tpart by showing you how to get the filenames you want.

You can rename the files, as you describe them, like this - assuming you have a list of filenames that match your description.

filenames = ['1.0.2.34.57_1.png', '1.0.2.34.58_1.png', '1.0.2.34.59_1.png']

for file in filenames:

    # Do what you want with the file...
    # Image processing and modelling, etc.

    # Replace the extension with your custom ending
    final_filename = file.replace('.png', '_mask.png')

Another way to do it, breaking down the removal and addition into two steps

for file in filenames:
    file_without_ext = file.replace('.png', '')

    # Add you custom ending
    final_filename = file_without_extension + '_mask.png'

    # Another variant - joining the extension to the file_without_ext using "_"
    final_filename = '_'.join(file_without_ext, 'mask.png')

The three variants of final_filename will be identical.

Now you can save the output of your image processing using the final_filename.


Let me give you some pointers (assuming that I'm right on this, which might not necessarily be true, so proceed with caution :-). First, I'd figure out the applicable terminology. It seems to me that your case can be categorized as multivariate sampling from a categorical distribution (see this section on categorical distribution sampling). Perhaps, the simplest approach to it is to use R ecosystem's rich functionality. In particular, standard stats package contains rmultinom function (link).

If you need more complex types of sampling, there are other packages that might be worth exploring, for example sampling (link), miscF (link), offering rMultinom function (link). If your complex sampling is focused on survey data, consider reading this interesting paper "Complex Sampling and R" by Thomas Lumley.

If you use languages other than R, check multinomial function from Python's numpy package and, for Stata, this blog post. Finally, if you are interested in Bayesian statistics, the following two documents seems to be relevant: this blog post and this survey paper. Hope this helps.


Or must I loose most of the efficiency gained by programming in C by calling on R scripts or other languages?

Do the opposite: learn C/C++ to write R extensions. Use C/C++ only for the performance critical sections of your new algorithms, use R to build your analysis, import data, make plots etc.

If you want to go beyond R, I'd recommend learning python. There are many libraries available such as scikit-learn for machine learning algorithms or PyBrain for building Neural Networks etc. (and use pylab/matplotlib for plotting and iPython notebooks to develop your analyses). Again, C/C++ is useful to implement time critical algorithms as python extensions.


Stop CNN model at high accuracy and low loss rate?

Early stopping is determined based on the validation set's results (either loss, accuracy or some other special metric).
Usually early stopping is checked every single epoch so you will need to check your validation accuracy/loss after each epoch. You don't have to print it, but if it is already calculated, there is no reason to withhold it from yourself. (you can also check validation results every x epochs if it slows your training process too much).
There are many sources that show how to use early stopping with tensorflow:

Early Stopping with TensorFlow and TFLearn

Early stopping with tf.estimator, how?

Early-stopping functionality for use with tf.estimator API.


XGBClassifier: make the output of predict_proba ascending regarding a specific feature

You can enforce such monotonicity with the monotone_constraints (hyper)parameter:

https://xgboost.readthedocs.io/en/latest/tutorials/monotonic.html
https://stackoverflow.com/questions/43076451/how-to-enforce-monotonic-constraints-in-xgboost-with-scikitlearn


Predicting time of event from multiple timeseries

This kind of problem can be modelled in two different ways, which I call time-of-event and time-to-event. The first is what you have tried out so far, and illustrated in your picture.
Time-of-event
Feed in the entire time-series, and use as target the time of the event. Target can be encoded as the second from start of series, a percentage of the duration of entire series etc.
Advantages:

Simple. Standard regression setup.

Disadvantages:

Time-series might be very long, difficult for model to learn.
Few instances in training set. 1 per time series.
Single prediction for one whole series, no distribution of values

Time-to-event
Divide the time-series into multiple overlapping analysis windows. For each window, compute the distance (in time) to the event of interest, and use this as the target. The target can be expressed in seconds, or number of time-steps. Note that it may be positive (in future, relative to window) or negative (in the past, relative to the window).
At training time, the simplest setup is to treat each such analysis-window,time-target pair is treated as independent instances.
At prediction time each individual analysis window then passed through the model. Prediction for each individual window then has to be aggregated to give a single prediction. In the simplest case this can be just the mean or median. More advanced cases would for example weight predictions based on their time (maybe
Advantages:

Multiplies the number of training instances in dataset
Model input has lower data complexity. Since only seeing section of time
Can get predictions in real-time/streaming fashion
Get multiple predictions, forming a kind of distribution

Disadvantages

Adds another function block, aggregating of windowed predictions into a single prediction
Adds additional hyper-parameters. Window length, overlap, aggregation method
Overlap means that model runs many times on each data point. Might increase compute time for predictions

The window length can be quite short relative to entire time series. Ex a 15 minute series with 60 second windows using 5 second forwarding between each window (91.66% overlap). That would give 15min*60s/5s = 180 instances per series.
Or the window length can be quite close to the entire time series. Ex a 15 minute series with 10 minute windows and 1 minute forwarding between each window (90% overlap). Giving 15min/1min = 15 instances per series.
What works best here is highly problem and data dependent.
The use of multiple analysis windows like this is effectively a data augmentation technique (time-shift). Since it is applied also at prediction time, it is also an example of Test-Time Augmentation (TTA).
Models
I would always recommend starting with a non-time series model as a baseline. Summarize the features inside each window (mean,std,(max-min) of each variable), and use Linear Regression, Random Forest to see how it performs. This gives an indication of the complexity of the problem. And is super-fast to do some initial hyper-parameters scans on (for example on window length). And you should plot these features/predictions as part of Exploratory Data Analysis / Error Analysis!
After that I would try a simple LSTM / GRU model, or maybe a 1D CNN.


Are Word2Vec and Doc2Vec both distributional representation or distributed representation?

Effectively, Word2Vec/Doc2Vec is based on distributional hypothesis where the context for each word is its nearby words. Similarly, LSA takes the entire document as the context. Both techniques solve the word embedding problem - embed words into a continuous vector space while keeping semantically related words close together.

On the other hand, LDA isn't made to solve the same problem. They deal with a different problem called topic modeling, which is finding latent topics in a set of documents.


How to split temporal sequences to sub-sequences in a meaningful yet unsupervised manner?

It seems like your problem is not a typical "time series" problem: in data science and related problems, we normally look at evenly-spaced time series (that is, a measurement every interval, with each interval being the same length, such as 1 second, 3 months, 24 microseconds, etc).

There exist methods for working with events that can take some random time to happen, see survival analysis. If the coordinate domain (e.g. cell location) is important, you probably need a much more structured statistical model. Either way, it feels like unsupervised learning won't help you much.

Unfortunately, you haven't explained your data format very well. Without a
good understanding of your $x \rightarrow y$ relationship, nobody will really be able to help you. What is being observed? What do the numbers mean? (Time until the next event?) What groups are you trying to get? (Sequences of cell events?)


Advice on transitioning from Andrew Ng's Stanford Coursera Machine Learning (in MATLAB) into Python?

There are so many ways to expand your horizons on ML/DL. Currently I share the same conundrum, so to speak.

In my opinion it is not necessary. Since you have completed the course, if you are confident about the material and know how each algorithm works, re-implementing them on another tool would not be that helpful since there are existing frameworks that you can use. On the other hand if you are not that confident, re-implementing everything on a lower level  (not via APIs) can serve as a second learning experience, making you more fluent in python and more familiar with tensorflow. So, pick a scenario in regards to your case. For me a combination seems to work. I find papers along with other theoretical material and then try to apply it via the frameworks on projects that I like.
The greatest source of projects on any level is Kaggle. You can start with the Titanic competition, choose an algorithm that you like and explore what can be done. Make your submission and then head to another kernel that is more to your liking. You can also study certain pipelines on existing kernels that worked for other people. Considering concepts and methodologies, I would recommend an approach where you study each and every tool that you use and really understand the math behind it. There is no one-pipeline/framework solves all problems (No Free Lunch Theorem).

You also mention the deeplearning.ai specialization, it's a great source of information and Andrew Ng gives you intuition on aspects that are not very obvious when studying on your own. Another thing that helped me a lot is talking with people in the industry. Finding a mentor who inspires you and can give you his/her input on such questions is the best way to move forward. Go to AI/ML meetups and share your interest with other people, get to know companies in your area or even start an internship!

Good luck with your journey!


The closest would be like Jan has mentioned inhis answer, the Levenstein's distance (also popularly called the edit distance).

  In information theory and computer science, the Levenshtein distance
  is a string metric for measuring the difference between two sequences.
  Informally, the Levenshtein distance between two words is the minimum
  number of single-character edits (i.e. insertions, deletions or
  substitutions) required to change one word into the other.

It is a very commonly used metric for identifying similar words. Nltk already has an implementation for the edit distance metric, which can be invoked in the following way:

import nltk
nltk.edit_distance("humpty", "dumpty")

The above code would return 1, as only one letter is different between the two words.


Converting nominal data to numeric - is using dictionaries the right approach?

Usually, when dealing with nominal attributes, you want to use the binary vectoriser approach. Unless you have a very large number of nominal values, it typically doesn't matter that you're adding a few attributes. If you do have a huge amount of nominal values, you can try things like dimensionality reduction on the binary attributes (although this reduces interpretability) or joining some nominal values together.
Yes.
If the nominal attribute is discrete but ordered (called an ordinal attribute) -- for example, the severity of a cancer (stage 1, 2, 3 or 4) --
then you can use whatever numbers you feel are appropriate. It really depends on your domain knowledge, and the choice of model. For some models like decision trees, it doesn't matter, but for others like logistic regression the choices of the numbers can be influential. I would recommend just continuing as you have been by using intervals of 1, unless your knowledge of the domain says that particular adjacent nominal values should be further or closer than the others.


Is my Lasagne/Theano neural network running too slow?

Found the answer

I found out why it was going slowly, after doing some profiling and reading this:

It turns out that if you do a convolution with a stride != 1, then it doesn't use the GPU for that operation (or perhaps the gradient of that operation), and makes things run much slower.  I was only using a non-1 stride to save execution time anyway, but turns out I'm better off getting rid of the stride.

Also note that in your convoution layer, you have to completely get rid of stride=X.  Just having stride=1 is not sufficient.


Identifying sequences of actions required to complete tasks, based on data of completed tasks

What is the best way to solve this?
Based on my knowledge of linear regression and classification, I believe that both are not the right approach to solve the task at hand.

Not sure about the best way, but one indeed can formulate this kind of problem as a classification problem. I assume that additionally to the data included in your sample dataset you also have information about the environment for all 1,000 observations.
For example, one can reshape the dataset as follows.
Underlying data (I've inserted env as a placeholder for the information about environment, you might have several variables describing the environment):
|task |env |action1 |action2 |action3 |action4 |action5 |
|:----|:---|:-------|:-------|:-------|:-------|:-------|
|T1   |E1  |A5      |A1      |A3      |END     |NA      |
|T2   |E2  |A7      |A12     |A4      |A1      |END     |
|T1   |E3  |A5      |A3      |A13     |END     |NA      |

Transformed dataset that can be passed to the classifier
|task |env | num_action|prev_action |action |
|:----|:---|----------:|:-----------|:------|
|T1   |E1  |          1|START       |A5     |
|T2   |E2  |          1|START       |A7     |
|T1   |E3  |          1|START       |A5     |
|T1   |E1  |          2|A5          |A1     |
|T2   |E2  |          2|A7          |A12    |
|T1   |E3  |          2|A5          |A3     |
|T1   |E1  |          3|A1          |A3     |
|T2   |E2  |          3|A12         |A4     |
|T1   |E3  |          3|A3          |A13    |
|T1   |E1  |          4|A3          |END    |
|T2   |E2  |          4|A4          |A1     |
|T1   |E3  |          4|A13         |END    |
|T2   |E2  |          5|A1          |END    |

Then one can apply any multiclass classification (e.g. multinomial logistic regression) and try to predict the column action using the first four columns as predictors.


statistics or robust statistics for identifying multivariate outliers

Multivariate outlier detection can be quite tricky and even 2D data can be difficult to visually decipher at times.  You are spot-on in looking for robust statistical treatments analogous to 95% quantiles.

Where as normally distributed data naturally aligns with the chi square distribution, the gold standard for robust statistics in n dimensions would be to use Mahalanobis distances and then eliminate data beyond 95% or 99% quantiles in Mahalanobis space.

Plug and play capabilities are available in scikit-learn and in R.

Here is an excellent theoretical and practical treatment of the methodology:

And here is a big picture viewpoint with some heuristics.

Additionally there is a very sophisticated treatments called PCOUT for outlier detection that instead rely on principal component decomposition.  There is a corresponding R package, but the theoretical treatment is behind a paywall:

P. Filzmoser, R. Maronna, M. Werner. Outlier identification in high dimensions, Computational Statistics and Data Analysis, 52, 1694-1711, 2008

Hope this helps!


As far as I remember Kmodes is used for Categorical data, even in the documentation I couldn't find anything related to Mixed Data Type, if you have some reference do share.
I've used Kproto for Mixed Data types as you have mentioned and the package is ClustMixType, it combination of Kmeans and Kmodes.
Do have a look and let me know if have any questions.


Here is one in keras thats not that basic, but still interesting.


Does discretization of continuous features also lose information about distance?

You are absolutely correct. Also, sometime the binning could be arbitrary as well, but sometimes it depends on how you use the insight.

Using marketing as an example, we often see age grouped into buckets like 18-24, 25-34, etc.

Would a 34 year and 35-year old behave dramatically differently? Probably not. But then, it might still make sense to use such grouping, because many ad platform targeting tools use the same bin definition. It doesn't make sense logically but practically.

Of course, it really depends on the situation and the kind of analyses you need to do. For some distribution, you might want to do a log scale before you bin, etc.

Back to your example, if you feel that height should be a continuous feature and most models can handle it, is there a particular reason why you would want to discretize it?


Splitting binary classification into smaller susbsets

If you try to get the best accuracy, etc... for a given question you should always learn on a training set that is labeled exactly according to your questions. You shouldn't expect to get better results if you are using more granular class labels. The classifier then would then try to pick up the differences in the classes and try to separate them apart. Since in practice your variables in the training set will not perfectly explain the more granular classification question you shouldn't expect to get a better answer for your less granular classification problem.

If you are not happy with the accuracy of your model you try the following instead:

review the explanatory variables. Think about what might influence the classification problem. Maybe there us a clever way to construct new variables (from your existing ones) that helps. It's nowpossible to give a general advise on that since you have to consider the properties of your classifier
if your class distribution is very skewed you might consider over/undersampling
you might run more different classifiers and then classify based on the majority vote. Note that you will most likely sacrifice explainability of your model.

Also you seem to have some missunderstanding, when you write 'you would assign it to human if it doesn't fall into any of the granular classes'. Note that you always try to pick class labels covering the whole universe (all possible classes). This can be always defined as the complement of the other classes. Also you will have to have instances for each class in your training set.


How to run AgglomerativeClustering on a big data in python?

You can't.

By definition, the algorithm needs O(n²) memory and O(n³) runtime.

This does not scale to big data.

Use a different algorithm. Or subsample your data.

Results don't necessarily get better just because you use more data. In many cases it really does not matter. The quality of estimating the mean grows with sqrt(n), so it quite quickly does not pay off to use more data, as this will only affect small digits of the result.


Fitting lines through large point clouds

Based on the feedback and trying to find more effective approach I developed the following algorithm using a dedicated distance measure.

Following steps are performed:

1) Define a distance metric returning:

zero - if the points do not belong to a line

Euclidian distance of the points - if the points constitute a line according to the defined parameters, i.e.

their distance is higher or equal than the min_line_length and
their distance is lower or equal than the max_line_length and
the line consists of at least min_line_points points with a distance lower that line_width/2 from the line

2) Calculate distance matrix using this distance measure (use sample of the data for large data sets; adjust the line parameters accordingly)

3) Find the points A and B with maximum distance - break to step 5) if the distance is zero

Note that if the distance is higher than zero the points A and B are building a line based on our definition

4) Get all points belonging to the line AB and remove them from the distance matrix. Repeat the step 3) to find another line

5) Check the coverage of the point with the selected lines, if substantial number of points remains uncovered, repeat the whole algorithm with adjusted line parameters.

6) In case that data sample was used - reassign all points to the lines and recalculate the boundary points.

Following parameters are used:

line width - line_width/2 is the allowed distance of the point from the ideal line =   r line_width

minimum line length  - points with shorter distance are not considered to belong to the same line =  r min_line_length

maximum line length  - points with longer distance are not considered to belong to the same line =  r max_line_length

minimum points on a line - lines with less points are ignored = r min_line_points

With your data (after some fiddling with parameters) I got a good result covering all 30 lines.

More details can be found in the knitr script


I'd rather not rely on clustering (clearly, DBSCAN would be the first method to try).

Instead, I'd look for a projection that removes the correlation (x - y maybe?), then define a grid to separate the data.


If you follow the linked literature (down the rabbit hole for a few levels), you end up at a paper from 2005 from Kervann and Boulanger
- at least that's as deep as I got.

In that linked webpage, they define the patch-based image noise reduction methods as such:

  The main idea is to associate with each pixel the weighted sum of data points within an adaptive neighborhood.

So a patch is an area of a single image, like a convolutional kernel, but it doesn't convolve.

They talk about adaptive patches, meaning that you need to (perhaps randomly) select a pixel, then adapt the patch size used in order to include enough surrounding information to reproduce a homogenous patch as the output.

It seems as though training with clean images, to which noise is added (additive Gaussian white noise),  are used to train. This will help with robustness of final models by reducing the variance, but doing so also must introduce a bias to re-create areas where the noise is somehow uniform. The first link above, if you scroll down, shows many examples of typical images to be de-noised. The noise is not always so uniform.

Here is a picture taken from that 2005 paper, where they show patch-regions (marked in yellow). Page 5 gives a nice short description of the general idea. Patch sizes in their work were typically $7 x 7$ or $9 x 9$ in pixel size.


Taking your question title literally:

  Is RL applicable to environments that are totally RANDOM?

The answer would be no. In a totally random environment there is little to nothing that can be learned.

However, you don't actually have a totally random environment. You have some quantities that fluctuate a lot in a way that you don't understand. Otherwise your environment behaves very logically - if there is demand for Y items, and you have X in stock, then you will end up with X - Y in stock if X > Y, or 0 in stock and Y - X incomplete orders otherwise. This is a very structured rule that you can definitely associate rewards with and learn. Presumably you have costs for ordering and holding stock and opportunity costs for not supplying orders.

A simple variant of your situation is used as a toy example in Sutton & Barto: Reinforcement Learning, an Introduction in chapter 4, called "Jack’s Car Rental" where the goal is to optimise stock at two locations with random demand occurring at either location. This toy problem is made easier by defining the distribution, and using it in a model-based way. But that is not necessary in general.

  Is RL really applicable to such situations?

In your case, probably yes. Although you do have to assume that "totally random" is just a way of phrasing "highly variable" and not literally 0 one day, 30 million the next day and 7.5 the next day. There are going to be limits to the orders and they will follow some distribution.

With very high variance then you are likely to find it hard to reach a balance point of costs for holding stock vs costs for missing orders, but in principle this is solvable and RL is a reasonable tool to attempt the solution.

  If it does - then what will improve the performance?

Check whether any fluctuations in demand depend on any variables that you can collect and add those variables to the state. For instance, if demand has some weekly or seasonal variation, then those parts of the date should be part of your state representation.

Understanding and predicting the distribution of demand, even if not the exact values, would also help with simulations and planning algorithms. The RL algorithm will over time learn the likely distribution, but it can only base its own predictions on state variables that you have let it observe.

You might be able to go one better: If demand is mostly independent of the stock levels that you hold, then you can separate the problem of predicting it and use more robust supervised learning to add a "predicted demand" feature to the state. The RL will on top of this prediction learn the costs associated with just trusting this prediction vs allowing for some extra incoming orders just in case etc.


Help to choose algorithm for computing difference between 2 texts?

algorithm shouldn't do deep semantic analyzis, only kind of word counting, word vectorization, substring search

For such tasks, test if built-in Elastic Search / SOLR models work well. These products have multiple models that can be tuned with configuration. With these models, you provide a document and find other documents that are "similar".

Some of the available algorithms are :

Cosine Similarity
Okapi_BM25
DFR
LM Jelinek Mercer

Docs :

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html

https://lucene.apache.org/solr/guide/6_6/morelikethis.html


What ML algorithm can I use for building a "recommended" list for players?

Before jumping into machine learning solutions, it would be good to think more about the problem you're solving. If there are only 20 games and some are unavailable at any given time, then a well laid-out menu with good navigation is superior to a recommender system. Recommender systems are only appropriate when people cannot adequately parse all of the available options.

If you do want personalized recommendations, you don't even have to start with machine learning models. You can simply recommend that players keep playing the same games or the most popular games.

And if it turns out that machine learned models are best, I suggest looking at association rule mining based on unary data (which gives you shopping-basket recommendations: people who played games A, B, and C also played games D and E) or some variety of collaborative filtering based on ratings data (which gives you a user-item preference space). That totally depends on what sort of feedback you get from users about their individual game experiences.


Query regarding (.output_shape) parameters used in CNN model

The output shape in a 2D convolutional layer depends on :

The input size W, H
The filter size F
The padding P
The stride S
The number of filters or depth D

Here, in your model, W=3, H=4, F=2, P=0 (default padding in Keras), S=1 (default stride in Keras), D=128 as you defined it.

The transformation is defined by the mathematical convolution operation, giving you an output shape as:

$\frac{W−F+2P}{S} + 1$

This output shape shows how your input reacts (in terms of size) when convolved with your 128 filters one by one.


why does my calibration curve for platts and isotonic have less points than my uncalibrated model?

The default strategy for calibration_curve is 'uniform', i.e. each of the bins has equal width.  If, after calibration, your model makes no predictions inside a bin, there will be no point plotted for that range.
You could change to strategy='quantile', which would guarantee 10 points plotted for each curve; you'll get many more of the red/yellow dots further to the left.

Also, ideally you should not be fitting the calibration on the same data that you trained the original model.  From the User Guide,

An already fitted classifier can be calibrated by setting cv="prefit". In this case, the data is only used to fit the regressor. It is up to the user make sure that the data used for fitting the classifier is disjoint from the data used for fitting the regressor.


XGBRegressor vs. xgboost.train huge speed difference? - see this answer, likely the defaults (num_boost_round=10 vs. n_estimators=100) are the cause of this.


Extract sentences from beginning of news in single document summarization

Keselman, Schubert

Computational models for text summarization

The paper deals with methods (models) for text summarization. The reference (base) model was "first sentence model":

  As a baseline for our models we used a trivial model that repeats the
  first sentence of the input document.

Then, various experiments and results are presented, like this one: (notice that "first sentence model" is always present as "baseline")

Moreover, one of the datasets for training and evaluation of models in this paper is DUC, which may be interesting to you.

Steinberger (doctoral Thesis, 2005)

Text Summarization within the LSA Framework

In section 2.1, the author discusses document summarization approaches based on sentence extraction. He identifies five approaches:

Surface Level Approaches
Corpus-based Approaches
Cohesion-based Approaches
Rhetoric-based Approaches
Graph-based Approaches

(The "First Sentence Approach" belongs to the *Surface Level Approaches") The author further describes these approaches and compares them.

Khodra, Widyantoro, Aziz, Trilaksono (Journal of ICT Research and Applications, 2011)

Free Model of Sentence Classifier for Automatic Extraction of Topic Sentences

The author identifies and tests methods for identifying the most important sentences in a text (see list of 58 items below, called features). Surprisingly, in the conclusion, it is said that position of the sentence is a dominant feature, meaning that including all other features into consideration leads only to small improvement.

position
sentence length
number of words before a main verb
adjective incidence
existential there incidence
incidence of 3rd person singular grammatical form
anaphora incidence
coordinators incidence
cardinal number incidence
incidence of past tense endings
Hypernymy
Polysemy
concreteness index
affect_formulai
bad_formulaic
comparison_formulaic
continue_formulaic
contrast_formulaic
detail_formulaic
future_formulaic
gap_formulaic
good_formulaic
here_formulaic
in_order_to_formulaic
method_formulaic
no_textstructure_formulaic
similarity_formulaic
them_formulaic
textstructure_formulaic
tradition_formulaic
us_previous_formulaic
affect
argumentation
better_solution
change
comparison
continue
contrast
interest
need
presentation
problem
research
solution
textstructure
use
copula
aim_ref_agent
gap_agent
general_agent
problem_agent
ref_agent
ref_us_agent
solution_agent
textstructure_agent
them_agent
them_pronoun_agent
us_agent

For you, the most important part of the paper may be table 5:

Read carefully the explanation of the table in the paper, and the whole Section 4.3.

Other papers worth examining:

Luhn (1958)

The Automatic Creation of Literature Abstracts

Kupiec, Pedersen, Chen (1995)

A Trainable Document Summarizer

Yang, Pedersen (1997)

A Comparative Study on Feature Selection in Text Categorization

Sebastiani (2002)

Machine Learning in Automated Text Categorization


How to compare 10000 data frames in Python?

Simple approach:

Define an appropriate distance or similarity measure between two such data frames. It's unlikely that there is a standard measure for whatever game this represents. For example you could have a distance measure which sums every value in each data frame and then return the absolute difference between the two values, but this is unlikely to correctly represent the semantic of the game.
Implement this distance or similarity measure between two such data frames as a function
Call this function for every possible pair of data frames (that's 49,995,000 comparisons, it's doable).

Fair warning: a heatmap of an unordered matrix 10k x 10k might not be very exploitable.
Note that once you have the distance/similarity function, you could also:

use a clustering algorithm to group the data frames by similarity
detect outliers


The effect of imbalanced distribution of data

The best way forward here depends highly on the real life question you try to answer.

Let's say you want to make a medical diagnosis:

'Sick with exotic Illness X' or 'Not sick with exotic illness X' in this case you might want to catch all instances of being sick as a warning sign and could live with 'false positives'.

Conversely your algorithm will be used to predict 'customers likely to cancel soon', in this case it would not be a good idea to proactively talk to 'false positives' i.e. customers who did not plan to cancel about why they might be dissatisfied.

In either cases your training set and indeed reality might be severely unbalanced but the cost and consequences of this varies.

In the first case I would recommend using balancing methods (like the aforementioned Under-/Oversampling, etc.) to improve recognition of the minority class while in the second case that might be unnecessary.

In any case I would practically go on to do the following:

Include balancing/sampling in your beauty contest of algorithms and parameters and check the impact on the accuracy of predicting the test set (which is left unbalanced as in the original).

This will simply show you whether the inherent bias of the training set is problematic for your real world case (i.e. produces models that never identify the minority class) or not.


What are the methods to ensure that the population split for A/B test is random?

Just split them 50/50 (or another percentage, depending on what you want). Once you want to test a specific metric, make sure that if you can only generate that for a portion of the A group because of some specific property (ie you need information C for this metric) to compare it only to the people in group B that also have information C to avoid bias.

Example: I have 200,000 customers. I split them 50/50. I'm testing my new recommendation method against a random advertisement. However my recommender only works if they have searched in the last month. After getting back the results, I will have to check that against the control group filtered to have only people that have searched in the past month. People that have searched in the past month are much more likely to have a stronger kind of connection to you, so if you would check against the whole group you would overestimate the performance of your recommender.

Alternatively you could already split up the groups knowing whether they have information C or not, but you might want to test different things so splitting before hand can give you extra info, at the small cost that your groups might be slightly unequal, but for a big sample size this is not an issue at all.


There could be significant problems from using one time chunk in a time series to predict the next chunk. For example, if you were using data from September and October to predict retail shopping expenditure in November and December, you would be very wrong. Instead, you need to note the annual trend of increased shopping between Thanksgiving and Christmas to make correct predictions.

So, while this sort of skirts the questions you asked, I think that if you are having difficulty using one month to predict the next month, you need to start looking at longer-term trends, especially annual ones. I would investigate this first before looking into population instability.


Weights for keywords in a set of documents using Term Frequency and Inverse Document Frequency

TF - IDF stands for  term frequency–inverse document frequency
TF counts the frequency of a term / total #terms in a given document. For each term in a document, this value changes.
IDF counts the log of ratio of total document / term appearing in #documents . This value is constant for a given unique term. Greater the idf value for a term, higher its significance.
Example:

  Document 1: This is a sample example.

  Document 2: This is another example.

Lets calculate for term = "is":
TF(is, Document 1) = 1/5
TF(is, Document 2) = 1/4
IDF(is) = log(2/2) = 0
TFIDF = TF*IDF
TFIDF(is, document 1) = (1/5)*0 = 0
TFIDF(is, document 2) = (1/4)*0 = 0
It means that the term "is" is not a significant term in the list of documents(corpus).
Lets consider term = "another"
TF(another, document 1) = 0/5 = 0
TF(another, document 2) = 1/4
IDF(another) = log(2/1) = 0.301
TFIDF(another, document 1) = 0*0.301 = 0
TFIDF(another, document 2) = (1/4)*0.301
You can observe from both examples that TF varies per document while IDF is constant.
You can convert your entire 270 documents into term document matrix.
Demo in python:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

d = pd.Series(['this is a sample example','this is another example'])
df = pd.DataFrame(d)
tfidf_vectorizer = TfidfVectorizer(analyzer='word', min_df=0)
# if you want, say only top 2 features(terms)
# tfidf_vectorizer = TfidfVectorizer(analyzer='word', min_df=0, max_features=2, max_df = 3)
# Terms with given below:
# occurred in too many documents (max_df, tfidf score = 3)
# occurred in too few documents (min_df, tfidf score = 0)
# cut off by feature selection (max_features, tfidf score = 2).
tfidf = tfidf_vectorizer.fit_transform(df[0])
print tfidf_vectorizer.vocabulary_
# output: {u'this': 4, u'sample': 3, u'is': 2, u'example': 1, u'another': 0}
print tfidf_vectorizer.idf_
# output(constant): [ 1.40546511  1.          1.          1.40546511  1.        ]
print tfidf
# output:
#(0, 1)        0.448320873199    Document 1, term = example
#(0, 3)        0.630099344518    Document 1, term = sample
#(0, 2)        0.448320873199    Document 1, term = is
#(0, 4)        0.448320873199    Document 1, term = this
#(1, 0)        0.630099344518    Document 2, term = another
#(1, 1)        0.448320873199    Document 2, term = example
#(1, 2)        0.448320873199    Document 2, term = is
#(1, 4)        0.448320873199    Document 2, term = this

Source:
https://en.wikipedia.org/wiki/Tf%E2%80%93idf
http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html


ValueError: Input contains NaN, infinity or a value too large for dtype('float64')

I have tried so many suggested solutions but I found this one solve the problem.

data =data[~data.isin([np.nan, np.inf, -np.inf]).any(1)]


Can continuous dataset have negative values?

Yes, you can. Why would a negative temperature make no sense? Unless the temperature is measured in Kelvin...

You should know what each feature describes, and based on that, make sure the data makes sense. As of the dataset itself, with no context, it is indeed possible for any numerical feature to take negative numbers.

Hope this helps!


Using class weights in Keras with multiple binary outputs which are not simply one-hot-encoded

The solution to this question is to use sample_weight in the model.fit() function, (and as the third tuple entry in validation_data if you're using it). For each label sample you simply classify if it is rare or common using some algorithm, and set the weight accordingly.

For me this could be done with the following:

sample_weight = np.max(label_data, axis=1)) * 49 + 1

Here I took advantage of the fact that the common case is all zero per label row, e.g.:

     label_rows                             weights   label type
max([0, 0, 0, 0, 0])*49+1 = 0*49+1 =  0+1 =  1      # common
max([0, 0, 1, 1, 1])*49+1 = 1*49+1 = 49+1 = 50      # rare
max([0, 0, 0, 0, 0])*49+1 = 0*49+1 =  0+1 =  1      # common
max([0, 0, 0, 0, 1])*49+1 = 1*49+1 = 49+1 = 50      # rare
max([0, 0, 0, 1, 1])*49+1 = 1*49+1 = 49+1 = 50      # rare
...

That said, I ultimately recoded my labels so that there were only one positive entry in each label row, making it possible to set class weights. Not because weighting of the samples, but because it gives a better interpretation as a probabilistic output (using softmax). The following function does the weighting:

def create_class_weights(label_dim, rare_weight=10.0, common_weight=1.0):
    class_weight = {
        idx: rare_weight for idx in range(label_dim)
    }
    class_weight[label_dim-1] = common_weight
    return class_weight

It assumes the common case is the last per label row.