How to determine the abnormality of a specific variable by taking into account all the other variables in the data?

If you want to focus on the outliers wrt the class, you can do as follows:
Using Isolation Forest
import pandas as pd
import numpy as np

from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, FunctionTransformer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import IsolationForest
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline

import plotly.express as px

X, y = load_iris(return_X_y= True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 42)

out_model = Pipeline([("model", IsolationForest(random_state= 42))]).fit(X_train)

visualizer = Pipeline([("scaler",StandardScaler()),
                ("decomposer",PCA(n_components= 3)),
                ("framer", FunctionTransformer(lambda x: pd.DataFrame(x, columns = ["p1","p2","p3"])))]).fit(X_train)

outliers = out_model.predict(X_train)
X3D = visualizer.transform(X_train)

px.scatter_3d(data_frame=X3D, x = "p1", y = "p2", z = "p3", color = y_train, symbol=outliers )

pd.crosstab(index = y_train, columns= outliers, margins= True, normalize= 0 )

Output:

We can see that ~ 35% of observations that belong to class 2 are marked as outliers
This is a good starting point to analyze the reasons behind this.
Hope it helps


How to scale prediction back after preprocessing

When you decided you have to scale your data, you usually have to follow these steps:

For training:

Scale / Standarize the training set
Store the scaling / standarization factors of the training set
Train the model

For predicting:

Scale / standarize the input data, but very important, with the scaling/standarization factors stored during the training process. You dont have to compute the min, max or mean values of the new data.
Predict

The reason is that you have to map the new data to the same feature-space used for the training process, so you have to scale/std it with the same factors, otherwise, you are changing the feature space.


You are limited to
50 groups.

KDNuggets used number of members, growth rate, number of conversations and member engagment in their
analysis of data science groups.


How does the Naive Bayes algorithm function effectively as a classifier, despite the assumptions of conditional indpendence and bag of words?

The main reason is that in many cases (but not always) the model obtains enough evidence to make the right decision just from knowing which words appear and don't appear in the document (possibly also using their frequency, but this is not always needed either).

Let's take the textbook example of topic detection from news documents. A 'sports' article is likely to contain at least a few words which are unambiguously related to sports, and the same holds for many topic as long as the topics are sufficiently distinct.

In general tasks which are related to the general semantics of the text work reasonably well with unigrams (single words, unordered) as features, whether with NB or other methods. It's different for tasks which require taking syntax into account, or which require a deeper understanding of the semantics.


Quantify how much your clusters change with each batch.

Then stop if the change becomes smaller than a threshold.


scale_pos_weight using XGBoost's Learning API

Yes, you can use scale_pos_weight in the native python API; it goes in the params dictionary.  E.g.,

params = {'objective': 'binary:logistic',
          'scale_pos_weight': 2.5}
model = xgboost.train(params, dmat)

https://xgboost.readthedocs.io/en/latest/parameter.html#parameters-for-tree-booster
https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/speedtest.py


How can one quickly look up people from a large database?

That is problem is call identification, mapping a percept to a specific entity.
One common option is hashing, take a percept and map it to a specific, unique integer. If two different percepts map to the same integer, they are the same entity. If two different percepts do not map to the same integer, they are different entities. Hashing takes constant time to look-up an entity no matter how many entities.
In the case of facial recognition, the hashing function is best learned through Deep Learning.


How to handle Memory issues in training Word Embeddings on Large Datasets?

I'm not familiar with Torch, but since basically word2vec and doc2vec are considered, These models learn from each sentences and so there is no need to have all the sentences in the memory. You could iterate via each sentence in the corpora and let the model learn from each of the sentence. And that is probably how people train on huge corpora with or without high computation machines.

A short example in python:

class SentenceIterator(object):
   def __iter__(file_name):
      for line in open(file_name)
         yield line

sentences = SentenceIterator("path/to/file")
for line in sentences:
    model.train(line)

In this way, The memory is loaded with only one sentence at a time and when it is done the memory loads the next one.
For building the vocabulary, you can do the whole iterating via all the documents to build the vocab first and then train the data, depending upon the word-embedding functions implemented.


One option would be to cluster the longitude and latitude. Point estimates based on longitude and latitude would be wrong much of the time. Clustering would lower the precision of the data to increase the chance of the model being approximately correct.
Longitude and latitude can be clustering using spatially-aware indexing such as H3. Spatially-aware indexing allows for different size bins.


Yes, this is overfitting. Financial time series exhibit many peculiarities, including heteroscedasticity, high tail risk, all kinds of seasonality and all kinds of momentum. Machine learning algorithms tend to pick up these patterns and overfit on them. To my knowledge, there is no machine learning algorithm that can compete with classical time series analysis when it comes to understanding stock prices.

Please also keep in mind that if someone were able to make predictions like 'tomorrow the stock price is going to rise by 5%', they could earn billions within a few weeks.

Edit: You ask if there is any chance of training a network that is better than random guessing. Yes, for two reasons.

First, if you try hundreds of parameter combinations and evaluate them all on the same test set, you become the victim of what might be called 'overfitting of the second degree.' Performance on the test set becomes part of your decision process because you discard models that perform poorly on the test set and keep models that perform well. You are essentially training on the test set and will predict it with 100% accuracy eventually. But use this seemingly perfect model on new, previously unseen test data, and you will be disappointed.

Second, heteroscedasticity, seasonality and momentum all introduce a little bit of 'real' predictability into your time series. If you get very lucky, your algorithm might identify these patterns and at the same time ignore all the weird residual noise.
This is incredibly unlikely, and it is probably irrelevant. You should not compare your prediction accuracy to random guessing, but to a reasonable baseline model. For monthly prices of individual stocks, this could be the Fama-French three-factor model. (Daily or intraday stock prices will send you a bit deeper into the rabbit hole that is quantitative finance). I bet you five dollars that you will not be able to beat that model's performance on previously unseen data.

My advice to you: Don't clean the lion's cage on your first day as a zookeeper. Start with something nice and easy.


How to improve my f1 score in stories analyze

Author gender prediction belongs to the category of stylometry tasks, i.e. related the the style of the text rather than the content. In general this distinction requires a quite different approach than standard content-based tasks, here are a few of the most common differences:

Content words (nouns, adjectives, adverbs) are potentially less important, so TFIDF and similar methods make little sense. It depends on the data though.
Usually it's counter-productive to remove stop words, because they can be actually very useful and distinctive of an author's style (at least their proportion in a text).
For reasons which are not really clear, it's been found experimentally that bigrams and trigrams of characters are often better features than words. This has been observed at least in English, I don't know if this is applicable to other languages.

In general it's not recommended to target a specific level of performance unless it's a baseline known to be reachable, e.g. when another model can do it on similar data. That being said, in the case of this task I would say that 0.7 F1-score is reasonably realistic, but it really depends a lot on the data.
In case you are interested in researching the literature in order to apply state of the art methods, there have been some shared tasks about this question, sometimes mixed with other predictions such as age. search for "author profiling" and "gender profiling" in the PAN Shared Tasks. You might also find other relevant methods in the general category of "author classification/verification".


How to get feature importance from a keras deep learning model?

There is plenty of methods to calculate feature importance. I recommend trying two of them LIME and SHAP.

I don't want to copy-paste material and tutorial provided by the author so please refer to these two repositories.


Training machine learning models from log files

Hi and welcome to the community!

Sure you can use ML. I would start with brainstorming to formulate the problem. So one idea could be Sequence Learning:

Injecting a unique and recognizable number instead of F and training a NN-based model (RNNs are used in this case e.g. ESNs or LSTMs) will learn the dynamic of the data themselves as they model memory. But for training such models you need lots of data actually (to be more precise, you need long Time-Series). In this solution, the task is to see that unique and recognizable number in the prediction and output is a failure. See this for more info.
Hidden Markov Models are also designed for such tasks.

Another idea could be:

Segment each time series into intervals ending in a failure. For example:

  ts1 = [1,2,1,2,F,1,2,3,4,F1,2,F]

Becomes

> ts1_1 = [1,2,1,2]
> ts1_2 = [1,2,3,4]
> ts1_3 = [2]

Now you have a set of training samples (positive samples only!) for Time-Series 1 (e.g. Temperature). Now produce some negative samples from those sequence segments which do not result in "F" and train a model for e.g. classifying. Then on real-time prediction, you need to make a prediction after receiving every new point (temp for instance) and see if it is going to result in a failure.

DISCLAIMER: This way is not what you really do in practice but a nice starting idea for a newbie learner in the field!

I recommend having a look at this answer, this tutorial and this video as well.

Hope it helps!


Neural network approach to the cocktail party effect

Take a look at this.

No DNN, but math, if different channels are available.

DNN were used for single channel input, but have to be trained to the signals you want to separate.


How to validate different method of clustering

Could always try a silhouette score, although it may not perform well on high dimensional data.

Silhouette scores compare a datapoint's placement relative to its own cluster to its placement relative to other cluster/clusters.  The values returned will range between -1 (indicating a poor clustering) to +1 (indicating "good" clustering).  Using Wikipedia's example, use 2 clusters and assume there is an observation i that has been placed in Cluster 1.  The Silhouette score uses distance measures (Euclidean, for example, but others can also be used) to determine the average distance of i' to every other point in Cluster 1.  Then it determines the average distance of i to every point in Cluster 2.  The silhouette formula for point i would be:

silhouette(i) = $\frac{b(i)-a(i)}{Max\left \{ a(i), b(i) \right \}}$

Remembering for this example that i is in Cluster 1, then:

If an object is closer, on average, to the points in Cluster 1 than
Cluster 2, then a(i) < b(i) and silhouette score is positive for that
point.
If an object is closer, on average, to the points in Cluster 2 than
Cluster 1, then a(i) > b(i) and silhouette score is negative for that
point.

Run this comparison across all points and all clusters, then average, and you will obtain a global silhouette score.

You will see most people capture silhouette score in a for loop where they vary the number of clusters; this allows quick comparison of a range of clusters.


Calculating the entropy of a neural network

To start, see Information Theory, Inference, and Learning Algorithms by David J.C. MacKay, starting with chapter 40 for information capacity of a single neuron (two bits per weight) through to at least chapter 42 for Hopfield Networks (fully connected feedback). The classic reference for information of a Hopfield Network is Information Capacity of the Hopfield Model by Abu-Mostafa & St. Jacques, but the textbook should have enough.


is it possible to decide model without any data?

This is not a very strange situation in real world companies who nowadays want to build data science applications and other data related stuff, but without enough historical data (or none at all).
In this case, defining what a model is might help you, so:

are you/your superior considering only machine learning models? In that case, you need the data to train with
are you also considering a less sophisticated approach like rules-based models first? In that case, you can generate (i.e. programm it) such rules based on business knowledge before going directly into the machine learning pipeline, for which you need data

Another option, which I used once to check some ideas in advance (before having data) is to simulate some data based on the known data distributions which you know you might have in a near future; for instance, you might want to simulate clients ages, clientes accounts amounts... from other banks stored in some open-data platform.
In this case, you can model your data with for instance a kernel density estimator, to generate afterwards some synthetic samples. Below you can find what I onced made with a similar situation, where the orange bars are the open data retrieved with variables similar to what I would eventually have in my company (in this case I needed ages for each marital status, and I found it for a bank of a similar country) and used to generate the kernel density data generator (blue line):


The OLAP (On-Line Analytical Processing) cube with 4 dimensions

Your example is a star schema, it's just a star with three points (dimensions).
It's OK to have star schemas with more dimensions. Some large OLAP schemas can have tens of dimensions.
The star schema holds the underlying data. The cube is a convenient set of pre-aggregated values that make our run-time faster. The "cube" name is a handy visual and linguistic analogy but not a fundamental part of it's definition. Specifically, it is not limited to 3D. As in maths were a hypercube can be reasoned about in many dimension so an OLAP cube can be defined over many star schema dimensions. It just gets harder to draw in marketing literature, is all.


Trade-off between variable stability and discriminatory model quality

I think there might be a bit of confusion here: what is usually called a stable model is a model for which the performance doesn't vary (or not significantly) when sampling a different subset of training data or test data. In other words a model is stable if chance doesn't affect its performance. Typically one can use cross-validation in order to assess the stability of a model: if the variance is high between different splits then the model is unstable. Unstability is often a sign of overfitting, so one should be very cautious with an unstable model even if it seems to perform well: the risk is that the model might obtain good performance on the test set by chance, which means that it will actually perform poorly on new data but there will be no way to detect the problem. This is why people might favor a stable model to a high performance in some applications, because the consequences of deploying an unstable model in production would be costly.
But what you are talking about is not general model stability, it's stability of the performance across time for time-based data. That's a completely different question, it's about whether the model can represent time variations accurately. There are many types of models which can do that, but of course the more complex the model the more data it needs and the higher the risk of overfitting.


``Hierarchial features extraction'' in Multilayer Perceptron models

In MLP, you can imagine it as similar to finding features in different spaces.
Like for n features you come up with k feature by using PCA, similarly here MLP is converting the n features into k* features in different space.


Compute Baseline/Representative of Time-Series Data

Ad 1. Assuming the measurements at any given time are normally distributed (they shape approximately a bell curve), you could use simple standard deviation to detect outliers. Specifically, for any given time, you can calculate the mean and standard error. Then you calculate the mean sans outliers by taking into account only the measurements that fall at most some pre-set distance from the mean (e.g. given normal distribution, 68% of measurements fall within one standard deviation from the mean).

Pseudo-code example:

# Measurements at time t0 for all 10 days
t0 = np.array([0.1, 0.1, 1.4, .9, 1.25, 1.25, 1.5, 0.1, 0.3, 1.75])

# Get mean and standard error
mean0, std0 = t0.mean(), t0.std()

# Inliers are within one sigma from the mean
inliers = np.logical_and(mean0 - std0 < t0,
                         mean0 + std0 > t0)
# ==> [0, 0, 1, 1, 1, 1, 0, 0, 1, 0]

# And the baseline mean at time t0 is
baseline0 = t0[inliers].mean()
# ==> 1.02

Ad 2. You can find the most similar days to the baseline by using any appropriate distance measure (i.e. for time series: Euclidean or dynamic time warping). The result, then, consists of those days where distance is the least.


How do I compare coefficients from my glm when I have more than one factor variable in my formula?

R chooses the baseline level by itself unless you specify it. You can read here how to specify it:
https://stackoverflow.com/questions/3872070/how-to-force-r-to-use-a-specified-factor-level-as-reference-in-a-regression

Then you can change the reference (baseline) level to get all the possible results. This is easy to do but I am not sure if it is the best way to go, since you need to repeat yourself many times. But I think this might be better if you could apply it on your own data:

https://stats.stackexchange.com/questions/60352/comparing-levels-of-factors-after-a-glm-in-r

This is the link to the R package documentation:
https://cran.r-project.org/web/packages/multcomp/multcomp.pdf


1- The number of features: In terms of neural network model it represents the number of neurons in the projection(hidden) layer. As the projection layer is built upon distributional hypothesis, numerical vector for each word signifies it's relation with its context words.

These features are learnt by the neural network as this is unsupervised method. Each vector has several set of semantic characteristics.
For instance, let's take the classical example, V(King) -V(man) + V(Women) ~ V(Queen) and each word represented by 300-d vector. V(King) will have semantic characteristics of Royality, kingdom, masculinity, human in the vector in a certain order. V(man) will have masculinity, human, work in a certain order. Thus when V(King)-V(Man) is done, masculinity,human characteristics will get nullified and when added with V(Women) which having femininity, human characteristics will be added thus resulting in a vector much similar to V(Queen). The interesting thing is, these characteristics are encoded in the vector in a certain order so that numerical computations such as addition, subtraction works perfectly. This is due to the nature of unsupervised learning method in neural network.

2- There are two approximation algorithms. Hierarchical softmax and negative sampling. When the sample parameter is given, it takes negative sampling. In case of hierarchical softmax, for each word vector its context words are given positive outputs and all other words in vocabulary are given negative outputs. The issue of time complexity is resolved by negative sampling. As in negative sampling, rather than the whole vocabulary, only a sampled part of vocabulary is given negative outputs and the vectors are trained which is so much faster than former method.


Small mini-batch size leads to a big variance in the gradients. In theory, with a sufficiently small learning rate, you can learn anything even with very small batches.
In practice, Transformers are known to work best with very large batches. You can simulate large batches by accumulating gradients from the mini-batches and only do the update once in several steps.
Also, when finetuning BERT, you might also think of fine-tuning only the last layer (or several last layers), so you save some memory on the parameter gradients and can have bigger batches.


How can I calculate Kernel matrix K for clustering based Kernel Principal Component Analysis?

Foremost, we must understand what is clustering.

It is an unsupervised algorithm, applicable to scenario's where the target class is unknown. So essentially it's a data preprocessing algorithm. Continuing further, cluster analysis, helps in detecting patterns in data. Now, to detect patterns in data, one must ensure the sample consist of statistically significant variables. To find such significant variables, we need to perform various data preprocessing tasks like outlier, missing value('s), correlation and dimensionality reduction detection and treatment. Post, performing these steps, the noise from the data would be removed and one would be able to reveal the signal. Otherwise, one can perform the principal component analysis (pca) to detect the signal. This signal('s), can then aid in determining the true patterns or clusters in the data.

Now, coming to the second part of the question. You can use the kernlab package in R to calculate the kernel. Perhaps these posts can help you further, 1, 2, 3, 4.


Why SVM classifier error is much smaller than KNN classifier error?

Given that we have not applied any preprocessing steps then the first thing that comes to mind is that SVM works better in higher dimensionalities while kNN suffer from the curse of dimensionality and therefore our dataset should have a large dimensionality especially in comparison to the number of instances. Therefore we have a very sparse space where the points can be separated by a hyperplane while the k-nearest neighbors give little information of the actual class.

Note that we should take under consideration that we have no knowledge if the test size follows the rule of thumb (10%-20%) and we have not repeat the experiment with multiple train-test splits to get a range of possible errors. We might have a very unlucky test set..

Finally we should take into account that we have little knowledge if the classes are imbalanced and how much.
A highly imbalanced class could cause most of classifications to belong to the majority class while SVM would be worse because it would try, in vain, to find a hyperplane that would separate the classes and thus have a larger error.
However here we have the opposite scenario. Therefore we should assume the case where the classes are more or less balanced but they are separable only in one (or a few) dimensions while pretty much mixed in the rest of the dimensions. Since knn does not learn any weights for each dimension, it assumes that the ones closest are the correct points while svm has taken a clear cut decision that anything beyond this line should be classified as such.


software for workflow integrating network analysis, predictive analytics, and performance metrics

In Orange, you can do something like this:

This takes the network, which is already containing class you'd like to predict, then training (or testing) the learner in Test & Score and evaluating it in Confusion Matrix. Then you can see misclassifications directly in the network graph.

There are a bunch of other learners and evaluation methods available. A big plus is also interactive data exploration (see how you can input wrongly classified data into Network Explorer?). However, there's no dashboard available yet. We make do with opening several windows side by side.

That's just my 2¢ on Orange. I suggest you to at least try all of them and see which one works best for you. :)


I haven't seen anything like that and very much doubt that such frameworks exist, at least, as complete frameworks. The reason for this is IMHO the fact that data transformation and cleaning is very domain- and project-specific. Having said that, there are multiple tools that can help with these activities in terms of partial automation and integration with and between existing statistical and Web frameworks.

For example, for Python, the use of data manipulation library pandas as well as machine learning library scikit-learn can be easily integrated with Web frameworks (especially Python-based, but not necessarily), as these libraries are also Python-based. These and other Python data science tools that might be of interest can be found here: http://pydata.org/downloads. Specifically, for cleaning and pre-processing tasks, which you asked about, pandas seem to be the first tool to explore. Again, for Python, the following discussion on StackOverflow on methods and approaches might be helpful: https://stackoverflow.com/q/14262433/2872891.

Consider an example of another platform. The use of pandas for data transformation and cleaning is rather low-level. The platform that I like very much and currently use as the platform of choice for data science tasks is R. Rich ecosystem of R packages especially shines in the area of data transformation and cleaning. This is because, in addition to very flexible low-level methods of performing these tasks, there are some R packages, which take a higher-level approach to the problem, which may potentially improve developer's productivity and decrease the amount of defects. In particular, I'm talking about two packages, which I find very promising: editrules and deducorrect. You can find more detailed information about these and other R packages for data transformation and cleaning in my another answer here on Data Science StackExchange (paper that I reference in the last link there could be especially useful, as it presents an approach to data transformation and cleaning that is generic enough, so that could be used as a framework for this on any decent platform): https://datascience.stackexchange.com/a/722/2452.

UPDATE: On the topic of recommender systems and their integration with data wrangling tools and Web frameworks, you may find my other answer here on DS SE useful: https://datascience.stackexchange.com/a/836/2452.


Non-brute force approach to finding permissible English word anagrams

A simple solution is store the words in a dictionary (since you have to store them in some data structure) whose key is the character distribution; e.g., Counter("hello") = {h: 1, e: 1, l: 2, o: 1}. Querying the dictionary would give you the anagrams.

For storing the immutable key (character distribution) can either use a tuple list, incurring a sort, or you can use a vector the length of your alphabet (26). So you can make a speed-space trade-off in the preparation stage; the look-ups are constant time, not counting the time it takes to calculate the character distribution of the query word. If you go the latter, fixed-width route, you can make another trade-off by hashing the key, since you know the cardinality of the input (number of unique words).


Why is it wrong to train and test a model on the same dataset?

Yes, you put it quite correctly.
As a teacher, you wouldn’t give your students an exam that’s got the exact same exercises you have provided as homework: you want to find out whether they (a) have actually understood the intuition behind the methods you taught them and (b) make sure they haven’t just memorised the homework exercises.


difference between supervised learning and imitation learning

In supervised learning each data example has a label.
Imitation learning is mapping from observations to actions and is generally considered part of reinforcement learning.
The primary difference between supervised learning and reinforcement learning are when the labels / rewards are available. Reinforcement learning labels / rewards are often time delayed. In supervised learning, the labels / rewards have to be available concurrently with all the data. In reinforcement learning, the agent might make several actions before receiving a reward.


Forecasting energy consumption with no historical data when there is a trend

Most time series models assume stationarity. Thus, you should account for trend and seasonality in your data and subtract the corresponding components from your original data. After that, you can model your normalized data also with arbitrary machine learning models; and forecast with new data the following months.

Carefully note that you may have features that explain the trend or seasonality component. If these components are also changing or expecting to change over time you could combine (e.g. add for additive time series models) predictions for all components. I.e. the sum of the trend, the seasonal and the residual prediction.

Importance of Normalization
E.g. https://stats.stackexchange.com/q/31387/214243

Detrending
Subtract the %yoy change for each month.

Seasonal decomposition
E.g.  http://www.statsmodels.org/stable/release/version0.6.html?highlight=seasonal#seasonal-decomposition

Forecasting
There is an abundance of statistical time series models (AR, ARIMA, GARCH,...). But when data is normalized common machine learning models like decision tree or random forecast may also be applicable (see sklearn library for this)


Your answer was incorrect because the axes were flipped and it didn't plot the entire line.

This was the correct answer I was looking for.

import matplotlib.pyplot as plt
import numpy as np

# define the parameters
g = -9.81 # m/s^2
dZ = 60 # m
dx = 2e5 # m
omega = 7.2921e-5 # rad/s
phi = np.linspace(10,90) # deg
f = 2 * omega * np.sin(np.radians(phi)) # coriolis frequency, s^-1

# compute geostrophic wind, x-component
u_g = -1. * g/f * dZ/dx

# plot phi vs V_g
fig, ax = plt.subplots()
ax.plot(phi, u_g, color='#FF33CC')
ax.set_xlabel('$\phi$')
ax.set_ylabel('$V_{g}$ (m $s^-1$)')
plt.title('Geostrophic Wind vs. Latitude')
plt.show()


Training Neural Networks with unknown length of input

Gravity

Basically you'd like to find a function that maps an input (an object with a mass and a location) to an output (new location). It's not necessary to have one set of input neurons for each different object. It's sufficient to encode the input variables generically for all objects.

Like a function:

$$f( x, y, z, mass ) \to ( x_n, y_n, z_n )$$

XOR

Training the XOR function is special as the early neural networks, the perceptrons, were unable to learn the XOR function (because it is not linear separable). Multi-Layer Perceptrons however are able to learn the XOR function. This is just to demonstrate that the NN implementation of Synaptic is capable of learning problems that are not linear separable


How do I control sensor data readings against (measured) outside influences?

In short, I think the focus of your question is mostly on how to deal with covariates (watch out for the not always clear use of this term and its synonyms - see here and here. What I mean by covariates is "In statistics, a covariate represents a source of variation that has not been controlled in the experiment and is believed to affect the dependent variable.")
If we first take a step back and ask what you really want from your analysis, it is to my understanding to investigate the relationship between two (or more) variables. These would be e.g. EDA values on one hand and some output variable such as e.g. emotional states based on heart beat data on the other (since I don't have much information on your outcome variable(s) at this point, I will simply call them y). Thus, you have EDA values as an independent variable and y as a dependent variable. So far, so good - bringing together dependent and independent variables statistically can be done in many ways, ranging from even just calculating a correlation between the two (given they are both numeric variables) across fitting linear models and up to machine learning techniques etc. Let us for now go with setting up a linear model:
Short-hand notation: y = EDA
Same model written out more detailed: y = intercept + EDA + error
You already know, however, that you have this other outside influence that affects your EDA-values. Luckily, you even tried to measure/quantify this influence via the accelerometer. Thus, besides dependent variables that you are interested in and the independent variables you are interested in, you now have a third type of variable: the independent variables you are not really interested in, but that you expect to have an influence on the variables that you are interested in. This is what I mean by covariates. Taking a covariate into a model can greatly improve the statistical model in terms of its ability to actually analyze the variables we are interested in. In other words: In the optimal case, a covariate explains so much of the unexplained noise in the data, that the variability that is left lets us draw conclusions about the relationship between the other variables that we could not see before. So the model from above may become:
y = EDA + COVARIATE
y = intercept + EDA + COVARIATE + error
Thus, the question comes up on how to decide whether or not to include such a covariate in your model/analysis. One analysis that is very much in this realm is ANCOVA - analysis of covariance.
I will stop here to make sure I understood the problem correctly and would be glad to hear whether this is helping so far.


Sparse connections in feedforward network tensorflow or pytorch?

We don't use sparse connections in feedforward to make use of an efficient matrix product operations.
But If you want to model that aspect, see dropout


Imputing features with NA values in classification task

Thank you for clarifying the question @Eisen. So the question looks at two main things:

To show which districts need more police presence.
To classify people as to whether they will commit 1+ traffic violations, regardless of the district, given their previous number of violations, by district.

For the first point, I think what would be a good idea is to yes show the break down of people visiting each of the districts and committing 1+ traffic violations. But, I think adding (95%) confidence intervals would be particularly helpful to see what is a reliable estimate of people who commit 1+ traffic violations in a given district.
In terms of the second point, maybe you can use a feedforward neural network which will take as input the traffic violation categories for each district and output whether the person has committed an accident in 2019 (Accident19).
The architecture is pretty much up to you, but the fail layer would need to be a 2-node softmax layer, which will create the probability distribution over the two classes (has had an accident in 2019 or not).
To represent this categorical data, I would suggest making the NA it's own category. Furthermore, best to represent the district traffic violation categories as one-hot encoded vectors. Then for a particular person, you concatenate these one-hot encoded vectors from all districts in your dataset.


line search in gradient descent applied to a convex function

This isn't a perfect example but hopefully the idea is clear.
If you start at a point just left of 0, small learning rates will never get it out of the local minimum at the left. But large enough learning rates to leave that local minimum will never let it settle at the global minimum at the right.

A decaying learning rate could work for such a point.


Intutitively what advantages are there to use more data than absolutely necessary while fitting a linear regression?

If your sample(a subset of the population) is able to represent your population, then it isn't required to use all your data. In other words, if your model can extract all the possible patterns from your sample, then it is good enough. It will also save you from time and space complexity while training. But, let's say your model1 is the one that trained with a subset of your all data, and model2 is trained using all the data you have (except validation and test sets). It is possible that while you did a performance evaluation, your test set hasn't perfectly shuffled and some type of the samples (or samples belong to a specific pattern) weren't in that test set, thus your accuracy was high in model1. In other words, the probability of your population's not including a specific pattern is lower than or equal to the probability of your subset's not including a specific pattern. Thus, if we imagine that your test data was poorly shuffled or your population still somehow includes specific types of data (or pattern) then it will perform better in future data. Unless you are 100% sure that your sample can fully represent your population (which is almost impossible most of the time), then it is always better to use more data which will reduce the chance of spurious results. It is also possible that your model2 can perform worse than model1 if you have outliers in it.
To conclude, unless you observe that using more data gives a rise to loss of performance, then always use all the data by considering your test set's not being perfect and subset's always containing less than or equal characteristics about your whole population. The model2 statistically guarantees you higher performance than model1 in unseen data if model2's data is outlier-free. Because it will learn better to generalize your data than model1.


Is the search for a specific n-gram the same like a string search?

Well, as you state the problem, it is true that the search for a certain sequence of strings/words is the same as looking for the corresponding n-gram.

However, keep in mind that an n-gram, when you use it for ML, (often) is represented as a factor. So a certain sequence of words or strings, is thought of carrying valuable information. Like „John Holmes“ is different from „Sherlock Holmes“, when it comes to, e.g. identifying book titles or covers.

In general, an n-gram is just a certain sequence of words/strings, which carry more information than only one word/string.


I am doing classification using SVM for two classes , is this result good and what does these values refer to?

Well, if it is a good results or bad results is decided by the task you are performing. But so far, it looks like good results. Coming to the metrics, precision, recall and F1 can be learnt from wikipedia here.
However, the interpretation of your results goes like this:
The left most extreme 0 or 1 indicates the class (in your binary classification task).  And the values 0.92, 1.00, 0.96 corresponds to precision, recall, and F1 of the class 0 respectively.
Likewise, the next line indicates precision, recall, and F1  for class 1.
support indicates support count, which in simple terms is the number of samples for a given class in the test set.
Macro average precision/recall/f1-score is the average precision/recall/f1-score for all classes.
Weighted average precision/recall/f1-score is the weighted average precision/recall/f1-score for all classes, here the support count is used as weights.


Interpreting Categorical Crossentropy Loss

Cross-entropy is an information-theoretic measure about probability distributions, and it's measured in units that are determined by the base of the logarithm used in its computation (nats for the natural logarithm or bits for $log_2$). There are already several posts about intuitive understanding of cross entropy and its relationship to KL divergence (which are good to understand).
I think the more interesting part of your question is what "2 categories off" means with respect to your problem. Generally in classification optimized using categorical cross entropy, the classes are pairwise orthogonal: the content in the image is of a dog or a cat or a ship or whatever, so the ground-truth vector is always one-hot (1 in the index of the correct class, 0 elsewhere). Your question implicitly places an ordering over the classes: for something to be a number of "categories off", there must be some sense of similarity that makes categories $i$ and $j$ more alike than $i$ and $k$, say. If that's the case, you could look into alternative loss functions (but probably don't have to, depending on what you're doing).


About data cleansing, to what extent should we do our work?

This is a question that is very dependent upon the data in question. Assuming that you can train your models in a reasonable time, I would start by not cleaning the data at all, and seeing how well your model performs, and then cleaning it a bit and redoing the experiment and so on.
This is because it is possible to overclean your data, essentially removing variations in the data which are actually useful to your modelling.
The amount of cleaning that you apply is yet another meta-variable in your model.


Using LSTMs for continous learning and predicting

To answer your questions in order:

The whole point of LSTM (or any time series model) is to predict previously unseen values. If states are not reset - then there is a risk of data leakage - whereby forecasts from the training set will lead into the test set. This would mean that your model would perform quite well on the existing set of data - but would perform badly on new/unseen data. This defeats the purpose of implementing a time series model.
Depending on how similar the pattern is, it might be an option. However, it is just my personal experience that LSTMs are not particularly effective at forecasting commodity prices - the main reason being that these models tend to carry forward too much volatility in the forecasts, which tends to interfere with the overall trend of the data. Here is an example of the use of LSTMs in forecasting oil prices, which you might find of interest: Are LSTMs Always Effective At Time Series Analysis? A Case Study of Oil Prices


A clear visualization of a two-way ANOVA

You could go for a 3D plot, but they are only useful in specific situations (e.g. simple bar plot, a surface etc.), and definitely not here. Your best option, while keeping the overall design of your plot, seems to be to split your data by the factor with 2 levels, and plot each separately in 2 panels side by side. That way, you can compare similarities, baseline shifts, and interactions.


How prevalent are traditional CS questions in data science interviews?

It depends on whether you're going to write production code or even pseudo-code that will be implemented by others. If so, yes you need computer science skills. If you're going to merely analyze data, you can technically get away without it, but some companies (e.g., Facebook) expect all technical employees to know computer science.

Computer science is useful even to pure analysts because it will help you implement algorithms not covered by your libraries.

I make use of recursion and dynamic programming, for example, but I wrote production code. That's my deliverable, after analysis and prototyping.


What does "Find pattern" mean in data science?

First, I'm going to directly quote the two inner questions you put in this question:

  "Find Patterns", What does it mean?:

Find patterns is the term given to the process of analyzing data(Data Analysis) or dataset for insights in it that is to say; the classification of input into per-determined categories,recognize particular categories of input by their characteristics.

So when they ask you to find patterns,they assume that you don't necessarily know what is in that data(it can be structured,semi-structured and unstructured) so you can't even make decisions upon that data.therefore you have to find patterns in it. You have to look for them, which can often involve a lot of hunch-following and guess-work combined with a bit of solid testing.you tend to know the patterns already and you search for them based on their features or attributes etc.

  what steps should i carry out?

Ask Question
Do some background research
Construct hypothesis
Test with an experiment
Analyze results and draw effective conclusions.

Concerning Hypothesis

If the hypothesis is true,then proceed and report results
or Else,if it's false or partially true report results.

Hope this can give you a glimpse.


Is my general understanding of finding weights correct?

This is interesting !?!
The way you think about it is as solving a linear system equation, which is not exactly the case.
Imagine if you have thousands of data points , and you try to find the set of weights that solves ( satisfies) these points, what if an exact solution is not exist. What if there is no set of weights that exactly solves all these points. It becomes an optimization problem. What is the set of weight that minimizes the error. In this case the error is the difference between the actual output and the output you get by using the current weights.
In fact, in current machine learning technology, it is difficult even to construct such a linear system.
We use optimization techniques to solve this and find the optimal set of weights


How important is the input data for a ML model?

To answer your first question, the accuracy of the model highly depends on the "quality" of the input data. Basically, your training data should represent the same scenario as that of the final model deployment environment.

There are two possible reasons why the scenario you mentioned is happening,

When you added more data, maybe there is no good relationship between input features and label for the new examples. It is always said that less and clean data is better than large and messy data.
If 49000 records added afterward are from the same set(i.e. have a good relationship between label and features) as that of 1000 before, there are again two possible reasons

A. If accuracy on the train dataset is small along with test dataset. e.g. training accuracy is 70% and test accuracy is 65%, then you are underfitting data. Model is very complex and dataset is small in terms of the number of examples.

B. If your training accuracy is near 100% and test accuracy is 65%, you are overfitting data. Model is complex, so you should go with some simple algorithm.

NOTE* Since you haven't mentioned about training accuracy, it is difficult to say what out of the two above is happening.

Now coming to your second question about real-world deployment. There is something called model staleness over time which is basically the problem of reducing model accuracy over time. This is the article by a product manager at Google explaining the staleness problem and how it can be solved. This will answer your second question.

Let me know if something is not clear.


non-separable assignments of the vertices of a hypercube

You appear to be correct. I believe this is a simple proof of the opposite fact:

Let $s(n)$ denote the number of separable sets in the $n$-cube. Let $Q_0, Q_1$ denote the subcubes of one smaller dimension, with say last coordinates equal to 0, 1 respectively. The key fact is that any separable set of the whole cube intersects with each of $Q_0, Q_1$ in separable sets. (The separating hyperplane intersects the codimension-1 spaces to form separating hyperplanes.) So
$$s(n)\leq [s(n-1)]^2.$$

Now the proportion $r(n)$ satisfies
$$r(n)=
\frac{s(n)}{2^{2^n}}\leq
\frac{[s(n-1)]^2}{(2^{2^{n-1}})^2}=
[r(n-1)]^2<
r(n-1),$$
where the last inequality holds because $r(n-1)<1$ for $n>2$.


Can training examples with almost the same features but different output cause machine learning classification algorithms to perform poorly?

The answer is yes, highly similar instances in your dataset that have different target classes will cause your model to perform poorly.

The reason for this is at the core of how all classification machine learning algorithms work. The goal of a classifier is find a function which can separate the two classes. Thus, if these two classes are very mixed then the probability of making a classification error increases and thus you will lose precision in your resulting classification.

One method to correct this problem is to add more features to your dataset. You should try to find features which will distance the distributions of these two classes. For example if classifying cats and dogs, it would not be a good idea to use features such as: number of legs, number of eyes, etc. This will cause the classes to be indistinguishable. Try adding features such as: weight, frequency of cry, etc. This can be difficult to do often as collecting additional data is expensive. You can also try to transform your data to a new feature space. A transformation mapping your features to a different space can cause their distributions to distance themselves.


Help making a custom categorical loss function in Keras

The issue is very easy to solve assumming you still want to use crossentropy as your loss.

Tell your model to use temporal sample weights. You can do it like this by model.compile(sample_weight_mode='temporal', **other_params)
Generate your sample weights, I think you are smart enough to write your own implementation. The idea is as you said apply weights of 1 if it is supposed to be counted and apply 0 if it is not supposed to be counted. For example you have a sequence [3,5,1,-3,2,0,0,0,0] then your sample weights will be [1,1,1,1,1,0,0,0,0].
Supply the sample weights during fitting. Simply use model.fit(X,y,sample_weights=sample_weights, **other_fit_params).

Done. Now loss is only counted on non-zero entries of your output.


How to cluster categorical and numerical data in the same dataset?

You can use one-hot-encoding for all categorical features. Then normalizing numerical features (as one-hot is 0 and 1 then maybe normalizing your numerical data to [0,1] will bean intuitive starting poit). Then apply a dimensionality reduction technique as probably you will produce a sparse matrix with maybe considerable number of dimensions. Then do your clustering (try at least two different one. I would say k-means and DBSCAN).

A nice intuitive replacement to one-hot-encoding is to replace values with their fraction. For example in column gender you have 60 male and 40 female, then replace male with 0.6 and female with 0.4. The only drawback is when the number of categories are equal in a column! You need to be careful about those.


Unsupervised Clustering high dimentional data not having estimation for K

I point out several things first:

You have an unsupervised problem so the first to know is don't search for The Right Answer. There is no right answer and you take the best you get acording to some pre-defined criteria!
not knowing K is another problem: This is one of many questions in unsupervised learning which do not have any answer! All those methods like elbow are just heuristics.
Kmeans doesn't report one of the nodes as center for the cluster: K-Means can be modified to do so. The simplest trick is to find the closest neighbor of final representatives from the data point and set them as representatives of clusters.
Some methods that you mentioned like K-Means and Sillouette assume gaussian clusters. If your clusters are not properly compact and well-separated, then they mislead you.

After these, let's have a look at your problem. Try to get an insight about your data with visualization techniques. tSNE is pretty sensitive to its parameters and needs more investigation. Start with PCA, KPCA or LLE to get some idea about data. In case they didn't give you a proper embedding, then go for UMAP. UMAP is also sensitive on parametrs but less than tSNE and it's pretty faster. It will give you more compact clusters if it finds any which is a bless for your work. At the end if none of them worked maybe try tuning tSNE parameters and see if you can find a good embedding.

On top of UMAP or tSNE you can try your clustering alorithm and it will be a logical pipeline for your problem.

Good Luck!


Preparing multiple training time-series for Keras LSTM regression model training

The standard way to treat unequal lengths is to pad them.

However, if you have very long sequences, it is often handled by splitting them into short sequences - in this case, it is up to you, what sequence length to choose.

Then you will have to decide, whether you need stateful LSTM or no, but it is out of the scope of this question.


How can we define missing rating in recommender system?

For the Collaborative Filtering while using Matrix Factorization you don't necessary use a vanilla SVD over a dense matrix. As for the most cases the input dataset of ratings is vary sparse and your U/I rating matrix will have most of values set to 0.0. So, when SVD is done for input date the final recommendation will suffer of that.

And this was the main breakthrough in the publication of using SGD for MF in Netflix Prize contest in 2006. All is described by the author here. In short - you calculate your lost function only over the seen ratings set, not over the all matrix (as all the zeros or other value actually represent unknown/unobserved cases). Thus, the natural choice is to use sparse matrix for your input data.


How do you set sigma for the Gaussian similarity kernel?

Updated Answer

According to a reference paper in Spectral Clustering (von Luxburg) the $\sigma$ is simply set to 1. A further tuning can be applied with some visualization inspection but I did not find any discussion regarding setting this parameter.

Using code snippet below you see the effect:

import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import pdist

def gausker(x1,x2,sigma):
    dist = np.linalg.norm(x1-x2)
    return np.exp(-dist**2/(2.*(sigma**2.)))

data = np.array([[0,0],[1,1],[1,0],[0,1],[10,10],[10,9],[9,10],[9,9]])
plt.figure()
plt.plot(data[:,0],data[:,1],'o',ms=20)
plt.show()
s = np.std(pdist(data))
for sigma in [1, s, 10, 100]:
    gaus = np.zeros((8,8))
    for ii in range(8):
        for jj in range(8):
            gaus[ii,jj] = gausker(data[ii,:],data[jj,:],sigma)
    plt.figure()
    plt.imshow(gaus,extent=[0, 1, 0, 1])
    plt.colorbar()
    plt.title(str(sigma))
    plt.show()

For Machine Learning algorithms is better to have more distinction. The Gaussian similarity kernel cares about local similarities. The image show the kernel for $\sigma=1$

Conceptually is similar to a k-nearest neighbors graph, since it considers local neighborhood and almost disregards the relationship between two nodes far apart.


As mentioned in the comments, your data doesn't have that much of an imbalance. A good way to check how the imbalance is affecting your model, is by calculating precision and recall. These values will be quite low while your accuracy will shoot high because it will predict the majority class for every sample. However the 'maybe normal' case is dangerous. It would be better to set a different threshold , probably a lower one so that even the marginally abnormal cases will be classified as abnormal. A precision recall curve or a ROC curve is a good way to decide on the cutoff.


Credit card fraud detection - anomaly detection based on amount of money to be withdrawn?

What's the underlying model of how much someone requests from an ATM? It doesn't seem like it's a simple distribution like a Gaussian, where comparing new amounts to the mean is sensible. Consider a person who always pulls out either \$40 or \$400. Ideally we want to build a distribution of what normal transactions from a user look like, and notice if new datapoints don't look like they're sampled from that distribution.

idclark's suggestion, to look at the nearest n datapoints from that user and compute the distance from just them, is a good and fast implementation of that sort of test.

One other possibility is to try to find similar users, and then aggregate data across users. If I only have 10 withdrawals from each user, I'm not going to be able to reject any new withdrawals with confidence, but if I have seven clusters of users, with a thousand withdrawals per cluster, I can notice when a user who was in a particular cluster deviates from the overall cluster distribution. (This also helps you make use of knowledge about which previous transactions were fraudulent.)


Predicting object by features probabilities

What you are attempting to do is so much like the Bayes decision theory; you can find the math behind it here. In probability theory and statistics, Bayes’ theorem (alternatively Bayes’ law or Bayes' rule, also written as Bayes’s theorem) describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For example, if cancer is related to age, then, using Bayes’ theorem, a person’s age can be used to more accurately assess the probability that they have cancer, compared to the assessment of the probability of cancer made without knowledge of the person's age. One of the many applications of Bayes' theorem is Bayesian inference, a particular approach to statistical inference. When applied, the probabilities involved in Bayes' theorem may have different probability interpretations. With the Bayesian probability interpretation the theorem expresses how a subjective degree of belief should rationally change to account for availability of related evidence. Bayesian inference is fundamental to Bayesian statistics.

Take a look at here and here too.


Most Efficient Machine Learning Algorithm for Text Analysis

I don't have any reference right now, but you should find a lot of papers about it on Google Scholar. For all I know, CNN and RNN (mostly LSTMs) are the two best types of models for sentiment analysis. You will even find some CNN+LSTM architecture that work really well on this kind of tasks.


A simplified example: suppose two companies: the webpage owner and a book store. And a customer named Jane, interested in reading.

The web page owner has some free space on its page, which can be sold for advertisement. The book store wants to place an advertisement on the same web page, in order to increase sales. Both companies meet eachother at an auction, where the web space is sold to the highest bidder in real-time.

So, the book store is bidding on the right to show an advertisement to Jane, who is visiting the page of the webpage owner.

The machine learning is done on the part of the book store, who receives information about the web page visitor. This can be all sort of information that the web page owner wants to release, and that could be of any use to the bookstore.

Based on this information, it is decided weather to make a bidding for the advertisement space or not, and the amount up to which the book store will want to make a bidding.

Without cookie matching, the webpage visitor Jane will probably not be identifiable by the bookstore, so the store must decide on bidding, based on parameters like geographical location of the customer, browser version (just to name a few ).

With cookie matching, each visitor/customer gets a unique identifier at the book store. Based on this identifier, the book store has more information of the visitor, like: what ads have been served before, and how long ago ? This visit to the web page can be linked to earlier visits and this information will likely ease the decision making process for doing the bidding on the advertisement space.

( There is more to it as there can be two more intermediary companies: one that holds the auction and one that delivers the ad )


Use edgeCol and nodeCol:

library("arulesViz")
data(Groceries)
rules <- apriori(Groceries, parameter=list(support=0.005, confidence=0.5))
plot(rules, method="graph",
     control=list(nodeCol="yellow", edgeCol="blue", type="items"))


Building a tag-based recommendation engine given a set of user tags?

One solution could be to train a single embedding space, StarSpace is one such implementation. That single embedding space would contain all users, documents, and tags. Then it is a nearest neighbor search to recommend any combination. Given a user, find the nearest documents. Given a tag, find the nearest tags …

For new entities (i.e., users, documents, or tags), split the individual entity into parts. For example, a document will have tokens or tag will be associated with documents that have tokens. Then find the average embedding of all the parts. That location in the embedding space is the approximately semantic meaning of new entities.

Overall, this is a complex, open-end problem, thus there are many possible ways to create useful solutions. The next best solutions depend on what has already been done and what the next feature that would add the most value to end-users.


Understanding LSTM Training and Validation Graph and their metrics (LSTM Keras)

From visually inspecting the graph, we see that the validation loss and accuracy has improved with each epoch - with the training loss and accuracy higher than that of validation. This indicates that accuracy has improved with training.
As suggested in another post, one potential solution is to calculate the exponential moving average of the validation loss to remove the oscillations and better determine the improvement in this metric.
If you are finding that the test accuracy is higher than that of training, this might suggest underfitting. This could imply that more training on your model is required, or has been over-regularized.


How to deal with word length variability while using char-level one-hot encoding?

Your padded vector should be OK as a starting model. You could take it a step further and have a specific "no character" encoding, as that will be a clearer learning signal.

You say that you are using a feed-forward neural network. In that case, there is a possible change and improvement to your model, which is to use Recurrent Neural Network (RNN), probably a long short-term memory (LSTM) variation, because that has had some success in language models.

In a RNN, instead of having the character encoding repeated N times, you have it represented just once, and run the network N times for each sample, feeding in each character in turn. Instead of a "no character" symbol, you would have an "end of sequence" symbol. For an open-ended model with no specific target, it is common to have the output of the network be a prediction of the next character. This allows you to explore what the model has learnt by sampling from its predictions - a lot like using n-grams to generate realistic-looking words.

The advantage of a RNN for character-level language modelling is that the model is built around the assumption that each input is somehow equivalent - the same encoding is used at each time step. This is clearly true for character strings, so it can lead to better models from less training data.

The main disadvantage of using a RNN is that it is more complex to understand, build and train such a model.


Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model  then  model.transform(test) contains a node of probability, for more details please check the below code, going to show you the probability node and others useful nodes also for iris dataset.

Partition dataset randomly into Training and Test sets. Set seed for reproducibility

(trainingData, testData) = irisdf.randomSplit([0.7, 0.3], seed = 100)

trainingData.cache()
testData.cache()

print trainingData.count()
print testData.count()

Output:

103
47

Next, we will use the VectorAssembler() to merge our feature columns into a single vector column, which we will be passing into our Naive Bayes model. Again, we will not transform the dataset just yet as we will be passing the VectorAssembler into our ML Pipeline.

from pyspark.ml.feature import VectorAssembler
vecAssembler = VectorAssembler(inputCols=["SepalLength", "SepalWidth", "PetalLength", "PetalWidth"], outputCol="features")

For iris dataset, it has three classes namely setosa, versicolor and virginica. So let's create a Multiclass Naive Bayes Classifier using pysaprk library ml.

from pyspark.ml.classification import NaiveBayes
from pyspark.ml import Pipeline

# Train a NaiveBayes model
nb = NaiveBayes(smoothing=1.0, modelType="multinomial")

# Chain labelIndexer, vecAssembler and NBmodel in a pipeline
pipeline = Pipeline(stages=[labelIndexer, vecAssembler, nb])

# Run stages in pipeline and train model
model = pipeline.fit(trainingData)

Analyse the created mode model, from which we can make predictions.

predictions = model.transform(testData)
# Display what results we can view
predictions.printSchema()

Output

root
 |-- SepalLength: double (nullable = true)
 |-- SepalWidth: double (nullable = true)
 |-- PetalLength: double (nullable = true)
 |-- PetalWidth: double (nullable = true)
 |-- Species: string (nullable = true)
 |-- label: double (nullable = true)
 |-- features: vector (nullable = true)
 |-- rawPrediction: vector (nullable = true)
 |-- probability: vector (nullable = true)
 |-- prediction: double (nullable = true)

You can also select a particular node to view for some dataset as:

# DISPLAY Selected nodes only
display(predictions.select("label", "prediction", "probability"))

Above will show you in tabular formate.

Reference:

spark
Models using pipeline
https://mike.seddon.ca/natural-language-processing-with-apache-spark-ml-and-amazon-reviews-part-1/
https://stackoverflow.com/questions/31028806/how-to-create-correct-data-frame-for-classification-in-spark-ml


Structure of Convolutional Neural Network to analyze a sequence of frames

I'm not sure if I have to use a TimeDistributed layer or not

You definitely don't have to use TimeDistributed. You have other choices, that may be equally valid, depending on your data:

Flatten your example data into 81920 features per example and use a simple Dense layer.
Use a Conv3D layer.
Use some form of RNN, such as LSTM.

From your data description, I would expect either a TimeDistributed or 3D CNN based approach to be a good first bet. Intuition suggests that the CNN would work better if there was minor change between frames (because it has capability to directly find subtle frame differences), the TimeDistributed approach would work better processing larger changes (because it will ignore frame differences until the fully-connected layer).

  Should I use a TimeDistributed layer?

Only you can answer this, by trying it and measuring performance of your classifier. However, it should function correctly, and intuition suggests it would be a good choice if your frames are in sequence but visually disjoint.


Deep Neural Network and Activation Function

Activation function is an essential part of hidden layers in DNN. Without it the network won't learn any other than a linear equation as a result and usage of DNN is meant to find more complex patterns than linear. There is a mathematical proof that the network reduces to include one weight instead of a weight matrix when no Activation function is present, thus the importance is obvious and huge.
Activation function sort of makes the DNN to test combinations of hidden layer values and that way it helps the network to converge towards the dataset form when compared to validation data and updated accordingly.
The meaning is the same as a steering wheel for a car: without it the brightest car is useless, it would apply for straight roads only if any (self-driving cars excluded).
That is a simplification of the concept, hope you got something out of it.
Very much more complex and thorough answers included in What is the purpose of an activation function in neural networks? like given by Fatemeh in comments. There you get more points on why and how the Activation function is needed and used.


How do I load FastText pretrained model with Gensim?

Here's the link for the methods available for fasttext implementation in gensim fasttext.py

from gensim.models.wrappers import FastText

model = FastText.load_fasttext_format('wiki.simple')

print(model.most_similar('teacher'))
# Output = [('headteacher', 0.8075869083404541), ('schoolteacher', 0.7955552339553833), ('teachers', 0.733420729637146), ('teaches', 0.6839243173599243), ('meacher', 0.6825737357139587), ('teach', 0.6285147070884705), ('taught', 0.6244685649871826), ('teaching', 0.6199781894683838), ('schoolmaster', 0.6037642955780029), ('lessons', 0.5812176465988159)]

print(model.similarity('teacher', 'teaches'))
# Output = 0.683924396754


Predict the outcome of a new time series (mailing)

Think about this problem in two parts.

Part 1)
How many samples do you need in order to have the desired confidence in the Null hypothesis?

Part 2) How long does it take to get that many samples?

Part 1 can be answered with A-priori power analysis.  Use this technique to determine the sample size needed to get the p-value desired.

Part 2 can be addressed with an algorithm applied to the data you have collected thus far.  Your predictors in this case would be:

Column 1: the probability that your statistic of interest has leveled off (is not changing significantly with additional input).

Column 2: date/day/time start. You probably will need to codify this.

Column 3: number of samples at finish.  These should be the number which you determined from your prior analysis.

Your response/output in this case would be:

Column 4: the date/day/time finish.

Of course would need to do some data analysis to determine the appropriate algorithm to use on this data set.  The end product should be a function that takes as input a target probability, date/day/time start and a target sample size, while outputting a date/day/time finish.

I hope this at least gives you some ideas on how to proceed.


Calculating mean of data frame inside a series object

In the absence of a MultiIndex (the Right Way$^\mathrm{TM}$), the apply method can do what you want; e.g.

df.assign(
    max_retweet=df.tweets.apply(lambda x: x.retweet_count.argmax('retweet_count')),
    avg_retweet=df.tweets.apply(lambda x: x.retweet_count.mean())
)

       avg_retweet  max_retweet
26662     0.045476          187
32316     0.821538          427
25879     0.633681          583
43411     0.112465          421
28840     0.472222            8

Also, don't share data with pickle; it's dangerous.


Smallest Possible Dataset for Text Classification using BERT

Sorry, but there’s no rule and amount we are able to quantify.

I’ve used it (multilingual) for 700 texts with a 20 multilabel classification and I had worse results than with a custom deep net (but with pretrained word embeddings).

But you know, in fact these questions are hard to answer. Why? Because with very low quantities of data, you can’t accurately assess the performance of a model - it is statistically insignificant.


Clustering customer data stored in ElasticSearch

One algorithm that can be used for this is the k-means clustering algorithm.

Basically:

Randomly choose k datapoints from your set, m_1, ..., m_k.
"Until convergence":

Assign your data points to k clusters, where cluster i is the set of points for which m_i is the closest of your current means
Replace each m_i by the mean of all points assigned to cluster i.

It is good practice to repeat this algorithm several times, then choose the outcome that minimizes distances between the points of each cluster i and the center m_i.

Of course, you have to know k to start here; you can use cross-validation to choose this parameter, though.


ngram and RNN prediction rate wrt word index

Recurrent Neural Network (RNN) create a single state vector over time. Thus that curve is to be expected. Initially, the state vector does have enough information to make a quality prediction. Then quickly reaches asymptotic performances. Overall, the predictions are between 15% and 22% correct.
The shape of the graph might be a function of sentence length in the training corpus. Possibly sentences could be between 3 and 7 words long. The drop could be because there is less training data for longer sentences.


I have mentioned this in other posts also:

One can use Conv1d of Keras for usual features table data of shape (nrows, ncols). To input features, following 2 steps are needed:

xtrain.reshape(nrows, ncols, 1)
# For conv1d statement:
input_shape = (ncols, 1)

For example, taking first 4 features of iris dataset:

To see usual format and its shape:

iris_array = np.array(irisdf.iloc[:,:4].values)
print(iris_array[:5])
print(iris_array.shape)

The output shows usual format and its shape:

[[5.1 3.5 1.4 0.2]
 [4.9 3.  1.4 0.2]
 [4.7 3.2 1.3 0.2]
 [4.6 3.1 1.5 0.2]
 [5.  3.6 1.4 0.2]]

(150, 4)

Following code alters the format:

nrows, ncols = iris_array.shape
iris_array = iris_array.reshape(nrows, ncols, 1)
print(iris_array[:5])
print(iris_array.shape)

Output of above code data format and its shape:

[[[5.1]
  [3.5]
  [1.4]
  [0.2]]

 [[4.9]
  [3. ]
  [1.4]
  [0.2]]

 [[4.7]
  [3.2]
  [1.3]
  [0.2]]

 [[4.6]
  [3.1]
  [1.5]
  [0.2]]

 [[5. ]
  [3.6]
  [1.4]
  [0.2]]]

(150, 4, 1)

This works well for Conv1d of Keras. For input_shape (4,1) is needed.


NLP problem : Choosing the optimal parser for extracting quantitative values from text

This is an NER problem. Rather than you splitting your sentence to words and finding the right word from dict, I suggest you use an NER (may be spacy NER mentioned by @jindrich).

This NER will point out right block of information from the text your sentences.

Once you get an Entity then you can parse its value. If it is quantitative then it is easy to parse (with simple preprocessing) and if it is qualitative then you may have to convert string to numbers like one to 1 . There are free libraries for that.


How to explain MAE/MSE at each node of decision tree for regression in sklearn python?

Can I conclude that the error at my node is 60 +-13 i.e my values in this particular sample split ranges from 60-13 to 60+13.

No you cannot, because the actual error values depend on the data. For example you might have 1 instance with error 41.11 and 9 instances with error 0:
$$MSE=\frac{41.11^2+0^2+...+0^2}{10}=169$$
This example shows that the only guarantee you have for an individual absolute error is that it's lower or equal than $\sqrt{MSE \times n}$.
The MAE (mean absolute error) is easier to interpret: it's literally the mean of the absolute value of the error.


Max-pooling vs. zero padding: Loosing spatial information

Pooling (max/mean/etc) has two primary benefits: it significantly reduces computational complexity (at the cost of potentially important data) and it helps the network achieve spatial invariance by making the spatial relativity between features less relevant.

However, the spatial invariance achieved through pooling is not always ideal.    For instance, given a CNN trained to identify images of automobiles, it would probably classify an image containing a hood/trunk, tires, windows, car doors, etc. as an automobile, regardless of the components' relative positioning.

On the other hand, I do not know of significant costs to using padding.  Zero padding is critical for deep networks; without it, our volumes would rapidly collapse through the layers.  It therefore helps to maintain desirable volume sizes and to preserve the border data.  It also helps the geometry of CNNs to work out smoothly by preventing mismatched dimensions between layers.

You might find this paper interesting.  Hinton and his team propose a new network called Capsule Network (CapsNet) that addresses the problems with pooling and provides an alternative solution for acquiring spatial invariance.  One of the many exciting parts of their algorithm is that the location of features relative to one another is a primary asset in the classification process, bringing computers closer to a "true understanding" of objectness.


Dividing a dataset to parallelize machine learning training on the cloud

There are many ways to parallelism machine learning. It is often better to distribute the model parameters, not the data.
Training models only a subset of data will result in worse parameter estimates than training a model on random samples of the data.
Additionally, moving data around is more expensive than moving parameters.


How do you, analytically, show you are not using too many features?

My approach would be quite similar to yours with a baseline, maybe just a bit more general:
Assuming it's practical (i.e. the training process is not too costly), you could train/test multiple models with various number of features, e.g. 10%, 20%...,100% of the features. For every subset you use an appropriate feature selection method, preferably something like genetic algorithm but it might be too costly. If it's not possible a simple individual feature selection method, e.g. using information gain, but it's likely not to be optimal. Once all the models have been trained, tested and evaluated, a plot of the performance as a function of the number of features should (hopefully) show a curve which increases less and less but never reaches the plateau. If it does reach the plateau before your chosen number of features, it means that you could actually cut down the number features.


Convolutional Network for Text Classification

This wildml blog post has a very clear explanation of how to use 1D convolution on text. And Debo, DS at x.ai, provided some example Keras code to classify text using a character-based model (input documents are sequences of one-hot encoded characters rather than words or POS tags):

from keras.models import Model
from keras.layers import Input, Dense, Dropout, Flatten
from keras.layers.convolutional import Convolution1D, MaxPooling1D

inputs = Input(shape=(maxlen, vocab_size), name='input', dtype='float32')
conv = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[0],
                     border_mode='valid', activation='relu',
                     input_shape=(maxlen, vocab_size))(inputs)
conv = MaxPooling1D(pool_length=3)(conv)
conv1 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[1],
                      border_mode='valid', activation='relu')(conv)
conv1 = MaxPooling1D(pool_length=3)(conv1)
conv2 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[2],
                      border_mode='valid', activation='relu')(conv1)
conv3 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[3],
                      border_mode='valid', activation='relu')(conv2)
conv4 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[4],
                      border_mode='valid', activation='relu')(conv3)
conv5 = Convolution1D(nb_filter=nb_filter, filter_length=filter_kernels[5],
                      border_mode='valid', activation='relu')(conv4)
conv5 = MaxPooling1D(pool_length=3)(conv5)
conv5 = Flatten()(conv5)
z = Dropout(0.5)(Dense(dense_outputs, activation='relu')(conv5))
z = Dropout(0.5)(Dense(dense_outputs, activation='relu')(z))

pred = Dense(n_out, activation='softmax', name='output')(z)
model = Model(input=inputs, output=pred)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop',
              metrics=['accuracy'])

The last 3 lines are important. You can't use softmax on your output and you can't use 'categorical_crossentropy' for multi-label tagging (your problem). Your text tagging problem should be broken down into multiple binary classification problems, or you need to use a different loss function like 'binary_crossentropy'. And for binary_crossentropy, use a sigmoid activation function rather than softmax on the output. See this SO answer for details on multi-label tagging problems in keras and TF.

If you want a more thorough explanation, check out Chapter 7 in my book, NLP In Action.


What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?

By hidden states, they mean outputs of the layers, i.e., what you get after the feed-forward sub-layer. For Transformer-XL, it is important that these are also what you use as an input to the self-attention. Therefore, at inference time, if you want to compute the states recursively by segments (presumably because you cannot fit the entire input int he memory), this is the only thing you need to remember from the previous steps to continue the computation.
There is no encoder, you can imagine Transformer-XL as a decoder-only model. Transfering the states just means remembering them, so you can do the self-attention over them, but you can no longer back-propagate through them because you only remember the values and the entire graph telling how you got them.
The dots in the scheme correspond to the hidden states: one state per input subword and per layer. The lines between them are the self-attention links.


Linear regression : ValueError: operands could not be broadcast together with shapes (3,) (1338,)

It looks like you're trying to multiply a matrix and a vector point-wise. Such an operation is not defined. I think you should use X.dot(w), where X is a feature $\bf matrix$ and w is the wights $\bf vector$. np.dot can operate with objects of different nature (like matrices and vectors). So in h I'd write return X.dot(w). Also, I'd call it only with a matrix X (for consistency), even if there's an object it would be $X \in \mathbb{R}^{1 \times d}$ in that case. Watch carefully after shapes of the objects which enter functions.


Python First Project: Algorithm that find a book and its author from a quote

This is not really a data science project. It's more a programming problem as you are trying to build a program with multiple functions to gather data, sort data, etc.

Data science is about finding, researching, analyzing relationships within data itself. Data scientists don't work on APIS, neither SQL or other data sources (but it doesn't mean they don't know about it).

https://stackoverflow.com/ should be a right place for your question, specially if you have some code. I am sure they will help you there with Google API


What exactly are the data augmentation experimental Keras' layers doing?

if you look into the code you can figure what's exactly happening.
Take tf.keras.layers.experimental.preprocessing.RandomRotation for example
def call(self, inputs, training=True):
  ...
  def random_flipped_inputs():
    flipped_outputs = inputs
    if self.horizontal:
      flipped_outputs = image_ops.random_flip_left_right(flipped_outputs,
                                                         self.seed)
    ...
    return flipped_outputs

You will see it's gonna to random_flip_left_right on your input images, and the comments inside it is
"""Randomly flip an image horizontally (left to right).
With a 1 in 2 chance, outputs the contents of `image` flipped along the
second dimension, which is `width`.  Otherwise output the image as-is.
When passing a batch of images, each image will be randomly flipped
independent of other images."""
Example usage:
...
>>> images = np.array(
... [
...     [[[1], [2]], [[3], [4]]],
...     [[[5], [6]], [[7], [8]]]
... ])
>>> tf.image.random_flip_left_right(images, 6).numpy().tolist()
[[[[2], [1]], [[4], [3]]], [[[5], [6]], [[7], [8]]]]

So basically what it is doing is each image will have a 50% chance to horizontally be flipped, so if you have 10 epochs, image_1 will have possibility to be [original, flipped, original, flipped, ...], and so on so forth to other images. Seeing from the outside would identify we have 10 different datasets(particularly on multiple data augmentation).


Abbreviation in Orange's contingency table

ARI stands for adjusted Rand index and AMI stands for adjusted mutual information.

They are metrics for clustering.

Remark:

ARI might take negative values.
The AMI takes a value of 1 when the two partitions are identical and 0 when the MI between two partitions equals the value expected due to chance alone.


Is this enough information to fully define a neural network?

Yes.

The neural network mentioned in this paper is simple and straightforward: it is a fully connected network with 2 hidden layers.
 The authors specified the network structure, the number of neurons in each layer, and the activation functions in each layer and the final loss function.

Given those information, the network is fully specified. For instance, you can create such a network using

sklearn.neural_network.MLPRegressor(hidden_layer_sizes=(50,7), activation=‘logistic’)


clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure

I hope I followed your question correctly. If you have those data points on as 2D vectors, that would mean, that you have, say $N=50,000$ data points each represented as $\left(x_i,y_i\right)$ , right? If angle and length are what you are concerned about, then your formulation seems right. You can convert each data point $\left(x_i,y_i\right)$ into $\left(\theta_i, d_i\right)$, which are the angle and length of the vector. These two parameters $\left(\theta_i, d_i\right)$ act as two features in each of your samples and you can run k-means on this. You have to be careful about using a consistent measure on your angle (always anti clockwise or clockwise). Your dissimilarity measure seems quite correct as far as I can tell. I'm sure you are aware of this python package but just for the sake of completion you can use  this or in MATLAB you can use this. I'm not sure about R, so pardon me if that was the essential part of your question.


What's the difference between Haar-like Features and Haar Cascades?

Haar-like features are, as the name says, features. They are basically some filters, just like the CNN filters. The main difference is that CNN learns the filters by training, while Haar filters are hand designed. These filters are called Haar-like features due to their similarity with Haar wavelets.

Haar cascades are a bunch of Haar-like features arranged into a classifier.


I will rephrase you setup to make sure I understood it correctly.

  You have two time series $x = (x_1, x_2, .., x_n)$ and $y = (y_1, y_2,..,y_n)$, where $n=4500$. You want to understand if $x_t = y_{t-1}$, where $t \in [1,n]$ and by $=$ means approximately equal in statistical sense.

There is one way but you have to make some assumptions.

In general a time series can be thought as a signal plus additional noise. The signal is an explicit function which models some phenomenon and the noise is some alteration of the output which is random by nature. Using this model you split $x_t$ and $y_{t-1}$ into something like:

$$x_t = s^{(x)}_t + w^{(x)}_t$$
$$y_{t-1} = s^{(y)}_{t-1} + w^{(y)}_{t-1}$$

where $s^{(x)}_t$ is the signal of $x$, $w^{(x)}_t$ is the noise of $x$, respectively $s^{(y)}_{t-1}$ is the signal of $y$, and $w^{(y)}_{t-1}$ is the noise in $y$. Notice that $s^{(x)}_t$ and $s^{(y)}_{t-1}$ are well determined and noises are random.

Using this notation the idea of approximately equal in statistical sense means that the signal is the same and if there are difference in samples, than the differences are caused by random components (aka. noises). This means that if you assume that signals are the same and remains only the noise. So what your null hypothesis is $H_0 : x_t-y_{t-1}=0$ can be written as:
$$H_0 : s^{(x)}_t + w^{(x)}_t - s^{(y)}_{t-1} - w^{(y)}_{t-1} = 0$$
$$H_0 : w^{(x)}_t - w^{(y)}_{t-1} = 0$$

Now here comes the assumptions. You have two sorts of random variables (noises) which pose some difficulties when you are trying to apply tools based on central limit theorem.

One problem is that the random noises depends on time. You have to assume that the noise does not change with time, in both series. This is translated as independent and identically distributed or ($w^{(x)}_t = w^{(x)}$ and $w^{(y)}_{t-1} = w^{(y)}$).

So, if you assume that noises are independent and identically distributed your question can be worked out with a paired difference test. This means you can test if the differences depart significantly from $0$.

You can use a t-test dependent t-test for paired samples if you additionally assume that the difference is normally distributed. You cannot use z-test on paired differences because you do not know the variance of the difference's distribution (when you use sample variance for variance in a z-test, you fall into a t-test).

If you do not want to assume that the differences are normally distributed (that can be checked graphically at least with histograms or qq plots for skewness) you can use Wilcoxon signed-rank test, which should be fine for the number of samples you have. I would pick this test, for the same reason.

Attention: I restate again that those tests are valid if an only if the noises in both series are independent of time. Also you can test only if they different. You cannot prove that they are equal, you can only state that you do not have evidence that they are different, which for most cases should be enough.


Let's look at the layers before the reshaping stage since everything after that is simply a densely connected neural network.

Backpropagation in Max Pooling

Max pooling takes a window of values and only the maximum value passes through. This means that the error can only be contributed by the maximum values, thus only the weights for these values will be updated.

Backpropagation in the Convolutional Layers

This is the same as for the densely connected layer. You will take the derivative of the cross-correlation function (mathematically accurate name for convolution layer). Then use that layer in the backpropagation algorithm.

An Example

Let's look at this following example

Forward pass

The forward pass of the convolutional layer can be expressed as

$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$

where in our case $k_1$ and $k_2$ is the size of the kernel, in our case $k_1=k_2=2$. So this says for an output $x_{0,0} = 0.25$ like you found. $m$ and $n$ iterate across the dimensions of the kernel.

Backpropagation

Assuming you are using the mean squared error (MSE) defined as

$E = \frac{1}{2}\sum_p (t_p - y_p)^2$,

we want to determine

$\frac{\partial E}{\partial w^l_{m', n'}}$ in order to update the weights. $m'$ and $n'$ are the indices in the kernel matrix not be confused with its iterators. For example $w^1_{0,0} = -0.13$ in our example. We can also see that for an input image $H$x$K$ the output dimension after the convolutional layer will be

$(H-k_1+1)$x$(W-k_2+1)$.

In our case that would be $4$x$4$ as you showed. Let's calculate the error term. Each term found in the output space has been influenced by the kernel weights. The kernel weight $w^1_{0,0} = -0.13$ contributed to the output $x^1_{0,0} = 0.25$ and every single other output. Thus we express its contribution to the total error as

$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \frac{\partial E}{\partial x^l_{i, j}} \frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}}$.

This iterates across the entire output space, determines the error that output is contributing and then determines the contribution factor of the kernel weight with respect to that output.

Let us call the contribution to the error from the output space delta for simplicity and to keep track of the backpropagated error,

$\frac{\partial E}{\partial x^l_{i, j}} = \delta^l_{i,j}$.

The contribution from the weights

The convolution is defined as

$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$,

thus,

$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = \frac{\partial}{\partial w^l_{m', n'}} (\sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l)$.

By expanding the summation we end up observing that the derivative will only be non-zero when $m=m'$ and $n=n'$. We then get:

$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = o^{l-1}_{i+m', j+n'}$.

Then back in our error term

$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \delta_{i,j}^l o^{l-1}_{i+m', j+n'}$.

Stochastic gradient descent

$w^{(t+1)} = w^{(t)} - \eta \frac{\partial E}{\partial w^l_{m', n'}}$

Let's calculate some of them

import numpy as np
from scipy import signal
o = np.array([(0.51, 0.9, 0.88, 0.84, 0.05),
              (0.4, 0.62, 0.22, 0.59, 0.1),
              (0.11, 0.2, 0.74, 0.33, 0.14),
              (0.47, 0.01, 0.85, 0.7, 0.09),
              (0.76, 0.19, 0.72, 0.17, 0.57)])

d = np.array([(0, 0, 0.0686, 0),
              (0, 0.0364, 0, 0),
              (0, 0.0467, 0, 0),
              (0, 0, 0, -0.0681)])

gradient = signal.convolve2d(np.rot90(np.rot90(d)), o, 'valid')

  array([[ 0.044606,  0.094061], [ 0.011262,  0.068288]])

Now you can put that into the SGD equation in place of $\frac{\partial E}{\partial w}$.

Please let me know if there are errors in the derivation.


Confidence Score For Trained Sentiment Analyser Model

I was able to solve it myself after some further research.
I will be briefly describing my approach here.
Cheers!

The idea is to find the confidence interval which was also same as finding the distance from the decision boundary / hyper-plane, in my case.

If you are using the Scikit Learn API, there is a method called predict_proba() for several classification models like Logistic Regression, SVM, Random Forest, etc. If your classifier does not provide one, you can wrap it with the CalibratedClassifierCV which can be found in sklearn.calibration, then use the above method to calculate the distance from the decision boundary.

If you are looking for custom in-depth implementation, here are some papers / references that might help.

Ref - 1
Ref - 2


1x1 Convolution. How does the math work?

Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).

I don't use torch, but keras, but the principle applies I think.

When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.

The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!

So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.

And that's the shrink suggested by Ng.

Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.

Edit: Simple example

I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.

import numpy as np

M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])

C = np.ones(shape=(2, 2))

c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T)


Look for previous date in dataframe that has certain column category in R

If I understand you correctly, you want to retrieve the date of the previous same day category, same holiday class, of the same hour.

If it is the case, perhaps you can try:

1) Segment rows within the same hour, day category, and holiday group into the same cluster (or filter, etc.)

2) For each cluster, Rank the Date (first day in the group is 0, second = 1, ... nth day = n - 1, etc.) from the lowest, etc.

3) For each Rank r, assign the PreviousDaySameDayCategory by looking up the Date for row r - 1 for r - 1 >= 0 within the same cluster

It is rather straightforward to achieve with dplyr. Please let me know if the following makes sense to you.

(For some reasons, I am not able to run your solution)

library(dplyr)
rankedDf <- mydf %>%
  group_by(DayCategory, Hour, Holidays) %>%
  arrange(DayCategory, Hour, Holidays, Date) %>%
  mutate(rowRank = order(Date), previousRowRank = order(Date) - 1) %>%
  left_join(., ., by = c("previousRowRank" = "rowRank", "DayCategory", "Hour", "Holidays")) %>%
  select(
    Date.POSIXct = Date.POSIXct.x,
    Date = Date.x,
    WeekDay = WeekDay.x,
    DayCategory,
    Hour,
    Holidays,
    myvalue = myvalue.x,
    PreviousDaySameDayCategory = Date.y,
    PreviousValueSameDayCategory = myvalue.y
  )
print.data.frame(rankedDf, nrow = 500)

Note: I also include DayCategory, etc. in the arrange so that it's easier to understand how it works; the code works just fine with only Date as the parameter.