Which features can help to differentiate these two density?

Regarding two probability distributions $A,B$ on the line, that have densities, the following are equivalent:

$ A = B$ (i.e.: given any Borel set $\mathscr S$, the probability assign to $\mathscr S$ by A equals the probability assign to $\mathscr S$ by B)
$A(-\infty,a] = B(-\infty,a] $, for any $a \in \mathbb R$, i.e.: $A,B$ have the same cumulative distribution function
$A,B$ have the same densities.
In the case of your question, the two densities are clearly different, so the two probabilities are different.

From your question, it seems you want something you can measure (perhaps on sample from these distributions). There are infinitely many quantities that you can measure that when apply to $A$ and $B$ produce different results. The statement above gives some hints for a possible solution.

From your graphs, calling $A$ the first pdf, and $B$ the second:
$$ F_A(0.35) = A(-\infty, 0.35] = \text{ probability that an observation from the first distribution is < 0.35 } = 0 $$
since the density is 0 to the left of 0.35. On the other hand,
$$ F_B(0.35) = B(-\infty, 0.35] = \text{ probability that an observation from the second distribution is < 0.35 } > 0 $$
since the density is positive from -0.2 to 0.35.

We can a little to the right of 0.35 and find a number $a_0$ such that $0 < F_A(a_0) < F_B(a_0) $.

Being less than $a_0$ is something one can measure on samples.

Let $A_1, .., A_n$ be an iid sample from the first distribution, and let $B_1,...,B_m$ be an iid sample from the second distribution. Then $x_j = I(A_j \le a_0)$ is a measure (Note: I am using the notation $I(S)$ = indicator function of a set $S$, or, in more typical notation in the context of probabilitie, $I(S)$ = indicator function of event $S$ = 1 if $S$ is true, 0 if $S$ is false).

Likewise, one can measure $y_j = I(B_j \le a_0).$

$\{x_j\}$ are iid Bernoulli, with $P(x_j=1) = P(A_j \le a_0) = F_A(a_0).$
$ \frac{1}{n} \sum_1^n x_j $ is an estimator of $F_A(a_0)$

$\{y_j\}$ are iid Bernoulli, with $P(y_j=1) = P(B_j \le a_0) = F_B(a_0).$
$ \frac{1}{m} \sum_1^m y_j $ is an estimator of $F_B(a_0)$

Now one can run a rest of the hyposesis that the tw means are the same.

Before proceeding, does this construction answer your question?


There are a number of ways to tackle this, I am going to focus on feature selection/extraction, because you mentioned PCA.
Sklearn itself offers a few feature selection/extraction algorithms already, see here, like SelectKBest. This would mean for you to maybe select specific frames, or samples, or even pixels (unlikely).
Further it has not only PCA but a few more, see here. I am going to mention PCA, NMF, ICA. While you already apparently tried this, it is important to note that these algorithms als have to be tuned correctly.
Now, on the other hand, like Graph4Me already mention you can use a CNN. For this you can use the structure of an AutoEncoder, which tries to learn a minimal representation of the input, to correctly restore the input as output. The Decoder-Encoder structure can be trained, to then only use the Encoder as a way to obtain dimensionally reduced training samples. A tutorial (for pytorch) is here, although it is for text, the same principle can be applied to images and videos.
As a final note, you can obviously try some simple preprocessing, like cropping the video, reducing the framerate, converting to grayscale if not already done, or even just black and white. I also hope you are already processing your data sequentially in every regard, if loading everything at once is an issue.


Is datawarehouse considered as datalake in big data environnment?

Check out this article: http://www.kdnuggets.com/2015/09/data-lake-vs-data-warehouse-key-differences.html

Looks like the answer is yes, your DWH can be defined as datalake because you are adding unstructured data.  Here is an excerpt:

  “A data lake is a storage repository that holds a vast amount of raw data in its native format, including structured, semi-structured, and unstructured data. The data structure and requirements are not defined until the data is needed.”


What algorithms should I use to perform job classification based on resume data?

Check out this link.

Here, they will take you through loading unstructured text to creating a wordcloud.  You can adapt this strategy and instead of creating a wordcloud, you can create a frequency matrix of terms used.  The idea is to take the unstructured text and structure it somehow.  You change everything to lowercase (or uppercase), remove stop words, and find frequent terms for each job function, via Document Term Matrices.  You also have the option of stemming the words.  If you stem words you will be able to detect different forms of words as the same word.  For example, 'programmed' and 'programming' could be stemmed to 'program'.  You can possibly add the occurrence of these frequent terms as a weighted feature in your ML model training.

You can also adapt this to frequent phrases, finding common groups of 2-3 words for each job function.

Example:

1) Load libraries and build the example data

library(tm)
library(SnowballC)

doc1 = "I am highly skilled in Java Programming.  I have spent 5 years developing bug-tracking systems and creating data managing system applications in C."
job1 = "Software Engineer"
doc2 = "Tested new software releases for major program enhancements.  Designed and executed test procedures and worked with relational databases.  I helped organize and lead meetings and work independently and in a group setting."
job2 = "Quality Assurance"
doc3 = "Developed large and complex web applications for client service center. Lead projects for upcoming releases and interact with consumers.  Perform database design and debugging of current releases."
job3 = "Software Engineer"
jobInfo = data.frame("text" = c(doc1,doc2,doc3),
                     "job" = c(job1,job2,job3))

2) Now we do some text structuring. I am positive there are quicker/shorter ways to do the following.

# Convert to lowercase
jobInfo$text = sapply(jobInfo$text,tolower)

# Remove Punctuation
jobInfo$text = sapply(jobInfo$text,function(x) gsub("[[:punct:]]"," ",x))

# Remove extra white space
jobInfo$text = sapply(jobInfo$text,function(x) gsub("[ ]+"," ",x))

# Remove stop words
jobInfo$text = sapply(jobInfo$text, function(x){
  paste(setdiff(strsplit(x," ")[[1]],stopwords()),collapse=" ")
})

# Stem words (Also try without stemming?)
jobInfo$text = sapply(jobInfo$text, function(x)  {
  paste(setdiff(wordStem(strsplit(x," ")[[1]]),""),collapse=" ")
})

3) Make a corpus source and document term matrix.

# Create Corpus Source
jobCorpus = Corpus(VectorSource(jobInfo$text))

# Create Document Term Matrix
jobDTM = DocumentTermMatrix(jobCorpus)

# Create Term Frequency Matrix
jobFreq = as.matrix(jobDTM)

Now we have the frequency matrix, jobFreq, that is a (3 by x) matrix, 3 entries and X number of words.

Where you go from here is up to you.  You can keep only specific (more common) words and use them as features in your model.  Another way is to keep it simple and have a percentage of words used in each job description, say "java" would have 80% occurrence in 'software engineer' and only 50% occurrence in 'quality assurance'.

Now it's time to go look up why 'assurance' has 1 'r' and 'occurrence' has 2 'r's.


How to determine the number of forward and backward passes in deep learning (CNN)?

Forward and backward passes of the whole dataset are called epochs. The number of epochs is a parameter of the training procedure that cannot be estimated a priori, and it depends on how low you want your training/validation errors. One approach is to train until your validation error is small enough, and as long as the number of epochs is smaller than a threshold. Another approach is early stopping: when the validation error achieves a minimum, stop training. For a broader explanation on early stopping, see this blog.

To sum up: the number of epochs is particular of every network and dataset, as well as optimization technique, and you should set them in order to have a low enough loss, but not overfit (early stopping is done in order to not overfit).


How YOLO training and prediction works for an object fall in multiple grid?

It seems after referring many documents, I found the answer of my question.

  First, likely to correct my understanding. I thought for labeling,
  bounding box size (width, height) will always be with in the range of
  particular grid dimension i.e. between 0 and 1. It's not correct and
  this is the only source of confusion atleast for me.

I believe that if I'm able to answer about the labeling process, I think it will clear most of the doubt raised in the form of question.

Steps are labeling:

Identify the center point of the image with respect to image boundary, refer RED dot.

Create a grid on top of the image.

Identify the grid where center point exist. In this case its 8th grid (Numbering start from 1).
Marked only that grid has an object(not others) and labeling should be something like this:

g8 is having object and bounding box information. bx and by are wrt to grid dimension, whereas width and height are wrt to its size, which is greater than one.

During YOLO model training, it learned this intelligence and can start predicting the height and width which can be greater than 1.
In a case if a car( or any object) falls in multiple gird and model training is perfect it should predict the height and width accordingly i.e > 1.


Maybe i'm missing something, but where is the connection between the global and local branches?

During the training of the local branch you used a dataset that was constructed from masked images. But now that branch should take as input the output of the 7x7x2048 conv layer from the global branch.

Maybe your training set constructed from coupled inputs for the 2 branches? I think you should add the implementation of your "FusionDataGenSequence".

What about your accuracy during the training?
Did you try different learning rates?


Why does OPTICS use the core-distance as a minimum for the reachability distance?

There are a few ways of looking at this. To make this easier I'm going to work describe things in terms of HDBSCAN*, a succesor to OPTICS. HDBSCAN* is essentially the same as OPTICS by removes the $\varepsilon$ parameter (or, if you like, fixes $\varepsilon = \infty$). HDBSCAN* also has a different approach for cluster extraction, but that's not overly relevant here. Thus under HDBSCAN* we define

$\text{core-dist}_\mathrm{MinPts}(p) = \text{Distance the the MinPts nearest neighbor of } p$

$\text{mutual-reachability-dist}_\mathrm{MinPts}(o, p) = \\ \qquad\max(\text{core-dist}_\mathrm{MinPts}(p), \text{core-dist}_\mathrm{MinPts}(o), \mathrm{dist}(p,o))$

much the same as OPTICS, but now we don't require any undefined values (we also include the core-distance of $o$ to ensure that mutual reachability is a metric).

One can view both OPTICS and HDBSCAN* as an extension of DBSCAN. DBSCAN takes two parameters MinPts and $\epsilon$ (note that this is a different use of epsilon than OPTICS -- if we focus on HDBSCAN* which has no such parameter we can avoid confusion: epsilon will refer only to the parameter used in DBSCAN).

HDBSCAN* generates a complete hierarchy of clusterings for a range of possible $\epsilon$ values, and thus for any fixed $\epsilon$ value you want the clustering at that level in the hierarchy to be the clustering DBSCAN would give for that $\epsilon$ value (OPTICS works similarly, but simply constrains the range of value that $\epsilon$ may take). To be in a cluster in DBSCAN you must be a core-point (I'm going to ignore border points for now); that is, the point must have at least MinPts other points within a ball of radius $\epsilon$ of itself. If a point isn't a core-point it is noise. The HDBSCAN* (and OPTICS*) reachability-distance captures this distinction by ensuring a point is not joined into a cluster until the DBSCAN $\epsilon$ value is such that the point is within the relevant distance of the other points in the cluster and the point is a core-point at that DBSCAN $\epsilon$ value. The alternative reachability-distance you suggest covers the first part, but not the second: noise points would be included in clusters as long as they were close enough -- and by transitive chaining this could result in clusters merging together prematurely via noise points.

An alternative way of looking at this is by comparing with other algorithms like Robust Single Linkage (which also produces a cluster hierarchy). Robust Single Linkage aims to improve single linkage clustering by making it more robust to noise. In single linkage at each distance-scale $r$ you form a graph by joining points with an edge if they are within $r$ of each other, and then the clustering at level $r$ of the hierarchy is simply the connected components of that graph. The problem is that single linkage is very susceptible to noise -- a point or two in the wrong place can join clusters together when perhaps they shouldn't be. Robust Single Linkage attempts to fix this by requiring a point to have at least $k$ neighbors within $\alpha r$ of it  before adding any edges to it. This discounts sparse points until larger distance-scales and makes the approach more robust to noise. It can be demonstrated that this approach provides convergence to the level set tree of the PDF that generated the data -- i.e. it does a good job of fixing the noise issues! If we fix $\alpha = 1$ and $k = \mathrm{MinPts}$ then we recover HDBSCAN*; if we fix $r \leq \varepsilon$ then we get essentially OPTICS. Again we are using core-distance to discount noise, and make the clustering more noise robust.


The preprocessing will depend on what your data is like: textual? numerical? If in whatever analysis you want to do the data is not allowed to have duplicates, you'll have to filter those out. If not, they'll get to stay, etc.
If you already know exactly what you want to do theoretically and just need to find some software, python pandas is very good. Load the data, clean it, group it, perform elementary analysis.


what exactly happens during each epoch in neural network training

You are updating your network parameters, that is, weights for fully connected layers, for the filters in the convolution operations, etc.
The hyperparameters are fixed once you start training your network. Hyperparameters are not intrinsic to the learning process and is something that the practitioner should tune carefully with GridSearch, Bayesian Optimization and Cross-Validation techniques.
You have just one loss function during training, and at each batch procesing you update your weights correcting your network and, at least theoretically, diminishing your loss function. So after the first epoch, you have reached a certain value, that will be update on the next epoch.
Think as you are on the top of a mountain, and you are climbing down, to no get tired, you count 10 steps and rest a little, after 10 steps you are not on the top again, you are going down from where you stopped, right? That is an analogy (I think it is bad, but if you understand it is ok haha).


Neural network: does bias equal to zero, is the same as, a layer without bias?

No, they are not the same:

In MLP_without_bias the bias will be zero after training, because of bias=False.

In MLP_with_bias_zero the bias is zero at initialization, but this will not prevent it from being updated during training.


Using classification of previous sample in neural networks

The approach you describe is predicting base on the features of the current step and the previous states.
If you believe that there is information on the previous states, than i would try to approach this problem in the Time Series approach.

The algorithms in this approach basically do what you request for, they predict based on the features of the current step and based on the previous states.
You can learn about the approach to understand it better in this cs231n video.

You can also look for tutorials on implementing models that fit the time-series problem.
The most popular model i know is caleld LSTM.
Here is a tutorial on implementing LSTM in keras.

I hope you will find it these helpful.
If you have any more questions or you would like more guidance, just let me know :)


Is it a good idea to use tensorflow instead of numpy for numerical approximations?

TensorFlow is a library that supports dataflow paradigm which makes it a good candidate for numerical calculations.

In general, TensorFlow data loading performance can be optimized using tf.data module. In particular, there is a concept called GPUs starvation, where GPUs are waiting for data to be loaded. GPU starvation can be minimized in TensorFlow with Pipelining, overlapping the preprocessing and model execution of a training step.

In particular, your code can be optimized by refactoring to Expectation_Value operate on vectors instead of scalars. Right now N and i are scalars and there is a for-loop that processes elements one at a time. tf.linalg.trace can take a tensor as an input so that N and i could be vectors. The function will be much faster once it operates on vectors and no longer uses a for-loop.


Query relating to Pandas Rows manipulation

Thank you for the above answer!
That definitely works. However, I found a more efficient way in terms of computation using np.rolling
df['D'] = df['A'].rolling(min_periods=1, window=3).mean()
df['B'] = np.where(df['B'].isnull,df['D'],df['B'])

np.rolling helps to compute the cumulative sum of previous n values.
np.where helps to apply some output based on a condition: syntax: np.where(condition,
value if true, value if false).
Column D can be dropped once it is used.


df['DateTime'] = df[['Year', 'Month', 'Day', 'Hour']].apply(lambda s : datetime.datetime(*s),axis = 1)

This might pop some error depending on the fact that that whether you have hour properly defined like it should be in between 0-23 , seconds in-between 0-59 etc...

Or try this

You can pass only the columns that you need to assemble.

In [33]: pd.to_datetime(df[['year', 'month', 'day']])
Out[33]:
0   2015-02-04
1   2016-03-05
dtype: datetime64[ns

]

From the docs itself,

pd.to_datetime looks for standard designations of the datetime component in the column names, including:

required: year, month, day
optional: hour, minute, second, millisecond, microsecond, nanosecond

Also Another Approach is that Since it's probably a .csv file, you can do this at the time of reading/parsing the file itself, pandas is very intelligent and interesting!

This is done by passing in the column names as a list([year, month, day, time ]) to the parameter  parse_dates, or via  infer_datetime_format to True or by using a custom date_parser in the function pd.read_csv()>...

Hope this helps!


Training, Testing and Validation Dataset

Generally numbers (percentages) do not matter.
What matters is that your Splitting (Train/test/Validation) does 2 things. Represent the real world sitatution and making sure the model can generalise given that ist evaluated on the holdout sets.
So what does that mean here exactly? You have 30k Images and 400 patients. Most likely patients(scans) will differ from each other so you should split according to patients also to make sure the model can generalise on slightly different distributions of images.
And according to percentages. You Need to make sure that Things you find in Train test and Validation represent your Problem. This can mean Splitting by Patient, Splitting by some other feature, checking the Distribution of data etc. but what it does not mean is that only cause you have 12% in one set you are sure.
What does that mean. Lets say you have 1000 rows of data. You split 90% 10% so in holdout you have 100 data Points. But in the Train set out of the 900, the majority of them are same similiar. And they differ from the 100 Points in holdout. Is this a good split? obviously not cause your model is learning Nothing.


Which algorithm would you use for text categorisation using clustering?

tf-idf vectors are an easy start, but it can be hard to cluster very high dimensional data.

You might try topic modeling (LDA, LSI for example) to reduce the dimensionality of your features.

A newer approach is paragraph vectors, which learn a distributed representation of arbitrary-length text. Here is an implementation in python.

Learning a reasonable, lower dimensional representation of the text can help with the issues that arise trying to cluster high dimensional data.


Why is the "dying ReLU" problem not present in most modern deep learning architectures?

In practice, dead ReLUs connections are not a major issue. Most deep learning networks can still learn an adequate representations with only sub-selection of possible connections. This is possible because deep learning networks are highly over-parameterized.
Even with the possible drawbacks backs of the dying ReLUs problem, the computational effectiveness and efficiency of ReLUs still make them one of the best options currently available.


I disagree with the assertion that SOM are frequently used. IMHO they are difficult to use and rather obscure.

Nevertheless they have some interesting properties, although I'd rather call them a visualization than a clustering technique.

What you are ignoring is the"map" property. Neurons aren't independent but connected in a particular layout. Without this, the method is useless AFAICT. You need nodes to usually not be empty, but rather the average of many points, and thus M<


Predicting Sequence based on Tabular Features

I think you have 2 different ways to approach this problem. But the variable number of features and targets make it challenging. I am not aware of any well established ways to treat such issues but I find the problem interesting and would like to share my opinion.

The first way is to define this as a classification problem. The order of cars matters, in the sense that it is loss of a problem if you classify car in class 1 as a class 2, compared to identifying it as a class - say - 15 in a 20 car case. This is an "Ordinal Classification" problem and as far as I know, there is no established loss function for this. But someone implemented their own and shared it, but I never used it so I don't know how it works. Of course, this is not a must a you can use a regular classifier loss function.

Now you have to deal with variable number of cars. If you have a maximum possible number of cars present at each time, you can use that to define the number of features in your Neural Network. For the cases where there are less number of cars you can set the features associated with the non-existent cars to a number (for example -1), and put a constraint to the loss function such that cars with such features are ordered after the present cars only, and drop them at the end. This might sound complicated, it certainly is so to type it here; so I hope I was able to explain it.

The second possible way is to define a clustering problem. In order to deal with variable car numbers at each case you can define the total number of cars as an additional feature. In the example you give above they all will have a car number feature = 3. And then use any clustering algorithm you can think of, such as KNN with K=max number of cars. But this also has drawbacks: When 3 cars are present, cars can be clustered as 1,2,3; but they can just as easily be clustered into 4,6,9; or even 1,1,1. And I have no idea how you would handle such an issue. Problems with the classification algorithm defined above is less likely to cause such problems, but it is much more difficult to implement.

I know these options are far from optimal so I really hope someone else comes up with a better answer. Good luck.


Undestanding Bayesian network with OpenMarkov

First off, I did not know OpenMarkov. Anyway, from its website it has a particular focus on learning Bayesian networks (Bayes nets). Thus, I assume your figure represents a Bayes net, yes. Syntactically, it also qualifies since it is a directed acyclic graph.

The arrows (edges) represent influences (conditional dependencies) observed in the data.
For instance, the conditional probability distribution of carb, P(carb | hp,cyl,disp), is defined by the values for hp,cyl,disp. However, arrows do not necessarily represent causal relationships.


Accessing directory of small files as one file

hadoop fs -getmerge <hdfs-output-directory> <local-file>

This command can be used to concatenate the HDFS files into a single local file.

Ref : http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#getmerge


Which insights a data scientist could derive from text-analysis?

Well, obviously the use cases depends on the industry. Also, I am assuming you are thinking of use cases that are somehow useful. But let's think of some examples:

I once worked with a book distributor that tagged each book they sold with keywords (Fantasy, Horror, etc). You can automate the tagging process if you have a sufficiently large dataset of already labeled books. You can do the same with your phrases (Inspiring, Funny, etc) but probably you don't have labeled data. It would be nice to be able to ask an app for a phrase of certain type :).
Something easier is sentiment analysis: are most of the phrases positive? Negative? You don't need labels for this.
Style transfer: maybe you have several phrases by, let's say, Shakespeare. You can try transferring his style to Einstein's phrases. This is hard but feasible: look up Generative Adversarial Networks.


Expectation Maximization Algorithm (EM) for Gaussian Mixture Models (GMMs)

One reason why you aren't getting fitted values close to the true values could be the initial values of the parameters used.
It's likely what you have found is a local maxima. You have to try a number of initial starts and then pick the one with that gives the highest likelihood.


How does Google's 'showing results for' work?

First of all, no one knows how google search works except what google officially publishes. But I give you a simple algorithm for query correction (I have implemented this previously in production).
It is a simple prediction based on spell-check. First you detect the (possible) typo. Then you find the query which maximizes the likelihood. This search is done in the huge database of user queries plus some language models.


You should normalize data but you dont have to. If you dont normalize your data bigger range data will have higher effect on the model.

Just draw a box plot to compare the ranges of your data. If they have different ranges (like 100->200 and 1->10) you should definitely normalize your data. (Like mean normalization)


How to define a custom metric function in R for Keras?

This answers suggests a wrapper function . Does that work in your usecase ?

https://stackoverflow.com/a/51415066/655855


Non-mutually exclusive classification sum of probabilities

You are correct: a probability cannot be larger than 1.

At the final layer, the activations (also known as logits), are passed through the final a softmax function in order to fulfill this constraint. The standard neural network does not have an implicit mechanism by it can ensure that constraint is met during training etc.

By non-mutual classification, we could be talking about something like classifying cat and dog images - in which case the label for each image either cat or dog. So they are mutually exclusive. This is a very common case - almost any form of image classification falls into this category.

You do not use a sigmomid function (or any other non-linearity for that matter) after the final layer, as there are no neurons following them, making a non-linearity somewhat redundant. Using a non-linearity for the purpose of fitting a non-linear model is different to the purpose of a final softmax function. This has exactly the purpose of scaling the final logits/activations into the nice range of [0, 1] that can be interpreted as probabilities. That allows us to make simple rules on how to classify the outputs - e.g. if p = [0.51, 0.49] then that sample was a cat, whereas p =[0.49, 0.51] is a dog.

I used those values in the example to highlight a further point; namely that you cannot interpret them as pure probabilities. Those examples don't mean the model was really unsure in both cases because all four "probabilities" we close to 0.5. The model gives more weight to the option is believes is correct - the relative magnitudes of those values are not directly interpretable.


What the differences between self-supervised/semi-supervised in NLP?

Semi-supervised learning is having label for a fraction of data, but in self-supervised there is no label available. Imagine a huge question/answer dataset. No one labels that data but you can learn question answering right? Because you are able to retrieve relation between question and answer from data.
Or in modeling documents you need sentences which are similar and sentences which are dissimilar in order to learn document embedding but these detailed labels are usually not available. In this case you count sentences from same document as similar and sentences from two different documents as dissimilar and train your model (example idea: you can run a topic modeling on data and make similar/dissimilar labels more accurate). It is called self training.


What is deconvolution operation used in Fully Convolutional Neural Networks?

Upsampling layer is used to increase the resolution of the image. In segmentation, we first downsample the image to get the features and then upsample the image to generate the segments.

For deconvolution operation we pad the image with zeroes and then do a convolution operation on that, hence it is upsampled.

For eg: - If after downsampling the images becomes:

[[1, 1]
[1, 1]]

Then if we pad it with zeroes,

[[0, 0, 0, 0, 0, 0]
 [0, 0, 0, 0, 0, 0]
 [0, 0, 1, 1, 0, 0]
 [0, 0, 1, 1, 0, 0]
 [0, 0, 0, 0, 0, 0]
 [0, 0, 0, 0, 0, 0]]

Now, if we do convolution on the above image with a 3 x 3 filter, we will get an image of shape (4, 4). Thus we upsampled an image of shape (2, 2) to (4, 4).


What is preferred upsampling or zero padding?

It depends on application.
For upsampling, Keras documentation says:

Repeats the rows and columns of the data by size[0] and size[1]
respectively.

For zero padding, Keras documentation says:

This layer can add rows and columns of zeros at the top, bottom, left
and right side of an image tensor.

So, imagine You have picture of sharp edge shapes, and You want to build model that could learn to distinguish them. In that case, I would likely go with UpSampling, because You would have extra data, and Your shape structure wouldn't be ruined. While, with smooth curved edges, smoothness could be ruined. In that case, I would go with ZeroPadding, because it won't touch inner data, it will only add zeros side the borders.
This is my intuition about those two functions, don't take it for granted. The best option You could do is to test both on Your sample and see the output.


o(t) is not the result of concatenation of h(t-1) and x(t), but a simple matrix multiplication.
See wikipedia for further details:

https://en.wikipedia.org/wiki/Long_short-term_memory


Multivariate Linear Regression with exponential trend-line

I would suggest transforming "qty" into the log space. Even, you can do this using Excel. You can make a new column (for example qty_log), which is equal to the log of "qty". Then, you can fit a linear regression as follows:
$cost = a1 * log(qty) + a_2 * volume + a_0$
You will get much better results in this case.
This equation simply means that try to fit a model using volume and log of qty for predicting cost. You did so before but you used qty instead of log(qty)


Predicting Bitcoin price with LSTM (and other RNNs)

I'm not particularly expert in this but I'm quite sure that the variations in the price of anything depend mostly on external factors: news of the day, economic indicators, stockmarket movements, etc.
As a rule of thumb, if a human with a lot of time can't do it, usually a ML model can't do it either. In this case if an expert in finance is given the history of the price but no access to any other information I doubt they can predict anything other than a conservative guess that the price will stay the same.
I don't see how the model can do anything better than this without some good indicators as features, and probably it would need a large number of complex indicators. For example I would assume that news like these affect the Bitcoin price, but gathering this kind of information and integrating it into a model certainly requires a lot of work.


How to prevent a neural network from choosing the 'easiest' solution

This is fairly common with scenarios like predicting fraud when overall cases of fraud are uncommon. The model will frequently just say "no fraud!" and be right a large percent of the time. You need to assign a penalty, cost, or weight so that output 1 isn't so easy.

This article describes that further in a lot of detail:

https://blog.fineighbor.com/tensorflow-dealing-with-imbalanced-data-eb0108b10701


Prediction of a dependent variable having comma separated values

Your problem is called multi-label classification. If you search for this term you will find a lot of literature. In short, these are the most common approaches:

Treat each label independently and build a binary classifier per label (baseline approach, Eric Lecoutre's answer)
Predict the power set, means create a target variable that represents all possible combinations and predict it using a multi-class classifier (Upper_Case's answer)
Use a classifier that can deal with multi-labels inherently, e.g. a neural net with multiple output nodes and cross-entropy as loss function, or respective multi-label variants of decision trees or SVM. For example, scikit-learn supports multi-label implementation for decision trees.
Classifier chains: Train a binary classifier per label, but use the predictions of one classifier as input to another classifier

Depending on the nature of the data, one or the other approach will work best. Start with a binary classifier per label to get a baseline. If the labels are correlated (some labels frequently occur together), there is potential for improvement using one of the other methods. Using one of the existing implementations of multi-label classifiers is probably the easiest next step. The power set approach only makes sense if the number of possible combinations is reasonable. I am not sure if classifier chains are worth the effort, if you are not doing this for a Kaggle competition :)

As always, setup a proper cross-validation strategy, define the evaluation metric (e.g. micro/macro f-measure or cross-entropy loss) so that you are able to identify the best model.


You can build more complex models to try to capture the remaining variance. Here are several options:

Add interaction terms to model how two or more
independent variables together impact the target variable
Add polynomial terms to model the nonlinear relationship between an independent variable and the target variable
Add spines to approximate piecewise linear models
Fit isotonic regression to remove any assumption of the target function form
Fit non-parametric models, such as MARS


Which different visualizations to make for time series analysis?

I am also new to time series forecasting. I used simple lineplot to visualize the time series data. Some of the models used for time series are :

ARIMA
SARIMA
prophet
LSTM (deep neural networks)
LBATS
You can refer to this link to know what things you should consider while building a time series model


How to train a keras model on both original and augmented data from ImageDataGenerator?

ImageDataGenerator do augmentation on the fly base on the setting you give , so it did not separate “original” or “augmented “ data , just the possibility of data been augmented.
so your question actually is “ how to tune model when overfitting happens ”? I think you can start from adding dropout or reduce number of parameters. Hope this helps, cheers.


cosine_similarity returns matrix instead of single value

Based on the documentation cosine_similarity(X, Y=None, dense_output=True) returns an array with shape (n_samples_X, n_samples_Y). Your mistake is that you are passing [vec1, vec2] as the first input to the method. Also your vectors should be numpy arrays:

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
vec1 = np.array([[1,1,0,1,1]])
vec2 = np.array([[0,1,0,1,1]])
#print(cosine_similarity([vec1, vec2]))
print(cosine_similarity(vec1, vec2))

  X : ndarray or sparse array, shape: (n_samples_X, n_features)
      Input data.

So you have to specify the dimension.

np.array([1, 2]).shape

has funny shape:

  (2, )


how to update column in data frame based on condition

You access string functions with .str. This should work:

df.loc[df.Lvalue.str.istitle(), 'IsLCap'] = 1

Update
To check the length of the strings in the column you can use the string method .str.len:

df.loc[df.Lvalue.str.len() < 3, 'less_than_three'] = 1

Pandas Series string methods


A combination of multiple linear functions can help one model complex decision regions. Two or more linear functions can be combined to form a piece-wise approximation of a non linear function. Computation of non-linear activation function on a very deep network can be expensive (because of the exponential term involved). ReLu activation function which is a linear activation is used almost by every CNN model and is proved to give better results.
$$ f(x) = max(0,x-1) + max(0,x+1)$$
Consider the function above which is combination of two reLu's.  This approximates the sigmoid function(try plotting the same) and is computationally cheap.


How do we decide on the classification algorithm to use with huge training size?

I don't think there is a clear-cut criterion to decide on what method to use. From what I see, you have a good amount of data and the problem is "complex" (language). This are reasons to go for "deep" learning such as neural nets or boosting. The reason is that both can handle "non-linearity" very well.

Another thing that comes to my mind is that when you use for instance logit, you man have the problem that you have many features (words) and that only a certain share of the features might be relevant. So in this case you would try to get rid of features which are not helpful in making good predictions. With logit, a lasso approach (l1 penalty) would be a natural thing to do, to "shrink" features.


How does deep learning helps in detecting multiple objects in single image?

Although many solutions in production systems still use a sliding window as described below in this answer, the field of computer vision is moving quickly. Recent advances in this field include R-CNN and YOLO.

Detecting object matches in an image, when you already have an object classifier trained, is usually a matter of brute-force scanning through image patches.

Start with the largest expected patch size. E.g. if your image is 1024 x 768, but always a distance shot of a road maybe you do not expect any car to take up more than 80 x 80 pixels in the image. So you take an 80x80 block of pixels from one corner of the image, and ask your classifier what chance there is a car in that corner. Then take the next patch - perhaps move by 20 pixels.

Repeat for all possible positions, and decide which patches are most likely to contain cars.

Next, take a block size down (maybe 60 x 60, moving 15 pixels at a times) and repeat the same exercise again. Repeat this until you have hit the expected smallest block size for your goal.

Eventually you will have a list of areas within the image, with the probability that each contains a car.

Overlapped blocks both with high probability are most likely the same car, so the logic needs to have thresholds for merging blocks - usually taking the overlapped area with the highest probability score - and declaring there is only one car in that area.

As usual with ML approaches, you will need to experiment with correct meta-params - in this case block sizes, step sizes and rules for merging/splitting areas - in order to get the most accurate results.


I'm afraid I have zero experience with ElasticSearch, so I can't help with that but here are a few thoughts about the process you propose.
Overall I think the approach makes sense, especially in terms of efficiency but also check the following points:

This is not strictly speaking a clustering task, in the sense that the system doesn't find itself how to separate the clusters since this is based on a predefined threshold.
The threshold is crucial and you need a method to determine it. You might also want to plan for updating the threshold as the collection grows. The process is likely to lead to some very big clusters and many very small ones (unique document in a cluster).
If you assign a document to the best match cluster, it means that a document belongs to a single cluster. Depending on the application this could be problematic, with some documents expected in a cluster but found in another.


It turns out the Adaboost or VotingClassifier doesn't work for multilabel classification data. We get a NotImplementedError. I believe they are working towards it and might be out in next version or so.


How do I right feature selection for DBSCAN?

Automatic weighting will likely not be enough.

For examples standard scaler will assign twice as much weight to the one-hot encoded parts than to the other attributes. Plus, it is based on the assumption that all variables should have the same (assuming they are all normal distributed) weight. But should they all have the same weight? Does it even make sense to use distances of one-hot encoded data?

You are not ready to use DBSCAN (or any other clustering) if you don't have a reliable distance yet that discriminates similar from dissimilar instances.


Pass data to CNN with multiple outputs in keras

The first dimension is always the sample index. In the model summary this is written as None, so your last output (None, 15, 145) is 3D.

The error message shows that your target data has dimension (:, 15), rather than the required (:, 15, 145) so you will have to reformat it.


You ANN is expecting an input of size 11 as you specified with the parmaters input_dim

classifier.add(Dense(input_dim=11, units=6, kernel_initializer='uniform', activation='relu'))

But your training set is sending array of size 3.
It's probably due to tour dummy variables :

#Dummy Variable
onehotencoder = OneHotEncoder(categorical_features=[1])
X = onehotencoder.fit_transform(X).toarray()

you can set your ANN to get correct input:

classifier.add(Dense(input_dim=3, units=6, kernel_initializer='uniform', activation='relu'))


Is it Possible to Use Machine Learning for Ranking Alternatives?

Yes, you can rank those data points.

First, you need to define a quantitative measure of "potential danger" called a target variable. That could be a single variable (e.g., number of accidents) or weighted collection of variables, frequently called an index.

Then a regression model can be trained to predict that continuously-measured target variable and the data points can be rank-ordered.


Deep learning facial recognition research project

First, tensorflow would be a great resource as it is gaining a userbase with exponential growth. Tensorflow has gpu support for single node calculations, and has recently added a distributed version for larger calculations.  There are some tutorials on convolution ANN, which will probably be the direction you will go with a computer vision problem. Also, some books are coming soon and some are in alpha.  But enough of plugging a particular library!

A perspective based facial recognition problem would be interesting if data were available e.g. with two perspectives on faces, do recognition on a third perspective.
A relative based facial recognition problem could be interesting, again, if data were available e.g. given two parents, try to find the child.  One could probably scrape facebook to get a reasonable data set for this one.
Blending and aging faces and controlling for sex (hair length, facial hair, etc) for problems where a kid has been abducted and you have parents pictures and sibling pictures and the child has grown up.
A simple aging recognition problem could be interesting... given baby pictures, can you recognize the adult or vice versa.

Just some thoughts, the possibilities are endless. I suggest starting with a very simple stub of a problem and then slowly adding sophistication.

Hope this helps!


My answer would be second option
I think the use of PCA is to represent original high dimensional information/data in lower dimension by calculationg the direction/axes along which there is maximum variablity in data.
In first case, where you filter for 0-labeled observations and then do PCA so PCA would measure variablity based on a smaller version of original data, and would make different axes than the second case, where the PCA would measure variablity accross entire data and hence the axes made in second case could be different. So in the first option the dataset you get after PCA is not a correct representation of high dimensional 0-labeled observations.


Network Visualization - R code - Which Package is needed?

The visNetwork R package is the best I've worked with.  It renders with the vis.js Javascript package right in your RStudio window.  Nodes and edge visualization is fully customizable and may be data-driven.  They implement click-functionality on your graphs, so you can rearrange them or highlight a node and its neighbors.  I've found people really enjoy this in presentations, and the highlight functionality is great for making static PowerPoint presentations or figures.


Which is the fastest image pretrained model?

The answer will depend on some things such as your hardware and the image you process. Additional, we should distinguish if you are talking about a single run through the network in training mode or in inference mode. In the former, additional parameters are pre-computed and cached as well as several layers, such as dropout, being used, which are simply left out during inference. I will assume you want to simply produce a single prediction for a single image, so we are talking about inference time.

Factors

The basic correlation will be:

more parameters (i.e. learnable weights, bigger network) - slower than a model with less parameters
more recurrent units - slower than a convolutional network, which is slower than a full-connected network1
complicated activation functions - slower than simple ones, such as ReLU
deeper networks - slower than shallow networks (with same number of parameters) as less run in parallel on a GPU

Having listed a few factors in the final inference time required (time taken to produce one forward run through the network), I would guess that MobileNetV2 is probably among the fastest pre-trained model (available in Keras). We can see from the following table that this network has a small memory footprint of only 14 megabytes with ~3.5 million parameters. Compare that to your VGG test, with its ~138 million... 40 times more! In addition, the main workhorse layer of MobileNetV2 is a conv layer - they are essentially clever and smaller versions of residual networks.

Extra considerations

The reason I included the whole table above was to highlight that with small memory footprints and fast inference times, comes a cost: low accuracies!

If you compute the ratios of top-5 accuracy versus number of parameters (and generally versus memory), you might find a nice balance between inference time and performance.

1 Have a look at this comparison of CNNs with Recurrent modules


Can finite state machines be encoded as input/output for a neural network?

The better question might be - Why you do want to mix deep neural networks (DNN) and a finite state machine (FSM)?

There is research showing that a DNN can simulate any FSM. Since there are more frameworks for DNN and DNN can perform more tasks than FSM, it appears to be more useful just to forgo FSM altogether.

More specifically, "Neural network for synthesizing deterministic finite automata" shows how a relatively simple neural network (NN) can quickly and automatically learn the correct deterministic finite automaton (DFA).

There is a strong trend towards end-to-end DNN.


Multiprocessing or Parallel Computing Python Code

If you're using scikit-learn, there is a parameter for most learners called n_jobs. This parameter can be set to -1 to utilize all processors. For more details, see scikit-learn n_jobs parameter on CPU usage & memory


Baby cry detection model using binary classification through logistic regression

As your context, You convert your audio clips to spectrogram images. The spectrogram is a visual representation of the spectrum of frequencies of sound.
 Now you need to Extract Features from your dataset. There are many feature extraction technique for audio data (See this blog for knowledge).
  The state of the art features extraction is now using:  MFCC — Mel-Frequency Cepstral Coefficients.
 By using librosa library we can easily implement the MFCC.
 Here are some resources for MFCC :

Music Feature Extraction in Python - From towardsdatascience
Audio Data Analysis Using librosa - Kaggle Notebook (Check out Mel-Frequency Cepstral Coefficients(MFCCs) portion)
Stack Overflow Question : MP3 to MFCC

After you extract features (Converted into MFCC), Then you need fit them into the Logistic Regression Model as you wanted. I share some resources which experiment with MFCC Features and Logistic Regression:

Classification Logistic Regression/SVM on mfccs
Audio-Genre-Classification GitHub repo by dhruvesh13
Pitch detection - Kaggle Notebook

And for learn logistic regression, you may follow Machine Learning By Stanford, Coursera or follow this from youtube.


Artificially expanding the datasets through rotation of images in MNIST

The problem is that numbers are not invariant to rotations.

For example, see what happens when you rotate a 4 in steps of 90 degrees:

So unless your task includes recognizing numbers which are written sideways or upside-down this does not provide a proper data augmentation.


Deep Q Network gives same Q values and doesn't improve

I can see you are using "plain" deepQ, consider using double deep Q or duelling, but go first on your current implementation.

First of all, most of the times the problem is a bug, can you provide the history of the scores?. It is important to understand if the agent is learning or the best solution it can find is killing itself to minimize loses.

As per your hyperparameters, I will suggest two increase gamma first and decay epsilon slower.

Another tweak will be your alpha, reduce it a bit, seems too high.

Anyway, I need more details to help you, for example, Are you reading the screen ?, if so, did you reduce the image to something manageable?
Can you provide the full code?


What exactly is the input of decoder in autoencoder setup

Your are mixing two different beasts. Despite both having encoder and decoder parts, the way in which a normal feedforward image transformation network (i.e. the autoencoder) and an autoregressive model (i.e. the seq2seq) are actually used is very different:

In the type of image transformation done in a vanilla autoencoder, you input an image and get another image at the output. That's it, both at training and at inference time.
In an autoregressive model like seq2seq, you feed the network with the input and the initial tokens of the expected output sequence (i.e. the prefix), and you get the next token as output. During training, those initial tokens are normally from the gold data (aka teacher forcing), that is, you feed as prefix real tokens from the training data. In inference, those tokens are generated by the model itself: you generate the first token, then you generate the second token given the first one you previously generated, then you generate the third token given the previous 2 generated tokens, and so on.


On Creating an Interoperable Network Matrix

There is no single universally accept standard, however, many packages use edgelist for sparse (un)directed (un)weighted graphs. Regarding storing and sharing, I usually see compressed text files (in ascii format) instead of binary ones, due to binary compatibility problems.

Maybe you should take a look at this draft: Unravelling Graph-Exchange File Formats.


Confusion matrix doesn't display properly

Please check how many classes y_test has.

if Y is array-

np.unique(y_test)

If y is DataFrame column,

y_test.unique()

Looks like y_test has only 2 classes.

As far as I know HR-Employee-Attrition.csv dataset has only 2 classes.


Unstable results in test mode with fractional max pooling in PyTorch

The answer (maybe one of) is FractionalMaxPool2d layer I use. If I use say ResNet-34, train it a bit (say 1/6 of epoch), everything is ok. If I add one FractionalMaxPool2d layer:

...
self.layer1 = self._make_layer(block, 2, 64, layers[0])
self.maxpool2 = nn.FractionalMaxPool2d(kernel_size=3, output_size=38)
self.layer2 = self._make_layer(block, 3, 128, layers[1], stride=2,
...

(and run it in the forward method of course)

and repeat, the difference appears.


ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

First of all I just want to say that I am not a data engineer and there is definitely someone out there that can answer this better than me.
I do think that there is a lot of theory behind data engineering. It is also very interesting. I too thought that it was boring and I was more interested in just data science/ machine learning. I am not sure if I can say exactly what principles data engineering is based on but it is about how to best store data, access data and creating underlying systems for more efficient computing. The first paper I read that really got me interested in this stuff was the original paper for Spark.
I also just did a quick google for data engineering PhD and came across this. There is a lot of interesting new research going on with how to store data using "nano-structures".  There's also an area of research in quantum databases, which seems like a really interesting database abstraction.
I would be interested in hearing a more informed and complete answer from someone else who is in this field! In fact it might be useful to post this question on another stack exchange site.


In response to the comments suggesting subgradient descent methods: the problem in the question does not contain any sum or similar terms, therefore the variables are independent of each other and there is a closed-form solution:
$$ x^* =
\begin{cases}
    a - \frac{\lambda}{2},& \text{if } a > \frac{\lambda}{2}\\
    a + \frac{\lambda}{2},& \text{if } -a > \frac{\lambda}{2}\\
    0,              & \text{otherwise}
\end{cases}
$$


Cross validation Vs. Train Validate Test

If k-fold cross-validation is used to optimize the model parameters, the training set is split into k parts. Training happens k times, each time leaving out a different part of the training set. Typically, the error of these k-models is averaged. This is done for each of the model parameters to be tested, and the model with the lowest error is chosen. The test set has not been used so far.

Only at the very end the test set is used to test the performance of the (optimized) model.

# example: k-fold cross validation for hyperparameter optimization (k=3)

original data split into training and test set:

|---------------- train ---------------------|         |--- test ---|

cross-validation: test set is not used, error is calculated from
validation set (k-times) and averaged:

|---- train ------------------|- validation -|         |--- test ---|
|---- train ---|- validation -|---- train ---|         |--- test ---|
|- validation -|----------- train -----------|         |--- test ---|

final measure of model performance: model is trained on all training data
and the error is calculated from test set:

|---------------- train ---------------------|--- test ---|

In some cases, k-fold cross-validation is used on the entire data set if no parameter optimization is needed (this is rare, but it happens). In this case there would not be a validation set and the k parts are used as a test set one by one. The error of each of these k tests is typically averaged.

# example: k-fold cross validation

|----- test -----|------------ train --------------|
|----- train ----|----- test -----|----- train ----|
|------------ train --------------|----- test -----|


Which algorithm is perfect to determine best fit analyst based on multiple factors?

One approach is a set of models which take the ticket as an input and churn out scores for how critical the ticket is, what kind of skill set is required, what kind of satisfaction score will be needed, and the communication skills needed.

Based on these predictions, you could then write a simple rule-based algorithm to determine the optimal analyst to send the ticket to. These rules would be based on the analyst characteristics you mention in the question.

Depending on the form of the ticket input, the models above may require an element of natural language processing (NLP). You may also want to research recommendation systems - it's very likely that something similar has been done before.


What is the meaning of hand crafted features in computer vision problems?

"Hand Crafted" features refer to properties derived using various algorithms using the information present in the image itself. For example, two simple features that can be extracted from images are edges and corners. A basic edge detector algorithm works by finding areas where the image intensity "suddenly" changes. To understand that we need to remember that a typical image is nothing but a 2D matrix (or multiple matrices or tensor or n-dimensional array, when you have multiple channels like Red, Green, Blue, etc). In the case of an 8-bit gray-scale image (or a "black and white" image, although this latter definition is not quite accurate) the image is typically a 2D matrix with values ranging from 0 to 255, with 0 being completely black and 255 being completely white. Now imagine an image of a blackboard set against a totally white wall. As we move left-to-right in the image the values in one of the rows of the matrix might look like 255-255-255... since we will be "moving" along the wall. However, when we are about to hit the blackboard in the image it might look like 255-255-0-0-0...As you might have guessed, the blackboard "begins" in the image where the zeros start. In other words, the "intensity" of the image along the "x" axis has dropped rather suddenly (or a very large negative gradient along x), which means a typical edge detector will consider it to be a good candidate for an edge.

The algorithm that we just saw is only the very basic of algorithms, and others like Harris corners and Hogg detectors use slightly more "sophisticated" algorithms. Actually, even the Canny edge detector does a lot more than I what I just described, but that is besides the point. The point is that once you understand that an image is nothing more than a data matrix, or an n-dimensional array, the other algorithms are not that difficult to understand either.

As regards your last question:

  Is this related to learned features?

The "handcrafted features" were commonly used with "traditional" machine learning approaches for object recognition and computer vision like Support Vector Machines, for instance. However, "newer" approaches like convolutional neural networks typically do not have to be supplied with such hand-crafted features, as they are able to "learn" the features from the image data. Or to paraphrase Geoff Hinton, such feature extraction techniques were "what was common in image recognition before the field became silly".


Finding predictions using biglm without finding errors

biglm calls model.frame which "all the variables in the formula are included in the data frame" see documentation for model.frame. This is the issue that comes up when predict is called on the biglm class. It looks for those values in the predict function. To get around this you can just create a variable and encode it with 0. See below...

data(trees)
ff<-log(Volume)~log(Girth)+log(Height)

chunk1<-trees[1:10,]
chunk2<-trees[11:20,]
chunk3<-trees[21:31,]

a <- biglm(ff,chunk1)
summary(a)

#produces same error
chunk2 <- select(chunk2, -Volume)
predict(a, chunk2)

#Fixed
chunk2$Volume <- 0
predict(a, chunk2)


Currency Normalization for Salary Prediction

Within each currency (say SGD or INR), you can divide the salary by quartiles, for e.g the top 25 percentiles of salaries within each currency can be assigned a value of 4, the next 25th percentile a 3 and so and so forth. This way you are able to normalize salaries across currencies.


The neurons themselves only hold their activation values (or input values when we consider the input layer of neurons). The weights you mention actually indicate the strength of the connections between neurons in subsequent layers (and possibly the bias per layer). Using your terminology the weights would belong to the neurons in both the current and next layer.

For every neuron the strength of the connection to the other neurons is saved. In other words, all weights (and possibly biases) are saved.

For more information about the basics of neural networks, and more specifically multi-layer perceptrons, I can remmond this website.


Should I normalize data to be similar to Normal distribution?

Normalising input data to a neural network is known to improve convergence properties, i.e. the model should converge quicker because the normalised data will not be so likely to produce huge or tiny gradients. Huge gradients make training erratic and less likely to converge (without applying other tricks), while tiny loss gradients lead to neuron death whereby certain neurons may get stuck, unable to get to an optimal weight value.

As you pointed out, normalising can be trivially simply to perform, so it might be worthwhile for both the final results and as a learning experience to try it out and compare your results.

You might also consider using other techniques such as batch normalisation, which inherently give you similar properties to normalising the entire dataset at the beginning.


Linear Discriminant - Least Squares Classification Bishop 4.1.3

As Bishop points out throughout that section, least squares is ill-equipped for this problem, so maybe we shouldn't spend too much time understanding it.  On the other hand, clearing up misconceptions here may help elsewhere.

We would like for (i.e., least squares strives for) $\operatorname{val}(\mathcal{C}_2)$ to be close to 1 and the others close to 0.  But the values could very well be negative, positive, greater than 1, for any of the classes (the correct ones or not)!  Our final classification though is dealt with as Section 4.1.2 explains (just after equation 4.9): the class with largest $y$-value wins.
Yes, here $\operatorname{Tr}$ means the trace.  In "least squares," we're minimizing the mean squared error.  It happens that the sum of squared errors can be written nicely as the trace of these matrices, but the incidental entries of the matrix off the diagonal don't concern us here.


Unable to interpret this 100% stacked area chart

The statement is that the point (22,80) is on the barrier between the regions marked "Male advantage" and "Female advantage".


Logic Check: Building a SKLearn Pipeline

You are on the right path. It appears you might have analysis paralysis. You should start building, then see what works and what does not work.
Here is code to get you started:
from sklearn.ensemble             import RandomForestRegressor
from sklearn.feature_selection    import VarianceThreshold
from sklearn.model_selection      import GridSearchCV
from sklearn.pipeline             import Pipeline

regressor = Pipeline([('vt', VarianceThreshold(threshold=0.0)),
                      ('rf', RandomForestRegressor())])

gsvc = GridSearchCV(regressor, {'vt__threshold':    [0, .1, .5, .7, 1],
                                'rf__n_estimators': [10, 100, 1_000],
                                'rf__max_depth':    [2, 10, 100]})


R: comparison (6) is possible only for atomic and list types

The code works and leads to no error. The result is a definition of one variable rand one function r_hat.

You get the same with this snippet (I removed the loop as not relevant, what you do is creating the same function repeatedly in a loop)

> r<- matrix(c(2,4, 1,5,5,3, 4,5,1,2,3,4), nrow=3)
>
> r_hat<- function(u,x) {
+       if (r_hat >r[u,x]) {
+         print(r[u,x]-r_hat)
+       }
+ }

The problem is in the function, after calling it you get the mentioned error.

> r_hat(0,0)
Error in r_hat > r[u, x] :
  comparison (6) is possible only for atomic and list types
>

You reference r_hat within the function, are you trying to define a recursive function? Than the recursive call should have the parameters and the function some return value...

My guess is you tries to compare the function definition with an element of a matrix, which leads to this error.

If this is not enough for you and you explains what the code should deliver, I'll try to complete my answer.


SciKit-Learn: predict values sometimes different from top predict_proba entries?

This seems to be a bug in the library.

https://github.com/scikit-learn/scikit-learn/issues/17035

The issue is predict_proba does not convert from its internal representation of categories to the dataset's representation before returning the data. Client-side we can convert using some internal APIs:

> [text_clf._final_estimator.classes_[max(enumerate(prob), key=lambda p: p[1])[0]] for prob in text_clf.predict_proba(test.data)]
[61, 16, 11, 16, 11, 89, 26, 33, 16, 57, 16, 118, 11, 26, ...]
> [*text_clf.predict(test.data)]
[61, 16, 11, 16, 11, 89, 26, 33, 16, 57, 16, 118, 11, 26, ...]

But the real fix needs to happen in the library.


"Sample" as a noun usually refers to a single data point.  "Sample" as a verb is the act of extracting a subset of data points from some larger body (reality or a larger dataset).  The only way to be less ambiguous is use more specific words than "data" or "sample".

Example:

Say you collect 1MM data points from four different sensors in the field giving you four sets of 250k data points.  Say this data is too big for some demo of a model you're testing or an analysis you're running, so you pick 100k data points evenly split across the four sensors (giving four sets of 25k data points).

In this example, you're sampling twice.  First, to gather your 1MM data points sampling from reality.  Second, you sample again to decrease the size of your data set to something more manageable.  'Data' or 'sample' could refer to reality, the 1MM dataset, the 100k dataset, or any of the sensor-specific subsets.  To make it less ambiguous, establish a unique name as soon as possible for each possible definition you'll be working with.  ('reality' for the set of all possibly observed samples.  'the complete dataset', something derived from the source of the dataset, or even X for the full 1MM dataset.  'our trial dataset', or even Y for the small 100k dataset.

What you actually do comes down to context and what's appropriate for your intended audience, but the general answer is to use more specific words.


Deep fully connected NN with vanishing gradients

Vanishing gradients can occur because of adding so many layers to network. You can think neural networks as composite functions. During learning process gradients of loss function with respect to weights calculated according to chain rule, and because of large number of layers gradients can be so small by multiplications. It causes insufficient updates of weights towards optimum points of loss function. If your loss function barely decreasing, or there is no change at all then cause of this might be vanishing gradients.
Activation functions like tanh, sigmoid also can cause this problem, because their gradients become zero in edge points. Skip connections can ease vanishing gradient problem, because when connections are skipped gradient calculation follow a shorter path, and less multiplications can ease vanishing gradient problem. Resnets use skip connections to ease vanishing gradient problem: https://en.wikipedia.org/wiki/Residual_neural_network.
If you observe no decrease in loss function, then you can decrease number of layers gradually. It's a hyperparameter, and you can experiment with different numbers of layers to find optimum number which gives best results.


Understanding Classifier performance on text data

As the data is imbalanced and skewed towards few classes, that's why RF and Logistic results are biased causing high FP values, therefore high precision and low recall.
SVC on the other hand might have tried to create hyperplanes to get most out of the other side of curve, thereby producing different results.

To improve the results, try to use non linear kernels and also try to balance the input data (by scaling etc) before feeding to classifiers.


Loss function that penalizes under-predictions more than over-predictions?

Pick an asymmetric loss function. One option is quantile regression (linear but with different slopes for positive and negative errors).


should we include or exclude a variable in a logistic regression based on the description below?

You may want to add an indicator variable, 0 if the event has happened and 1 if it has not.  In a linear model, the coefficient on the original can fit to the "real" slope, while the coefficient on the indicator "fixes" for the missing values (and whatever imputation you use).

https://stats.stackexchange.com/a/105258/232706
https://datascience.stackexchange.com/a/64709/55122
https://datascience.stackexchange.com/a/69841/55122


How to detect if one tweet is agreeing with another

Not sure there is anything for that, you could check

is it a verbatim retweet
does it have the same sentiment
is the edit distance low

Or you can train your own model, where you label (agree) by hand and then build features.


Python sklearn - average classification reports

It maybe a little bit complicated, since I convert the reports to pandas.DataFrame for calculation. But I think it's worth it, because it works well with two or more report as well. Try below:

import pandas as pd
import numpy as np
from functools import reduce
def report_average(*args):
    report_list = list()
    for report in args:
        splited = [' '.join(x.split()) for x in report.split('\n\n')]
        header = [x for x in splited[0].split(' ')]
        data = np.array(splited[1].split(' ')).reshape(-1, len(header) + 1)
        data = np.delete(data, 0, 1).astype(float)
        avg_total = np.array([x for x in splited[2].split(' ')][3:]).astype(float).reshape(-1, len(header))
        df = pd.DataFrame(np.concatenate((data, avg_total)), columns=header)
        report_list.append(df)
    res = reduce(lambda x, y: x.add(y, fill_value=0), report_list) / len(report_list)
    return res.rename(index={res.index[-1]: 'avg / total'})

output:

report_average  = report_average(report_1, report_2)
print(report_average)
             precision  recall  f1-score  support
0                 0.75    0.75      0.75      2.0
1                 0.75    0.75      0.75      2.0
avg / total       0.75    0.75      0.75      4.0

report_3 = report_2
report_average  = report_average(report_1, report_2,report_3)
print(report_average)
             precision    recall  f1-score  support
0             0.666667  0.666667  0.666667      2.0
1             0.666667  0.666667  0.666667      2.0
avg / total   0.666667  0.666667  0.666667      4.0


Reversed Naive Bayes - likelihood and parameter estimation

To elaborate on my comment, the first image depicts conditional dependence of $X_i | Y$ for $i \in 1,...,m$.
Therefore, your likelihood for this model would be

$$
\begin{aligned}
P(Y = y) \times P(X_1 | Y = y) \times ... \times P(X_m | Y = y)
\end{aligned}
$$

The second image (bottom) depicts $Y | X_i$, therefore the likelihood would be

$$
P(X_1) \times P(X_2) \times ... \times P(Y | X_1, X_2, ..., X_m) \\
\ = P(X_1) \times P(X_2) \times ... \times P(Y | X_1) \times ... \times P(Y | X_m)
$$

Assuming that both scenarios do not have conditional independence, the two are not equivalent!


How do I use this depth first search code to obtain a topological sort?

It appears to me that you are doing too much with the code. For DFS, visit every node following adjacent nodes. As you do that, track the topological ordering.

Here is a straightforward implementation of recursive topological sorting a DAG in Python:

from collections import defaultdict

class Graph:

    def __init__(self):
        self.graph = defaultdict(list) # Node: [adjacency nodes]
        self.nodes = set()

    def add_edge(self, u, v):
        "Directed edge from vertex u to vertex v"
        self.graph[u].append(v)
        self.nodes.add(u)
        self.nodes.add(v)

    def mark_as_visited(self, v, visited, topological_ordering):
        visited[v] = True

        # Recur for all the vertices adjacent to this vertex
        for current_vertex in self.graph[v]:
            if visited[current_vertex] == False:
                self.mark_as_visited(current_vertex, visited, topological_ordering)

        topological_ordering.insert(0, v)

    def topological_sort(self):

        visited = [False]*len(self.nodes) # Mark all the vertices as not visited
        topological_ordering = []

        # Sort starting from all vertices one by one
        for i in range(len(self.nodes)):
            if visited[i] == False:
                self.mark_as_visited(i, visited, topological_ordering)

        return topological_ordering

# Create a sample DAG
g = Graph()
g.add_edge(5, 2)
g.add_edge(5, 0)
g.add_edge(4, 0)
g.add_edge(4, 1)
g.add_edge(2, 3)
g.add_edge(3, 1)

# Double check that the algorithm is correct for that DAG
assert g.topological_sort() == [5, 4, 2, 3, 1, 0]


Reinforcement learning with sparse acting agent

One option is to create a reward function to incentivize the behavior you want:

Small positive reward for doing nothing.
Large positive reward for doing the rare, correct action.
Negative reward for taking action when no-action is correct.
Negative reward for taking incorrect action.


Treating missing data in categorical features

You could break the column 2 from your example into number of columns :
Image,Video....
So the new features will be like:
Column1  Image  Video
Google     0      0
Google     0      0
Facebook   1      0
Facebook   0      1


How to implement PCA color augmentation as discussed in AlexNet

You should not apply *255.

delta was supposed to be added to renorm_image, because you calculated this delta using cov, which was based on renorm_image.

Then how would you restore renorm_image to your original image? *std + mean or *255?

Obviously you should apply *std + mean.

Therefore,

delta = (delta*255.).astype('int8')
pca_color_image = np.maximum(np.minimum(original_image + delta, 255), 0).astype('uint8')

should be changed to:

mean = np.mean(renorm_image, axis=0)
std = np.std(renorm_image, axis=0)
pca_augmentation_version_renorm_image = renorm_image + delta
pca_color_image = pca_augmentation_version_renorm_image * std + mean
pca_color_image = np.maximum(np.minimum(pca_color_image, 255), 0).astype('uint8')


The score method used in keras does not calculate accuracy like the sklearn's accuracy_score method. Check the internal implementation of evaluate method to understand more. If you want to calculate accuracy, I suggest you to use sklearn's accuracy_score by getting the predictions or manually calculate if it is easier for you.


When to consider a target Variable unbalanced?

There's not a strict threshold about what ratio is considered as unbalanced. But in general, 30 percent is not usually a sign of unbalanced classification.

You can although try different methods for checking if your classification method is accurate and predicts correctly or not, like resampling, over-sampling, under-sampling

increasing the number of records (samples) in the class with less frequency.
randomly removing the samples from the class with higher frequency.

...

from sklearn.utils import resample


Your features could be the other twitter accounts that the users follow with a value of 1 if your user follows that account or a value of 0 if the user does not. Here is a very simplified version, using something like your starting point.

# If you start like this:
user_id  feature_1  feature_2  feature_3  label
user_1   account_1  account_5  NA         1
user_2   account_2  account_3  account_4  0
user_3   account_1  account_2  account_5  1
user_4   account_3  NA         NA         0

# Transform your data to look like this:
user_id  account_1  account_2  account_3  account_ 4  account_5  label
user_1   1          0          0          0           1          1
user_2   0          1          1          1           0          0
user_3   1          1          0          0           1          1
user_4   0          0          1          0           0          0

This is a trivial example with only five accounts to follow. You may be dealing with millions of accounts to follow, which would be at best unwieldy. Since you already have a sense that some followed accounts are correlated with your label, you could simplify the features to only check those that are strongly correlated.

In my simple example above, we may only want to keep the features account_1, account_3, and account_5 since they correlate strongly to the label.


How does the bounding box regressor work in Fast R-CNN?

A very clear and in-depth explanation is provided by the slow R-CNN paper by Author(Girshick et. al) on page 12: C. Bounding-box regression and I simply paste here for quick reading:

  Moreover, the author took inspiration from an earlier paper and talked about the difference in the two techniques is below:

  After which in Fast-RCNN paper which you referenced to, the author changed the loss function for BB regression task from regularized least squares(ridge regression) to smooth L1 which is less sensitive to outliers!. Also, you embed this smooth L1 loss in the multi-task loss function so that we can jointly train for classification and bounding-box regression that wasn't done before in R-CNN or SPP-net!

  However, the same author has changed the loss function again in the upcoming paper faster-RCNN
  Later, in FCN
  Many a time, in order to learn about a topic, you need to do backtracking through research papers! :) Hope it helps!


trying to decrease overfitting with regularisation in CNN

Steps for overcoming overfitting are the followings:

 1. Add more data
 2. Use data augmentation
 3. Use architectures that generalize well
 4. Add regularization
 5. Reduce architecture complexity.

I would try to add Dropout, lately it is being the most used regularization technique in complex networks architectures. Start adding it in the later layers, since it is best no to lose initial info. Something like 0.6 or like that, and if it is not enough add dropout in the previous layer, and so on.


Décision tree, How to see under/over fitting with just looking at the leafs?

You can look at number of leaves. If number of leaves are very high compare to number of classes available for classification then tree might be overfitting or vice versa.

You can also check number/percentage of training examples correctly classify by each and every leaf. You can put some threshold on top of that so that every leaf has to classify threshold of examples correctly.

But again, cross validation and holdout method will give you better idea for overfitting or underfitting.


How do I deal with the fact that I have images which are not consistent with the class they belong in an image classification problem with CNN?

There are different ways of doing this, but the final idea in regard to this is that you need to clean your dataset.

You could go through it manually and then separate the images. This is extremely slow if you're dealing with a large dataset.

A faster less robust way would be to do a Principle component analysis

step1 :  Do PCA to reduce the dimensionality to a 2D or 3D space
step2 : Plot and see if there are any clusters. Usually, images which are different fall into different clusters
step3 : Cluster it via a convex clustering algorithm like K-means
step4 : Store the images belonging to a cluster into a folder.
step5 : Go through the folders and make the necessary cleaning.


R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable?

If I understand your question correctly you'd like to obtain the proportion of missing values of a specific column in your dataset.
This can be obtained with the plyr library

For example:

library(plyr)
df <- data.frame(Study=c('Study_1', 'Study_1', 'Study_2', 'Study_3'), col1=c(NA,NA,5,8), col2=c(8, NA, NA, 7))

column <- 'col1'           # is the column your are interested in

ddply(df, .(Study), function(x){
  length(which(is.na(x[column])))/nrow(x)*100
})

Study  V1
Study_1 100
Study_2   0
Study_3   0

Study_1 has 100% missing values and the other studies none, so except Study_1 all studies contains some information


Transitioning from a python script for data transformation to BigQuery

Possible Solution:
Assign a UUID to each email/user instead of an auto-increment ID (first making sure email does not already exist else use the existing UUID).
UUIDs can be created anywhere - not necessarily in DB - and are almost guaranteed to be unique accross time and space.

For example, the number of random version-4 UUIDs which need to be
generated in order to have a 50% probability of at least one collision
is 2.71 quintillion, computed as follows:
$${\displaystyle n\approx {\frac {1}{2}}+{\sqrt {{\frac{1}{4}}+2\times \ln(2)\times 2^{122}}}\approx 2.71\times 10^{18}.}$$
This number is equivalent to generating 1 billion UUIDs per second for
about 85 years. A file containing this many UUIDs, at 16 bytes per
UUID, would be about 45 exabytes.

Note: User Email can also be used as unique ID and makes everything easier.


How does Excel obtain the R² determination coefficient in an exponetial regression?

Please bear in mind that excel does not handle missing values well. If your data column has some blank cells in it, if you filter them out, the r2 can change (subtly). Excel is fine for an overview but use a more robust method in case you want to get formal figures for publication.


word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7

Let us try and understand how Word2Vector actually works before looking at distances:

There are 2 ways of generating vectors for a word :

Continuous bag of words
Skip grams

The following diagram explains the difference between the two approaches.

In case you want to further understand the nitty gritty of these two approaches, there are tons of blogs out there.

Now let us look at what the image is trying to tell us.

For CBOW, it looks at the neighboring words to provide a probability of another word co-occurring in that space.

For Skipgram, given a word, it provides probabilities of all the other words occurring in the neighborhood.

Why is this important to know?

Without understanding what these algorithms are doing, its not quite possible to solve an NLP problem as word vectors are the primary inputs to any deep learning / machine learning algorithms.

What are the other capabilities Word Vectors provide us with?

As you have vectorized Words in a n-dimensional space, it gives you the ability to perform fast operations such as computing distances in this n-dimensional space. One such way is cosine distance, that you have outlined to have used.

Understanding how distances are computed can also help. Here is a small explanation of how Cosine similarity works :

When you look at vectors project in n-dimensional space, you find the difference in the angle between these vectors. If the angle is small, lets say 0, then cos(0) = 1, which implies the distance between these vectors is very small, thereby making them similar vectors.

Given this information, what can you infer from your results of distances between words:

Your corpus has diverse distribution of words that could possible co-occur with a large number of words, thereby making distances between similar words not > 0.7
Given the variation of words and possible presence of large amount of stop words in your vocabulary, its hard for a shallow neural network to provide accurate vectors for your words.

Suggestions that can help improve the vector quality:

Using TFIDF, you can remove stop words in case they are not interesting to you.
Pre-process your text to bring them to standard form. For example, lower casing of text, removal of special characters, stripping of spaces.
Experiment with varying embedding sizes. Start with as little as 100, as you are interested in only finite number of words.
Visualize your vectors using tsne / PCA and projecting them to a 2 or 3 dimensional space.
Perform the same for different epochs to see how vectors are converging / diverging.
Its also very important to understand your corpus well, so that you know what these vectors imply and how they co-occur.

Hope this helps.


Implementing Batch normalisation in Neural network

Give the whole batch as an input, take the activated output of the first layer and pipe it the next until the final. .

calculate the gradient decent direction (derivative of the cost function) for each row. U will have the error for each row of ur training set.

Then add the errors and divide the result with number of the training set. Then finally update it weights.
I know from experience that tracking all these numbers is hard.

I don't know what language u use, but if it's Python, numpy will help u a a lot. For example, to add the errors, errors.sum(axis=0). X.dot(weight) to multiply ur entire dataset with ur weight.  So my suggestion is use some matrix library.

One thing I want to mention is batch processing is susceptible to local minima which u don't want...especially in complex functions. SDG is a nice algorithm. I'm not sure why u needed batch update. There are many interesting tweaks in NN, don't waste ur time on batch update, use SDG


This might not be the best solution:

I'm attaching the new code in which I split the data using a threshold and then plot them separately into 2 pie charts:

import matplotlib.pyplot as plt
import pandas as pd

plt.style.use('ggplot')

dic = [{'class a': 26.9,
       'Class b': 18,
       'Class c': 16.8,
       'Class d': 13,
       'Class e': 8.83,
       'Class f': 5.97,
       'Class g': 3.59,
       'Class h': 2.01,
       'Class i': 1.42,
       'Class j': 1.09,
       'Class k': 0.903,
       'Class l': 0.873,
       'Class m': 0.28,
       'Class n': 0.24,
       'Class o': 0.112}]

labels = dic.keys()
sizes = dic.values()

#converting the data into a dataframe
df = pd.DataFrame(dic)
dft = df.transpose()

dft.columns = ['Value']
dft['Class'] = dft.index

#defining the threshold
threshold = 5
dft_subset1 =  dft[dft['Value'] <= threshold]
dft_subset2 =  dft[dft['Value'] > threshold]

plot1 = dft_subset1.plot.pie(y='Value', figsize=(5, 5))
plot2 = dft_subset2.plot.pie(y='Value', figsize=(5, 5))

Output:


Predictive Analytics on distributed systems vs standalone system

There is nothing special about building something across many machines, or Hadoop, so maybe we can take that out of the question.

One general reason is that, even if the model building process were worse, building on more data could result in a better result.

You're asking why building many models on subsets of data could be better, but this is precisely why ensemble models like random forests can outperform single instances.

But lastly, it's not true that distributed algorithms always work by making an ensemble of small models, so the premise is not strictly true.


After performing data Augmentation on tf.data.Dataset, should i MERGE it with original tf.data.Dataset?

Yes, use all the data you can.

The goal behind data augmentation is to indeed create more data points from a smaller set of instances. Therefore, you definitely want to keep your original data in there, since especially, this is the only "pure" data you have.


Step-by-step construction of an RBF neural network

The hidden-layer has as many neurons as there are training samples.

This already sounds a bit fishy to me. The hidden layer of an RBF network resembles the Radial Basis functions. Thus, you have to think about how many centers shall your network have. This should not correspond to the number of training samples your have, since this would force the network to learn by heart. Let's look at the example you gave in the question:

Here I propose to use a network with four hidden neurons, corresponding to four RBF centers, marked with stars. This is, in the end, how your network should learn to generalize over the data you feed into it. Note that you do not have to set the class of an RBF network in prior, the network is supposed to learn the classes of each neuron by itself. How this is done is properly explained on the wikipedia page. A nice thing about RBF networks is that there is an analytical solution (over the Moore-Penrose pseudoinverse) for your weight vector.

  That does not seem to be correct however - if I have the same $\Sigma$ and $\mu$ for all the hidden nodes of the same class, each new test-input would result in the same activation for all these nodes..so what exactly are the particular $u_i$ and $σ_i$?

I suppose you are talking about the centers $\mu_i$ and their width $\sigma_i$. These values, like described in the link above, can be set via a clustering over the data you have. I've also seen applications that adapt the centers online, but this is usually not necessary. A simple unsupervised learning / clustering approach is typically sufficient if you have all training data available.

  Also, for perfect classification, I would set all the weights which belong to the correct class to 1, and all the others to zero - would that make sense?

If your designed network is able to solve your task, the analytical solution will do the job. Wikipedia describes how it is calculated.


Categorical vs continuous feature selection/engineering

For encoding categorical features, there is two common ways:

  Ordinal encoder

This is the way you mentioned as 'encoded as integers'. In this method, an integer starting from 0 is assigned to each category. The problem of this method is that it randomly prioritize categories. So in cases when there is no priority among categories, this encoding is meaningless as you mentioned. The only case it work is when assigning larger integer to some categories is meaningful.

  One-hot encoder

This method makes a feature vector (one-hot vector) for each categorical feature which is the same size as the number of categories. The method assigns each component of the vector to one of the categories. For each data sample, it assigns 1 to component which its corresponding category is present at the sample and assigns 0 to other components. The benefit of this method is that unlike ordinal encoder it does not prioritize any category.

So in your case, I highly recommend that you use one-hot encoder.


In case anyone has the same issue, please follow the github thread at:

https://github.com/CodeReclaimers/neat-python/issues/79


Survival Analysis: Pseudo Observation Vs Stratified Cox Regression. Which one is better?

I suggest using a model with more relaxed assumptions on proportionality of hazards. In my work I use piecewise constant hazard model, which works wonderfully. Its assumption is that the hazards are proportional in a time interval. It allows using numerical covariates with splines, and time-dependent covariates. Moreover in my experience the model is usually very well calibrated and does not overfit much.


Building Recommender for book paragraphs

You would have to come up with a metadata set of each paragraph and then create a content-based recommendation system based on that metadata. You could also do a community-based recommendation system if you have extensive user data.

As for the metadata set, you could use something like n-grams to "summarize" each paragraph and then have an index for each. From there, you would know what people like and then algorithmically choose the n-grams that person is most interested in and find the paragraphs that correspond to those n-grams.


First, transform your columns, then apply linear regression, but do you want to know about the influence of your features on your selected dependent variable?

Read this article: http://www.ritchieng.com/machine-learning-evaluate-linear-regression-model/

It provides great insight at how interpret the coefficients given by the algorithm can be interpreted and further discussed or tweaked.
Read it, then come here for further doubts, you're more than welcomed to do so.


How to handle such a large class imbalance in text data?

For the most part, as you add more classes to your multiclass classification problem, it becomes more difficult to construct a model. All algorithms can run into trouble because there are fewer examples of each class to learn from. In case the 27 class target feature does not work well for you, you can decrease the number of classes even more. To do this:

Continue what you already did by grouping small classes into the "other" category. Perhaps a model with 2 or 3 classes would be useful for you.
Try a semi-supervised learning approach. Try clustering of your different categories and group similar categories into the same cluster. Your model will now predict cluster A, cluster B, etc.


How to consider categorical variables in distance based algorithms like KNN or SVM?

The traditional way of dealing with categorical variables is to one-hot encode these. One hot encoding will transform one categorical variable with n values into n different features which values can be only 1 or 0, 1 indicating the presence of the corresponding category and 0 the absence.

As an example, instead of having:

---------------
|id  |Category|
---------------
| 0  |    A   |
---------------
| 1  |    B   |
---------------
| 2  |    A   |
---------------

You would have:

--------------------------------
|id  | Category_A | Category_B |
--------------------------------
| 0  |     1      |     0      |
--------------------------------
| 1  |     0      |     1      |
--------------------------------
| 2  |     1      |     0      |
--------------------------------

Distance based algorithms then use the vectors representing the features to compute distances, whether it is the euclidean, cosine, hamming distance... After one hot encoding, you can see that these vectors are exclusively numeric, the 'categorical' aspect being translated into the addition of new dimensions.

With one hot encoding, there are a few common things you must watch out for:

It is recommended to drop the last column created by one hot encoding, because it can be inferred from the others. As an example, if you have a feature F taking three possible values A, B and C, one hot encoding would produce three columns F_A, F_B and F_C. You have F_C = not(F_A + F_B), i.e. 'if the value is neither A nor B then it is C'. In this case you should keep only columns F_A and F_B (or F_A and F_C , F_B and F_C whatever) since the third column is redundant and adds therefore no information. Keep in mind that if you F allows the presence of missing values, this has to be taken in account by either imputing the values, or considering the 'missing value' as a fourth possible category.
One hot encoding augments the dimension and the sparsity of the dataset. This can also harm the performance of the algorithm, especially when you have a lot of categorical variables with a lot of possible values.In order to avoid that, you can apply dimensionality reduction (as an example with PCA) to the one hot encoded dataset, and/or perform feature selection.


A confusion matrix is a table that is often used to describe the performance of a classification model. The figure you have provided presents a binary case, but it is also used with more than 2 classes (there are just more rows/columns).

The rows refer to the actual Ground-Truth label/class of the input and the columns refer to the prediction provided by the model.

The name of the different cases are taken from the predictor's point of view.

True/False means that the prediction is the same as the ground truth and Negative/Positive refers to what was the prediction.

The 4 different cases in the confusion matrix:

True Positive (TP): The model's prediction is "Positive" and it is the same as the actual ground-truth class, which is "Positive", so this is a True Positive case.

False Negative (FN): The model's prediction is "Negative" and it is wrong because the actual ground-truth class is "Positive", so this is a False Negative case.

False Positive (FP): The model's prediction is "Positive" and it is wrong because the actual ground-truth class is "Negative", so this is a False Positive case.

True Negative (TN): The model's prediction is "Negative" and it is the same as the actual ground-truth class, which is "Negative", so this is a True Negative case.


Pyspark coverting timestamps from UTC to many timezones


Why does my model learn with Ragged Tensors but not Dense Tensors?

So turns out the answer was that the shape of the dense tensor was different across the training set and validation set. This was because the longest sequence differed in length between the two sets (same with the test set).