Linear Model to generate probability of each possible output

From what I understood -

Train a classifier with the features as you mentioned and the output probabilities like you have shown -

So, for Player A, you can use all the features (average runs scored overall, average score in past 10 innings etc.)
Treat the runs scored as a categorical variable and try to learn a classifier
Put it through the network with a softmax layer at the end - that will give you Sum of Probability should add up to 1

My recommendation

Each player will have scored different runs so you will not get enough data points in a particular category (so no. of people who score 83 will be really less for example)
Pre-group the ranges in your training data and then do the classifier.
Having a softmax layer at then end won't give you the "linear model" you want, but would definitely solve the problem with a lot more clarity.


Identify members who are likely to switch where they receive drug administration

Given that the data is labeled, just perform supervised approaches, they will almost always beat unsupervised.

Intuition why thats the case is because we dont have target function in unsupervised approach. In other words function that discriminates classes given our data set. I like to think that in unsupervised learning this function is identity function and not (what in reality is) some complex one. Given an entire data set, we are asking for a partitioning of the input space, with particular properties (the partitions contain almost all examples, each partition is not too large, the partitions are not too close to one another) but this kind of partitioning does not map to the real classes (since we are expecting classes that are clearly seperated in the dataitself but in reality they have some non-linear seperation, and this non-linear/complex behaviour is represented in terms of some function. That supervised approaches approximate (you can think of almost every supervised approaches as an optimisation))


How to find unique occurance of a value based an another value using pandas?

Assuming that you want to return students who only took one class, You can use groupby and filter:

df.groupby('Student_code', as_index=False).filter(lambda x: x['Subject'].count() ==1 )

  Student_code    Subject
3          124      Maths
4          125  Chemistry


How to calculate convolution for 2nd conv Layer in CNN, Do we need to average across all feature maps?

No, you don't average across all feature maps.
When the input has multiple channels, you need your convolution filter to have the same number of channels. Therefore, the filter "covers" the full depth of the input. Then, you simply perform the element-wise multiplication of the filter with the overlapping region in the input and add all the resulting elements together.
Therefore, if the input to a convolutional layer is a grayscale image, a filter is of dimensions [3,3,1], while if the input is a color image (with 3 channels), your filter is [3, 3, 3], like this (source):

The same happens with feature maps with N channels: we need a filter with N channels.


Theano vs Tensorflow for building Neural Networks for NLP tasks

At the end this comes down to one's opinion and experience but I ll attempt to give you a decent answer.

Theano has been around longer and you will be able to find more resources on it. Both to learn and people's published work on it.

On the other hand Tensorflow has been getting better and better since it came out. It has a free online course available and Google even released an open model for natural language understanding.

You may also have to check what hardware you have available. Tensorflow does not run on as many hardware as Theano currently. (Although they are actively working on Windows and OpenCl support.)

A good comparison on various Deep Learning Frameworks can be found there.


Why does Transfer Learning works better on smaller datasets than on larger ones?

In the article you provide, from page 11 results, I think one cannot conclude that transfer learning works better on smaller datasets than on larger ones.
If you look at the results of transfer learning score values (or RMSE) vs size of learning, it is also getting better while dataset size is increasing (for instance E2 or E5 or E8). So transfer learning does not work better on small datasets.
However, you might be looking at the IMP index which is based on the mean score (or RMSE) of learning with and without transfer learning.

IMP= (1−(WithTransfer)/(NoTransfer))×100

The index is based on two curves.

WithTransfer which will have good performances even at the beginning because when using relevant transfer learning, the model could already extract pertinent information from a very small testing dataset.
NoTransfer which will start with poor performances (difficulty to generalize) and  then increasing with the size of testing data.

The IMP index has then the expected curve you're pointed out, for example with E2 and E5.


Predicting sequence element based on the previous M and the following N elements

The framing of your problem is close to so-called language modeling task. Because your input data is fixed-length samples, you can use a seq2seq model with fixed-size context embedding.
What this means is you would essentially have an encoder, Bi-LSTM for example which encodes your input into a fixed representation (by concatenating the final output states of forward and backward LSTM), and a decoder, for example LSTM, which sequentially produces the output tokens.
Your objective function could be a mean of cross-entropy loss over each output token, or a more complex loss like CTC.
You can also simplify it by just predicting the masked tokens, instead of the whole sentence, as output of your neural network.
The fact that your tokens are integers makes no difference and actually simplifies the embedding. You can simply feed the data as is to an embedding layer in Keras or PyTorch.
If you use PyTorch, there is this tutorial that I would recommend, using transformer instead of LSTM.


Help in understanding the maths behind Logistic Regression

This feels like a bit of a convoluted way to introduce the concept, but alright :D Let me start at a slightly different point.

Maybe in Machine Learning or in other places you have encountered the $sigmoid$ function:
$$ sigmoid(S) = \frac{e^S}{1+e^S} = \frac{1}{1+e^{-S}} $$
The sigmoid has the nice property to map any real number $S$ to a number between 0 and 1. This is super when dealing with models that have to represent probabilities.
As in the slides they are looking for a (conditional) probability $p(x)$ they go ahead and implicitly set
$$ p(S) = sigmoid(S) $$
They just do it around an extra corner. In order to be consistent with the slides (and to save some tedious writing) I'll now keep talking about the probability $p(x)$ and think of the $sigmoid$-function.
$p(S)$ (i.e. the $sigmoid$) has an inverse function, called the $logit$. Let's try to find the inverse of $p(S)$:
$$ p = \frac{1}{1+e^{-S}} $$
and from there we move some stuff around
$$ 1+e^{-S} = \frac{1}{p} $$
$$ e^{-S} = \frac{1}{p} - 1 $$
$$ -S = \log \left[\frac{1}{p} - 1 \right] $$
$$ S = \log \left[\frac{1}{\frac{1}{p} - 1} \right] $$
$$ S = \log \left[\frac{p}{1 - p} \right] $$
This is where they start in your notes. We can take any number $S$, punch it into the logit, solve for $p$ and Boom! we've got ourselves a nice conditional probability.
So the next question is: what is $S$? $S$ is supposed to be some function of $x$. There is a wide variety of functions you could use. You could even put a massive Neural Network, but for now, let's stick to linear regression
$$ S(x) = \beta_0 + \beta x $$
If you decide to go for $S(x)$ being linear, you can now go step 3. backwards and end up at step 1. with the expression they also show in the notes
$$ p(x) = \frac{e^{\beta_0 + \beta x}}{1+e^{\beta_0 + \beta x}} = \frac{1}{1+e^{-(\beta_0 + \beta x)}} $$

So to answer your questions:

$1-p(x)$ is not in the denominator of the distribution, it seems to me they just introduce it in the $\log\frac{p}{1-p}$ in order to end up with the $sigmoid$.

$p(x)$ is not equal to $e^{\beta_0 + \beta x}$, but equal to the $sigmoid$.

$\log \frac{p}{1-p}$ is declared to be equal to $\beta_0 + \beta x$ in order to end up with the $sigmoid$.

I think it would have been more straight-forward to say that you want to map a linear function to something between 0 and 1, which you can do with the sigmoid and then you would have been done in one step. It seems a bit weird to introduce the inverse of the sigmoid, claiming that this was some property you want and then solve for $p$, but that might be a matter of taste.


Prerequisites for Elements of Statistical Learning

The authors of Elements of Statistical Learning have come out with a new book (Aug 2013) aimed at users without heavy math backgrounds.

An Introduction to Statistical Learning: with Applications in R

Download from here: http://faculty.marshall.usc.edu/gareth-james/


Some ideas:1. Could you assign features for jobs and assignments? Performance would be the labels. Then you could simply use an SVM for example.
2. Given a user representation matrix X, each row i is an agent, and a job representation matrix Y, each j is a job representation, and a matrix of performance P, where every row stands for a user and every column how well the agent performed on that job, you could learn X and Y through optimization. Check this: https://en.wikipedia.org/wiki/Non-negative_matrix_factorization
I hope this helps.


Why isn't local averaging (including KNN) used often for regression?

Local averaging regression is much more complicated model than a simple linear regression as it's a mixture of many individual smaller models. Local regression is actually quite popular in the literature, and has been tested extensively.

Local regression models require large dense sampling data set, which is not common for a statistical student. If you don't have complicated relationships to model, you probably don't need it.


Compute Jaccard distance between two lists of strings

Python lib textdistance is a "python library for comparing distance between two or more sequences by many algorithms." It includes the Jaccard index. Having the similarity, you can get the distance by $Jacc_{distance}(x,y) = 1  -Jacc_{similarity}(x,y)$.

Although installing and importing the whole module wouldn't make much sense if this is all you need, reading the module's README may give you new ideas on the various distances you could use in your project, and take advantage of having all of them already implemented, so you can test how each of them performs really quick.


Yes, this can be done: here’s a paper from 2015 indicating recent developments and there’s a handy dandy tutorial here to get you started using scikit-learn’s ICA implementation. Enjoy!


CNNs are applicable wherever the input signal contains spatial information. For instance, you can think of the following list:

1-D signal: Recorded voices are examples of one-dimensional inputs where there are relations between adjacent entries of inputs. This means adjacent entries have patterns which are valuable for different tasks, such as classification. You can employ 1-D convolutional layers for these input signals.
2-D signal: Images are an example of this kind, albeit they may have different channels, like RGB. It's clear that adjacent pixels are roughly like each other, and other than that, the adjacent pixels share patterns which may be repeated over the entire image multiple times. Consequently, 2-D convolutional layers can be employed for these signals.
3-D signal: Video frames are examples of this kind. Other than the similarity you can find inside each frame, the different frames that exist one after another in the time axis can have similar meaningful patterns which are replicated. You can employ 3-D convolutional layers for these signal.

About structured data where you can find them as the rows of relational databases which each column belongs to a specified feature, it does not have any meaning to use convolutional layers. The reason is clear. They do not have any spatial information. The adjacent rows should not share a common concept otherwise they would be redundant. Moreover, they are not spatially related to each other. For structured data, people utilise dense layers.


Significance of Object-Oriented Programming (OOP) in Data Science

It actually depends on the role you get as a Data Scientist. If you have to write production-quality code at a large software company, then you need to be knowing the basics of Object-Oriented Programming (OOP). Object-Oriented Design (OOD), however, is something you need not necessarily know in a Data Science role. Learning OOD in case you plan a switch to software engineering roles in future is one thing you can consider, though.
Regarding the choice of language for doing Data Science, I would suggest you  prefer Python over R as it's more versatile and a much more general purpose language. While R, on the other hand, is limited only to applications in statistical programming.


Orange 3.15 GUI window size issue (Windows 8.1)

This is a known issue: https://github.com/biolab/orange3/issues/3193

It is caused by a bug in Qt 5.9.6 (QTBUG-16252). You should downgrade Qt to 5.9.5.

To do that open the "Orange Command Prompt" from the Window 8.1 'Start Screen' (start menu) and type in conda install qt=5.9.5


How to re-train a model from false positives

In most cases, one shouldn't retrain a trained network with only the new data. Rather, train the network from scratch with the new and old data.

Adding new data and retraining the model just on that new set of data, will probably make your model fit to only that new data, thus forgetting general features from the other data it was trained on.

Also, instead of selecting your final model based on only validation loss, you should select your model based on a validation metric. For example, in your case, it could be accuracy, precision, recall etc


What are the simplest methods for the label noise problem?

If the labeling people is sufficient,
the best solution is to re-label the noisy data.


Why seq2seq models are superior to simple LSTMs?

Encoder-decoder architectures are not simply "more powerful" than a mere LSTM.
LSTMs can't be used (in their standard configuration) for general sequence transduction tasks. On the other hand, encoder-decoder architectures are conditional autoregressive models, that is, they generate a sequence element by element conditioning on another sequence. This difference is what justifies the rather different use cases for simple LSTMs and encoder-decoder architectures.

Among LSTM-based encoder-decoder architectures, we should distinguish different types by the way information is passed from the encoder to the decoder. The simplest form is to simply pass the last hidden state of the encoder LSTM to the first decoder LSTM; this implies that all the information from the input sequence is "compressed" into a fixed-length vector, which is known to be an information bottleneck. More complex forms include using attention mechanisms, where the encoder hidden states at each time step are combined into a different previous context vector for each decoder LSTM; here there is no bottleneck, and normally their results are much better.


I'll go ahead and post an answer for now; if someone has something better I'll accept theirs.

At this point the most powerful option appears to be accessing WEKA using jRuby. We spent yesterday scouring the 'net, and this combination was even used by a talk at RailsConf 2012, so I would guess if there were a comparable pure ruby package, they would have used it.

Note that if you know exactly what you need, there are plenty of individual libraries that either wrap standalone packages like libsvm or re-implement some individual algorithms like Naive Bayes in pure Ruby and will spare you from using jRuby.

But for a general-purpose library, WEKA and jRuby seem to be the best bet at this time.


Can I use categorical data and Decision Trees to regress a continuous variable?

Yes, most software implementations of trees will allow you to predict a continuous target variable with all binary predictors. This is because the predictors are only used as splits, and the prediction comes from the average value at a given terminal node. The predictions will not be truly continuous across all terminal nodes in the same way that linear regression is continuous, but in practice, this is generally not a problem. If your tree is under-fitting (not continuous enough) you can always add more terminal nodes. Also, one-hot encoding should be sufficient.


What is this called by gathering the meaning from a sentence?

It is called relation extraction, it is a subdomain of Natural Language Processing. Often, you first have to apply Named Entity recognition or Concept extraction to extract the entity or concept. You can then extract the relations betweens the entities with several approaches.


Best way to determine the number of features for RFE (recursive feature elimination)

I encourage you to look at this method https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html#sklearn.feature_selection.RFECV, it allows you to recursively test all of your features based on the scoring method of your choice including F1


re. "estimate latent variables"

Quantities that are trained in order to fit a "best" model within a family of models are called hyper-parameters.  To any instance of the model they are fixed.  To the optimisation routine they are an index into search space.  Adding constraints on the range of a hyper-parameter both reduces the search space of the optimisation and requires extra "feasibility" checks during typical gradient descent.

A variable is "latent" when it is purely internal to the model, i.e. not an observable.  The meaning of its scale would depend on the context and on your interpretation, since it cannot be compared to anything observed.  You rarely want to constrain that range inside the model.

I would suggest leaving the hyper-parameters and latent variables unconstrained and if you want to read an output train a "neuron"-like response to get what you want out : e.g. sigmoid / tanh / softmax


How to more simply see overlapping data for dozens of variables?

For this example specifically, I would suggest visualizing the data using a Chord Diagram.

Chord diagram from Delimited.io:

A chord diagram would allow you to see interactions and co-occurrences of likes between each of the heroes directly, including relative magnitude of the effects intuitively and immediately. You could also include other properties of the character (universe of origin, gender, time period of introduction etc) by use of color and/or positioning of the hero in question on the circumference of the diagram.

As chord diagrams are a graph-based approach, you'd want to transform the individual observations you have into what would be a (this has to be a first for this term) hero-like-cofrequency matrix formatted as follows:

+-----------+--------+------------+----------+
|           | Batman | Spiderrman | Superman |
+-----------+--------+------------+----------+
| Batman    | 0      | 2          | 3        |
+-----------+--------+------------+----------+
| Spiderman | 2      | 0          | 1        |
+-----------+--------+------------+----------+
| Superman  | 3      | 1          | 0        |
+-----------+--------+------------+----------+

Note that this represents an undirected graph, and that it is symmetric about the identity column.

As you're using Python, you may want to look into using Bokeh to create the chord diagram. One tutorial for doing so is available here.

Best of luck, true believer!


Can tanh be used as an output for a binary classifier?

If you label your data using -1 and 1 as classes, then yes you can. However, there are two reasons why data scientists normally prefer Sigmoid activations:

Loss functions, such as cross entropy based, are designed for data in the [0, 1] interval.
Better interpretability: data in [0, 1] can be thought as probabilities of belonging to acertain class, or as a model's confidence about it.

But yeah, you can use Tanh and train useful models with it.


DTW (Dynamic Time Warping) requires prior normalization?

I am glad you asked ;-)

In 99% of cases, you must z-normalize.

Want to know why? I wrote a tutorial on this, page 46
http://www.cs.unm.edu/~mueen/DTW.pdf


How to interpret the AUC score in this case?

If you want to figure it out how this ROC happens , you would better LIST the tuples including your "predicted" values and the "truth" values, and SORT with the "predicted" value , then PLOT the ROC .
In your case , the tuples and points should be like this :
  predicted truth (x,y)
    0.53 0  (6/6,14/14)
    0.55 0  (5/6,14/14)
    0.57 1  (4/6,14/14)
    0.59 0  (4/6,13/14)
    0.60 1  (3/6,13/14)
    0.62 1  (3/6,12/14)
    0.63 0  (3/6,11/14)
    0.64 0  (2/6,11/14)
    0.66 1  (1/6,11/14)
    0.68 1  (1/6,10/14)
    0.71 1  (1/6,9/14)
    0.73 1  (1/6,8/14)
    0.77 1  (1/6,7/14)
    0.78 0  (1/6,6/14)
    0.82 1  (0/6,6/14)
    0.86 1  (0/6,5/14)
    0.89 1  (0/6,4/14)
    0.92 1  (0/6,3/14)
    0.94 1  (0/6,2/14)
    0.96 1  (0/6,1/14)
            (0/6,0/14)

Put these points (x,y) on ROC picture , it should be like this :

Just much like that of yours !
By the way , if you want to know how these points are figured out , you can check the code written in scala in the below :
    def computeAuc(predict: BDV[Double],groundTruth: BDV[Double]): Double
    = {

        // retrieve number of positive and negative samples in ground truth
        val nPos = groundTruth.toArray.filter(_>0).length
        val nNeg = groundTruth.toArray.filter(_<=0).length

        // tuple predict with ground truth , and sort with predict
        val pair = predict.toArray.zip(groundTruth.toArray)
        val sortedPair = pair.sortBy(_._1)
        var auc = 0.0.toDouble
        val x = BDV.zeros[Double](predict.length + 1)
        val y = BDV.zeros[Double](predict.length + 1)
        x(0) = 1.0
        y(0) = 1.0

        // calculate auc incrementally
        var i = 1.toInt
        while(i < sortedPair.length) {
          y(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 > 0).length) / nPos
          x(i) = (1.0 * sortedPair.slice(i,pair.length).filter(_._2 <= 0).length) / nNeg
          auc = auc + (((y(i) + y(i - 1))*(x(i - 1) - x(i)))/2.0)
          i += 1
        }
        auc = auc + ((y(i - 1) * x(i - 1))/2.0)

        auc
   }

Finally , your imbalance problem is severe , and you would better do down-sampling or up-sampling before training .
Hopes this contributes you , good luck -)


What does it mean when someone says "Most of the data science algorithms are optimization problems"

Most algorithms try to minimizes some objecive functions.

For example, in linear regresssion, given $(x_i, y_i)$, we try to find $\hat{y}_i= \alpha_0 + \sum_{j=1}^d \alpha_j x_{i,j}$ and we we want it to be close to $y$. We try to minimize the mean square error in our estimation.

That is our objective function is
$$\min _\alpha\frac1n \sum_{i=1}^n (y_i-\alpha_o- \sum_{j=1}^d \alpha_j x_{i,j})^2.$$

We might have a model with unknown parameters and we can use maximum likelihood to find out our model, in that case, we maximize the likelihood function. Again, we get another optimization problem.

In general supervised classication error, we are trying to minimize the error which is typically formulated as a minimization of a loss function.  In SVM, we are trying to find a boundary that maximizes the margin between the two classes, again, we are trying to maximize an objective function.


How are non-restricted Boltzmann machines trained?

I will apologize in advance for linking to the Wikipedia article, but I'm not sure I can do a better job explaining it.  https://en.wikipedia.org/wiki/Boltzmann_machine

(Will extract selected passages to prevent link rot shortly.)

One of the reasons we see so much talk about training RBMs and not much talk about the other is, by my recollection of a talk by Hinton, "Intractable, except for very small problems or limited cases."  The activation of a hidden node depends not only on the state of visible nodes, but on its compatriots. This means our clever contrastive divergence and gradient decent methods don't work!

But you didn't ask why it's difficult, you asked how to do it.

The short, short answer is simulated annealing.

The long short answer, we select an initial state for the network, then perform updates until the network reaches a global thermal equilibrium, then try and adjust the weights while reducing the temperature until our activations approximate a global minimum.  The process repeats until we're happy with the outcome.


They serve different purposes.

KNN is unsupervised, Decision Tree (DT) supervised.
(KNN is supervised learning while K-means is unsupervised, I think this answer causes some confusion.)
KNN is used for clustering, DT for classification. (Both are used for classification.)

KNN determines neighborhoods, so there must be a distance metric. This implies that all the features must be numeric. Distance metrics may be affected by varying scales between attributes and also high-dimensional space.

DT, on the other hand, predicts a class for a given input vector. The attributes may be numeric or nominal.

So, if you want to find similar examples you could use KNN. If you want to classify examples you could use DT.


How to represent relation between users as a feature?

There's a couple of approaches you can take depending on the nature of your data. It sounds like you're trying to detect social anomalies in your data so you need to model the communication boundaries between them which leads to some sort of graph representation.

If you don't have too many users in the system (say $n$) then you can create an $n\times n$ matrix, $M$ over a time period that represents the communications between users. The component $M_{ij}$ could be either $1$ or $0$ if user $i$ and $j$ communicated or the number of times that they communicated.

If you have more data then you would want to represent the data in terms of nodes and edges. Nodes would be the users and edges would be the presence of a communication. This can be done manually or by using a library such as NetworkX.

Here's a Python tutorial on getting started with graph network analysis.

If you're doing this at a large scale then you might want to use a graph database such as Neo4J.


Negative value in information gain calculation through gini index

I quickly checked your calculation and you seem to have miscalculated the gini(annual income)

  gini(annual income)=1-((5/20)^2+(12/20)^2+(3/20)^2) = 0.445

When it actually equals 0.555 (you probably forgot the 1-... part) which is larger than 0.532 so you might be fine for the rest of the calculations.


Why normalize function has a different result on a matrix vs single value?

The normalize function works by calculating z-score for the given data. Now z-score is given by:
$$z_{i} = \frac{x_{i} - mean(X)}{s.d.(X)}$$
where $X$ is your original vector/matrix.

Now, when you give a vector as input it first calculates the mean and standard deviation for the row/column. Then the output you ask for is calculated the above formula. So, if you only ask for a scalar it only shows one value. You can see in your example the normalized value(or z-score) is same for that particular position i.e. $0.24444$.

But when you give only a single value as input the z-score is 0 by definition. For more on it's working and use take a look at the docs.


I'm not an expert here, so here's my (brute force?) method.

SeatGeek has open-sourced a python library called fuzzywuzzy which is great at text matching.  It has a function called token_set_ratio that compares two multi-word strings and scores their distance.  It can consider just the intersection of individual words and score only that intersection. eg "Barack Obama Bill Clinton Madeleine Albright" will get a score of 100 with "Bill Clinton", but also with "Bill Barack".

You'll need to build a list of known names, judging by the list they're famous people, which a quick google search shows hundreds of sites that have that ready for you.  With that list, you can just loop your way through, computing a score with token_set_ratio.  If the famous name beats some threshold score against your string, you append it to a list of captured names.


What are "belief maps" and "affinity maps"?

In computer vision, pose estimation is the detection of an object's orientation and positioning.

Belief Maps

A team from Nvidia have proposed "Deep Object Pose Estimation", which has two main components:

Detect objects in image and generate 2D keypoint mappings
Project 2D mapping to 6D

From what I can see, "belief maps" are generated by the first component of their system and describe elements of the 2D keypoint mappings that are then projected to 6D by the second component. From the article:

  The feedforward network takes as input an RGB image of size w×h×3 and
  branches to produce two different outputs, namely, belief maps and
  vector fields. There are nine belief maps, one for each of the
  projected 8 vertices of the 3D bounding boxes, and one for the
  centroids. Similarly, there are eight vector fields indicating the
  direction from each of the 8 vertices to the corresponding centroid...
  to enable the detection of multiple instances of the same type of object.

Affinity Fields

I've not come across affinity maps, but affinity fields are referred to above as inspiration for their approach to pose estimation, specifically in "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields". That article aims to solve the problem of how to attribute limbs to people in video containing more than one person.

Affinity maps are "an explicit nonparametric representation of the keypoints association that encodes both position and orientation of human limbs", which you can see a demo of in their video (which is pretty cool).

So bringing it back to the first article, affinity fields are analogous to the vector fields mentioned above. They're used to associate body parts with a particular person in 5, or to associate vertices of belief maps to a centroid representing an object in 4.

So belief maps represent the features extracted by their deep CNN, and vector/affinity fields help associate those vertices with an item that's been detected.


Is it possible that MLP has better accuracy than CNN?

Yes, it's possible that an MLP has better accuracy than a CNN.
Here's one discussion

CNN (and RNN) models are not general improvements to the MLP design.
They are specific choices that match certain types of problem. The CNN
design works best when there is some local pattern in the data (which
may repeat in other locations), and this is often the case when the
inputs are images, audio or other similar signals.

You say:

Or maybe it is mathematically proven that there is always some CNN
which will beat MLP for the given classification problem.

which is an interesting question.  MLP's and CNN's are actually "equivalent" in the sense that if you have an MLP, I can write down a CNN that is identical to it (produces the same output), and vice versa. See discussions here and here.
All this is to say, it's not "CNN vs MLP" that matters as much as the details of the architecture / optimization / etc. The first answer I linked to explains why typical CNN architectures work well.  The phrase "some local pattern in the data (which may repeat in other locations)" is a way of saying there is some translation equivariance in the data.  If there is not translation equivariance, an MLP might work better.


A practical reason to use Cross-entropy as a error-function in Neural networks?

During back-propagation training, you want to drive output node values to either 1.0 or 0.0 depending on the target values.
If you use MSE, the weight adjustment factor (the gradient) contains a term of (output) * (1 – output). As the computed output gets closer and closer to either 0.0 or 1.0 the value of (output) * (1 – output) gets smaller and smaller.
For example, if output = 0.6 then (output) * (1 – output) = 0.24 but if output is 0.95 then (output) * (1 – output) = 0.0475.
As the adjustment factor gets smaller and smaller, the change in weights gets smaller and smaller and training can stall out, so to speak.
But if you use cross-entropy error, the (output) * (1 – output) term goes away (the math is very cool). So, the weight changes don’t get smaller and smaller and so training isn’t s likely to stall out.

Note that this argument assumes you’re doing neural network classification, with either softmax output node activation plus multiclass logloss, or sigmoid output node activation plus binary  logloss.

Reference: https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/


How to use Machine Learning to discover important biomarkers in an unbalanced small data set

If your goal is to identify important features, I would say go for a Decision Tree which inherently calculates importance/separation capability of the features while selecting them for splitting the internal nodes. You can also go for an ensemble of decision trees such as RandomForest which will return feature importance based on their average impurity reduction throughout all its trees.

This article can help you set up a basic experiment.


One option is to print the confusion matrix to see the rate of misclassification by class.

Another option is to read through the misclassified documents to see if a pattern is apparent.


How does Pooling Layer in CNN introduce invariance to other transformations besides translation

This will become clearer if you try to visualize the convolutional filter functioning.

Let's say you have a filter that is extracting horizontal lines from the image. When the filter extracts the dominance of horizontal lines in the image and assign a value in the generated matrix. Now we have the weights of the horizontal lines after the operation of the filter. Among all the horizontal weights, the most significant will be the one that has maximum weights so, we apply pooling (Max-pooling in this case) and discard all the other weights to select only that one having the maximum impact. By doing so, the filter selects only the features having maximum wights and discard the remaining. As the significance of the required object is from the line having maximum weight, the transformation is invariant. This also makes to tensor smaller in size and the process becomes faster too.

Perhaps, you can also refer the CS231 for a better understanding of the explaination.


When training an image classifier, is it best practice to equally distribute the number of images in each category?

Depending on the method that you use, this answer might be different.

First, you split the data between training, validation, and test splits. Forget about the test set. Keep the validation data in its natural distribution of classes.

You have several choices with the training set data -

Balance the classes using down-sampling or up-sampling techniques
Keep imbalanced classes but attach weights to errors in each class (See som references here:
https://stats.stackexchange.com/questions/207977/balancing-classes-for-neural-network-training)


How to choose layer from which to unfreeze image classification model

Regarding #1: As you mentioned, the following amount of fitting you do on a pre-trained (PT) network depends on the relative size and similarity of your data and the data used to train the PT model.

If our dataset is smaller than the PT data, we should unfreeze fewer layers to avoid overfitting.
If our dataset is of a similar size as the PT data, we can unfreeze more layers since overfitting is less of a concern.

Essentially, more frozen layers means less overfitting, since those layers will retain the relatively generic features of the PT network. As such, you can use hyperparameter tuning (and the results of your cross-validation) to optimize this like you would any other hyperparameter (grid search, etc).

Regarding #2: Yes, you certainly can use the PT model to fit and make predictions. But, the more you understand the model's architecture, the better you will be able to tune the model, and ultimately, improve your results.


How to pass features extracted using CNN into RNN?

1 and 2. You are in the right direction, you need to extract the features using a CNN, then instead of predicting the class you want to reshape the last layer of features and feed it directly into the RNN.

A couple of things to pay attention to:

With a rather shallow CNN you aren't taking advantage of the higher-level feature extraction these architectures can offer. If all your images are as simple as the example you've shown it is adequate.
If you are considering a larger CNN, along with the RNN there is a substantial number of parameters to be trained. For this you need lots of data and lots of computational resources (either very strong GPUs or time).
In order for you to get the best of the two, I would suggest incorporating a pre-trained CNN into your model (and just fine tune the latter layers). This  pre-trained model can even be trained in generic images (i.e. ImageNet) and will substantially increase the performance of the CNN without computational cost. You then can train the latter layers of this CNN jointly with the RNN.

3.This is a good example of what you are trying to do. They basically try to recognize text from street photographs among other things with the same methodology you describe.

Similar methodologies can be found in other research domains such as multi-label image classification, sequence labelling, facial expression recognition, etc


What is the difference between topic modeling and clustering?

The purpose of topic modeling methods is to discover the latent themes (topics) assumed to have generated the documents of a corpus. Topic modeling methods are built on the distributional hypothesis, suggesting that similar words occur in similar contexts. To this end, they assume a generative process (a sequence of steps), which is  a set of assumptions that describe how the documents are generated. Given the assumptions of the generative process, inference is done, which results in learning the latent variables of the model. For instance, for Latent Dirichlet Allocation, this is the per-topic document distributions and the per-word document distributions. In this sense, a document can be represented by its per-topic distribution (doc$_1$ = 0.3$\times$Sports + 0.7$\times$Cinema). This later can be seen as a soft clustering approach, i.e., doc$_1$ belongs 30% in cluster Sports and 70% in Cinema. But topic models are not solely clustering methods, as can also been used for understanding, exploring, visualizing a collection.

On the other hand, clustering methods aim at partitioning data into coherent groups. Of course, what is coherent and how the partitioning is performed differs between the various clustering algorithms. The distance between the data instances is central for clustering methods and for this the instances can be represented in various ways: for documents this can be term frequencies (tf), tf-idf, and even with the per-document topic distributions learned by topic models.


Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields

NER

Basically the problem you have is Named Entity Recognition (NER).

This is a classic NLP problem and there are many machine learning approaches to it. Some recurring neural networks can be used, or the go-to method conditional random fields.

There are different techniques, usually in training, to overcome the issues (inconsistent reporting, nuanced terminology, abbreviations, spelling errors) that you mentioned.

Consideration

One important thing you need to consider is accuracy vs recall. You can choose to optimize one or the other, or both.

Accuracy = you try to redact something only if it really is some personal information.

Recall = you try to not miss any personal information.

Let say you have the sentence "Hello, my name is John, and my social security number is 123456789."

If you redact the sentence to: "Hello, -- ---- -- ----, --- -- ----- ------ ------ -- --------." You get really high recall (all personal information is gone)! But your accuracy is low, you redacted more than you should.

All this can be leveraged during prediction time. The probability score from your NER can make you optimize or the other.

Of course, this begs the questions: Why not redact EVERYTHING and so that 100% of personal information is not leaked?


It seems that you are collecting the lemmas in your docs. For that, you need a lemmatizer.
If available for your language, you should use an external lemmatizer. Some packages supporting lemmatization for different languages are StanfordNLP (or its equivalent for Python, Stanza), Spacy or NLTK.
Depending on the language, the approach to get a good lemmatization varies, but many times it involves expressing the language morphological knowledge as rules.
If no lemmatizer or stemmer is available in the language you are working with, another approach would be to use unsupervised approaches to segment words into morphemes, and use your linguistic knowledge of Georgian to devise some heuristic rules to identify the stem among them.
This kind of approach consists of a model trained to identify morphemes without any labels (i.e. unsupervisedly). The most relevant Python package for this is Morfessor. Also, there is a Python package called Polyglot that offers pre-trained Morfessor models in different languages, so you should check if yours is included.


The most common way to reduce overfitting in neural networks in your case are:

Dropout - Randomly removing connections during training
Data Augmentation - Creating variations in data, such as rotating or flipping images


Without knowing what algorithm was used to generate the check digit, it's not possible to say whether machine learning will be effective at reverse-engineering that data.

As an easy case, given sufficient data, it's likely that a recurrent neural network could learn the Luhn algorithm.  It would likely be a particularly inefficient and suboptimal way to do it, and you might need an awful lot of training data, but I expect it'd probably work.

As a hard case, if the check digit is generated using a cryptographically secure hash function and a secret key, then no machine learning algorithm is ever going to reverse-engineer that.

So it's simply not possible to say, in the abstract.  The answer depends on the check digit algorithm that is being used.

In general, machine learning isn't a silver bullet.  It isn't magic fairy dust that can be used to solve all problems.  My sense is that machine learning is probably not going to be particularly well-suited to this kind of situation.  It might be more effective to start by trying some common check digit algorithms (e.g., use linear algebra to check whether it is a linear checksum, and so on).


You probably did:

install.packages(ggplot)

This is a common mistake: when installing a package you need quotes, otherwise R interprets the name of the package as a string and since the string isn't defined you get an "object not found" error. The correct version is:

install.packages("ggplot")

(with single or double quotes, it doesn't matter)

Note: I would normally use the ggplot2 package, but maybe ggplot redirects to the same package:

install.packages("ggplot2")


What loss function to use for imbalanced classes (using PyTorch)?

What kind of loss function would I use here?

Cross-entropy is the go-to loss function for classification tasks, either balanced or imbalanced. It is the first choice when no preference is built from domain knowledge yet.

  This would need to be weighted I suppose? How does that work in practice?

Yes. Weight of class $c$ is the size of largest class divided by the size of class $c$.

For example, If class 1 has 900, class 2 has 15000, and class 3 has 800 samples, then their weights would be 16.67, 1.0, and 18.75 respectively.

You can also use the smallest class as nominator, which gives 0.889, 0.053, and 1.0 respectively. This is only a re-scaling, the relative weights are the same.

  Is this the right approach to begin with or are there other / better
  methods I could use?

Yes, this is the right approach.

EDIT:

Thanks to @Muppet, we can also use class over-sampling, which is equivalent to using class weights. This is accomplished by WeightedRandomSampler in PyTorch, using the same aforementioned weights.


What is the difference between explainable and interpretable machine learning?

I found this article by Cynthia Rudin which goes a bit more into the difference between the two terms that is in line with your source from O'Rourke.
At the core it is about the time and mechanism of the explanation:
A priori (Interpretable) vs. A posterio (Explainable)
I found this quote to be very helpful and inline with my own thoughts (emphasis mine):

Rather than trying to create models that are inherently interpretable, there has been a recent explosion of work on ‘explainable ML’, where a second (post hoc) model is created to explain the first black box model. This is problematic. Explanations are often not reliable, and can be misleading, as we discuss below. If we instead use models that are inherently interpretable, they provide their own explanations, which are faithful to what the model actually computes.

In short an interpretable model is able to output humanly understandable summaries of its calculation that allow us understand how it came to specific conclusions. Due to that a human would be able to actually create a specific desired outcome by selecting specific inputs.
A "merely" explainable model however does not deliver this input and we need a second model or mode of inspection to create a "Hypothesis about its mechanism" that will help explain the results but not allows to rebuild results by hand deterministically.


Is it better to use separately regularization methods for Neural Networks (L2/L1 & Dropout)

I am unsure there will be a formal way to show which is best in which situations as it depends on many factors like your dataset, architecture of the your ANN  - simply trying out different combinations is likely best.
It is worth noting that Dropout* is actually doing more than just regularization, it makes the model more robust,allowing it to try different nodes for prediction.
As for L1/L2 it just reduces overfitting by penalizing the higher weights.


What is better opensource alternative for identifying small face other than yolo?

Here is a model tailored for your problem. And here is the research paper for the model.


What is the difference between lightgbm.LGBMModel and lightgbm.LGBMClassifier?

Glancing at the source (available from your link), it appears that LGBMModel is the parent class for LGBMClassifier (and Ranker and Regressor).  You should probably stick with the Classifier; it enforces proper loss functions, adds an array of data classes, translates the model's score into class probabilities and from there into predicted classes, etc.


I would say that both the definitions you gave above refer to variance.

Variance is an index of model complexity. Thus:

high variance -> high complexity -> many degrees of freedom in model -> it overfits training data -> if dataset changes, accuracy degrades a lot

Bias is an index of the systematic error of the model, as it shows the difference between the expected value of various model's predictions for a given target output and the ground truth.


Transfer learning with many small datasets

I ended up doing what I proposed in the question: document(from customer X) -> preprocessing(shared among customers) -> feature extraction (shared among customers) -> classification (specific to customer X). I had a loss for each customer X.
I trained the network with a custom training loop: https://keras.io/guides/writing_a_training_loop_from_scratch/
The training converged, which is what I was most uncertain about. The testing accuracy was a bit better compared to training the model with a single customer.
I still did not manage to find any papers on this topic though.


Different number of features in train vs test when using Label Encoding

To solve the high cardinality problem, follow the recommendation of Erwan. To solve the second issue you are mentioning, fit the OneHotEncoder on the train set and use the transformer on the test set, like this:
from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(handle_unknown = 'ignore')

enc.fit(train)

X_train = enc.transform(train)

X_test = enc.transform(test)


I'm not getting the no of parameters concept in CNN


How can this CNN for the portfolio management problem be implemented in keras?

You can concatenate your tensors in Keras. I have written the model below

from keras.models import Sequential, Model
from keras.layers import Concatenate, Dense, LSTM, Input, concatenate, Flatten
from keras.optimizers import Adagrad

price_history = Input(shape=(11, 50, 3))
feature_maps = Conv2D(2, (1, 3), activation='relu')(price_history)
feature_maps = Conv2D(20, (1, 48), activation='relu')(feature_maps)

# Add the w from the last period
w_last = Input(shape=(11, 1, 1))
feature_maps = concatenate([feature_maps, w_last], axis = -1)
feature_map = Conv2D(1, (1, 1), activation='relu')(feature_maps)
feature_map = Flatten()(feature_map)

# Add the cash bias
cash_bias = Input(shape=(1,))
feature_map = concatenate([feature_map, cash_bias], axis = -1)
w = Activation('softmax')(feature_map)

model = Model(inputs=[price_history, w_last, cash_bias], outputs=w)
print(model.summary())

model.compile(optimizer=ada_grad, loss='binary_crossentropy',
               metrics=['accuracy'])

I am not sure about the loss function, I have not read the paper in detail. But this model describes what they have in the paper.


With this low number of sign ups I doubt you can profit from advanced statistics at this stage. Just chart your numbers in excel along the timeline and judge visually.


Why do I get different coefficients from Logistic regression in Python and SPSS

The LogisticRegression class from sklearn by default uses an L2 penalty as a regularization mechanism, which I assume is not used in the SPSS implementation of the logistic regression. This has been a discussion point before but the main reason for this is that sklearn is focused on prediction and getting the best prediction results (where regularisation is used to prevent overfitting) instead of fitting the model to the data and finding which factors affect the outcome. You can try setting the penalty argument to 'none' to see if that gives you the same results as in SPSS.


Efficient clustering of sparse binary vectors

You can efficiently implement e.g. Euclidean distance or Cosine on sparse data (iterate over nonzero values only!). Then you can use e.g. Hierarchical clustering.

But also consider frequent itemset mining. Usually, on sparse binary data, frequent itemsets are better than clusters.


Asynchronous Hyperparameter Optimization - Dependency between iterations

hyperopt proceeds sequentially, unless you let it use a parallel backend. Is N the max_evals parameter? Yes, you always want to select the hyperparams with the best validation loss. That's what it returns in the end. It may be that it found the best hyperparameters well before the final trial. It does not mean anything is wrong. It learns as it goes the distribution of the loss conditional on hyperparameters, and on purpose explores less-certain parts of the space that are most likely to yield improvement, but may not, especially towards the end of the search. This happens with entirely well-defined loss functions.


Pre trained dataset for Car damage detection

For this task you should use any model pre-trained on the COCO's dataset.
We managed to get quite good results with only 80 images with a pre-trained MaskRCNN.


How to choose validation set for production environment?

If you're simply re-training the XGBoost model periodically in order to account for the changing nature of your data, the best option is to hold out a recent set of data for testing (some variation of your option 3).

As you mention, option 1- that is, training on the entire available set and validating on the most recent 10 percent of the data, is very likely to overfit (and therefore overestimate its performance on future data).

Option 2 has the downfall that you're attempting to predict information that you have post-hoc (after the fact) information about- that is, you know what subsequent observation values were, which is very relevant to a time-series prediction. For example, if you're trying to predict what the value is tomorrow, and you know what the value is for the day after tomorrow already, you can do a much better job of predicting tomorrow. Unfortunately, you can never know this before the day after tomorrow begins.

Therefore, option 3 is the most valuable practice to determine the likelihood of your model to accurately predict future observations- you're using only data that would be available from the point of prediction (in the past relative to the predicted period) and holding that data out of the training set for the model.

The best practice in these circumstances is to hold yourself accountable for "not cheating"- that is, try your best to provide a fair and honest experiment on the prediction of your model on data it hasn't seen and would be available in the context of a future use situation.


Bagging vs Boosting, Bias vs Variance, Depth of trees

why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (very high variance)

Clearly it wouldn't make sense to bag a bunch of shallow trees/weak learners. The average of many bad predictions will still be pretty bad. For many problems decision stumps (a tree with a single split node) will produce results close to random. Combining many random predictions will generally not produce good results.

On the other hand, the depth of the trees in boosting limits the interaction effects between features, e.g. if you have 3 levels, you can only approximate second-order effects. For many ("most") applications low-level interaction effects are the most important ones. Hastie et al. in ESL (pdf) suggest that trees with more than 6 levels rarely show improvements over shallower trees. Selecting trees deeper than necessary will only introduce unnecessary variance into the model!

That should also partly explain the second question. If there are strong and higher-order interaction-effects in the data, deeper trees can perform better. However, trees that are too deep will underperform by increasing variance without additional benefits.


Running multiple times of a model is for model randomness or data randomness?

Usually, we report mean and variance for k-fold crossvalidation and similar techniques. We run the model multiple times in different data, but that can be applied to the second case too as, for example, a Neural Network can be initialized with random weights multiple times for test in the model.

If the paper does not explicitly say that they re-splited the dataset, then is more likely that this variance comes from random initialization. You should read the experimental procedures used and try to determine all sources of variance for papers results.

So it might be either or a combination of both dataset random splits and model random initialization


Clustering cartesian coordinates associated with 1 categorical feature

This isn't a clustering problem. Within each set (like W=3), you simply sort them by x coordinate and then y coordinate too. Find the points +/- 2 pixels in the x dimension, then in the y dimension, and take the points that are in both. This ought to be linear within a sorted list.


Developing KNN with custom distance matrices

I would personally take a look at collaborative filtering as this will take into account the information of the user itself, but also the information of similar users. If let's say, my best friend and I have the same taste in books, knowing his taste can help you guess mine! I'd recommend watching the Andrew Ng lectures on Coursera about recommender systems.

KNN might seem like a natural choice if you don't know about recommender systems (I have been there!) but please, make sure to read about them before you decide to use KNN.


how to draw a negative exponential plot of this situation with python

The model you are looking for is this:

$Y=A e^{-Bx} + c$, I know the implementation in R, because I don't know of a nonlinear estimation in Python

This code in R might work:

R=data.frame(X=c(1,2,3,4,5,6,7,8,9),Y=c(1,2,3,3,3,3,3,3,3)) # Data in which X, Y are your data
model=nls(formula = Y~A*exp(-B*X)+C,data=R)
summary(model)

There is a limitation to take into account, is explained in this link, is summarized in the impossibility for all possible models to exist, the "most inside" model should be linear.

First steps with Non-Linear Regression in R

Singular Gradient Error in nls with correct starting values


When to use Linear Regression and When to use Logistic regression - use cases

Logistic Regression is used when you know that the data is lineraly seperable/classifiable and the outcome is Binary or Dichotomous but it can extended when the dependent has more than 2 categories.

Linear Regression is used to find the relation and based on the relation between them you can predict the outcome, the dependent variable should be numeric.

What kind of usecases are you expecting? give an example so that we can extend it further.


library(quanteda)

mytext <- c(oldtext = "I want this cat")
dtm_old <- dfm(mytext)
dtm_old

newtext <- c(newtext = "You are the dog here")
dtm_new <- dfm(newtext)
dtm_new

dtm_matched <- dfm_match(dtm_new, featnames(dtm_old))
dtm_matched

$V_1$
Document-feature matrix of: 1 document, 4 features (100.0% sparse).
         features
docs      i want this cat
  newtext 0    0    0   0

$V_2$
Document-feature matrix of: 1 document, 4 features (75.0% sparse).
         features
docs      i want this cat
  newtext 1    0    0   0

$V_3$
Document-feature matrix of: 1 document, 4 features (75.0% sparse).
         features
docs      i want this cat
  newtext 0    0    0   3

Of course when using a "one hot" vectorizer, "cat" in $V_3$ would be 1 (instead of the count).


I think the answer(comment) of TwinPenguins provides you a good start. You could improve this by using a Metropolis Monte Carlo approach instead of just trying again and again generating new vectors.
What you could do is:

create a new vector with length similar to yours randomly sampled from the distribution of your interest and compute KLD$_{old}$ (cf. TwinPenguins)
modify this vector and compute KLD$_{new}$.
if (KLD$_{new}$ is better than KLD$_{old}$), the new vector becomes your old vector
else discard the changes
go back to 2, unless you reach a given target value of KLD

(Note that there may be issues with getting trapped in local minima, but search the Monte-Carlo literature and  you'll find a solution...you can always run multiple times, and then take the best result.)


result <- read.csv(“Keough.csv”, header=TRUE, sep=“,”, dec=“,”)


Setting BATCH SIZE when performing multi-class classification with imbalanced dataset

There are two common options:

Stratified sampling within each batch. Regardless of batch size, make sure each group is equally represented. The downside this approach is that it would significantly slow down training.

Train with increased batch size (say 32-256) and over the course of the epochs the random fluctuations will "average" out.


I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If you only have one document, I'm afraid that value is simply 1.

However, if you are looking to get rid of words such as the, as, and it which don't carry much meaning, nltk in Python has some useful tools to remove these "stop words" from your document and might help you.

Here is a helpful example.


can we make a word2vec NN of more than 3 layers using tensorflow?

Technically, any Word2vec is based on an encoder-decoder Neural Network architecture with a hidden layer that learns the word embedding. In theory it is perfectly feasible to implement a "deeper" word2vec model in TensorFlow.

You are going to find some practical problems, though. Word vectors are super long (for a small corpus, we are in the order to thousands). Training a deep Network with, say, 10.000 input nodes is going to be practically infeasible without an appropriate infrastructure.

To overcome this problem, word2vec libraries (such as gensim) recur to negative sampling. The inventors of word2vec (Mikolov et al.) found that the results from training a simple word2vec model can be approximated by a sequence of logistic regressions. You can find more on negative sampling here and on Google.

But if you make the Network deeper, logistic models (and therefore negative sampling) couldn't work anymore, making this task too computationally intensive for normal computers.

In other words: yeah you can create a word2vec Neural Network with more than 3 layers, but the computational power required makes it almost impossible for individual researchers with common hardware.


There can be some other factors that affect this, such as using simulated annealing (in a NN context) or other learning rate schedules. Are you using a specific LR schedule?

A schedule might be that the LR decreases by 50%, every time the validation loss of 5 epochs in a row does not decrease. This will help get closer and closer to a minimum of the loss. However, we know it is possible to get stuck within a local minimum, which may be far from optimal, so we can shake things up by increasing the LR once again, which will essentially throw the algorithm our of tyhe local minima and on its way to a new minima (at least that is the hope). This kind of schedule often produces loss curves like the ones you see.

Another alternative is simple that your batch sizes are quite small, and every once ina  while, you get a batch that consists of example which your model really struggles with, so the loss for that batch (and so the epoch) would spike in comparison to other epochs.

A final idea - thinking more about your data - if it is time-series e.g. of stock prices or weather - there could be a regime change/shift. Meaning that the underlying function or system suddenly switches to a new pattern. Something like this could throw your model off the scent for a while, and so produce bumps in the loss curve.

A small point on terminology: LSTM and GRU architectures are themselves RNNs. A recurrent network is one in which connections do not only move forward in a network, but can also go sidewards across a layer or indeed backwards. So it is a more general term, whereas LSTM/GRU layers are specific examples of RNNs.

If you can say a little more about your three model architectures, perhaps it might be clearer which names to use - and maybe even better understand these loss curves  :-)


Advice on where to continue in the field of data engineering and machine learning

Dataquest is a great place to start. It will help you bridge the gap from just learning to actually coding as a data analyst, data engineer, or data scientist. Unlike most online courses, there are no videos, and they provide an interactive coding and learning environment that makes it very easy to learn the practical techniques you'll actually need to work on real-world data and problems.
The best part about Dataquest is their guided projects that are part of every topic you learn. If you want to eventually get a job as a data analyst, scientist, or engineer, you are going to need a portfolio of projects to show what you know, and Dataquest helps guide you through the entire process.
fast.ai is another great place to learn about machine learning, and deep learning in particular, all for free. They teach using a top-down approach that will get you working on your own projects and data in no time.


How to plot two columns of single DataFrame on Y axis

Feeding your column names into the y values argument as a list works for me like so:
total_year[-15:].plot(x='year', y=['action', 'comedy'], figsize=(10,5), grid=True)

Using something like the answer at this link is better and gives you way more control over the labels and whatnot:
adding lines with plt.plot()


Modelling in python and scoring in MATLAB?

"Data.p" #yours pickle file
[data]=pickle.load(open("Data.p","rb"))


How to calculate the evaluation metrics (i.e., F1 score) in leave one subject out cv when a subject belongs to single class only

You should not use summary statistics if you only have 10 data points. With that few datapoints, the summary statistics estimates will not be robust.


Why do we have to remove most common words for text analysis?

The simplest way to explain why it may be advantageous to remove the most common words is that they don't give us much information. In your case of classifying racist tweets, words like "and", "a", "the", etc. don't help the classifier and may act as noise which negatively impacts performance.

I would not say that the removal of the n most popular words will guarantee that the model will be more accurate, but it is a parameter you can explore. Outside of complete removal of the most common words, you may want to look into techniques like subsampling.


PCA, why variance of eigen values is measure of its utility?

What is important is that the covariance matrix eigenvalues represent the variance explained by each component of the PCA.

PCA can be considered useful when it allows to reduce the dimension without removing too much variance from the dataset; that is, when there are a lot of low eigenvalues. As the sum of the eigenvalues is fixed because it is the total variance of the dataset (this is a property of the eigenvalues decomposition of a matrix: the trace of a matrix is equal to the sum of its eigenvalues), low eigenvalues for some eigenvectors means that there are other high eigenvalues to compensate; that is a high variance in the eigenvalues.

In a more mathematical way, if we denote by $\sigma_{tot}^2$ the total variance over the dataset (not the variance of eigenvalues), we know that the mean of eigenvalues will be $\frac{\sigma_{tot}^2}{n}$. As all eigenvalues are non-negative (property of the covariance matrix), and based on the trace conservation property, we deduce that all eigenvalues are lower or equal to $\sigma_{tot}^2$. Then, the convexity of the square function implies that maximum variance among eigenvalues is reached when one of the eigenvalues is equal to $\sigma_{tot}^2$, and all other are null. Hence, eigenvalues maximum variance is reached when the first component wears the total dataset variance, which is obviously the optimal PCA case (all features reduced to a single dimension).

Oppositely, the case with minimal variance of eigenvalues is the one where all eigenvalue are equal, meaning that there is no particular "principal" component (the projection of the dataset onto any direction gives the exact same variance). If I'm not mistaken in the math, this is only possible when the covariance matrix is a multiple of the identity matrix, meaning the dataset is already completely uncorrelated. In that case, PCA is simply useless.


class_weight on sklearn's DecisionTreeClassifier

From sklearn's documentation,

  The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

It puts bigger misclassification weights on minority classes than majority classes. This method has nothing to do with resampling; it modifies the misclassification cost matrix instead.

Changing the misclassification cost of each class is a different approach from resampling approaches. In my opinion, it won't give exactly the same result as oversampling the minority class. Having said that, these two approaches are both helpful to dealing with imbalanced (or unbalanced) data classification


Are support vector machines and logistic regression equivalent if data is linearly separable?

Very closely, but not exactly. This post by Georgios Drakos goes through the question mathematically and visually. Here is an image from the post:

which compares soft-margin SVM (SVM) and Logistic Regression (LR).


What is the difference between residual sum of squares and ordinary least squares?

Here is a definition from Wikipedia:

  In statistics, the residual sum of squares (RSS) is the sum of the
  squares of residuals. It is a measure of the discrepancy between the
  data and an estimation model; Ordinary least squares (OLS) is a method
  for estimating the unknown parameters in a linear regression model,
  with the goal of minimizing the differences between the observed
  responses in some arbitrary dataset and the responses predicted by the
  linear approximation of the data.

So RSS is a measure of how good the model approximates the data while OLS is a method of constructing a good model.


According to the Wikipedia article, it doesn't look like there is a definitive research article that introduced the k-means clustering algorithm.
Hugo Steinhaus had the original idea in 1956. The standard algorithm was proposed by Stuart Lloyd at Bell Labs in 1957, but apparently that wasn't published until 1982.


When visualizing data that has <1 or <5 ppm how do you display this?

Is it a univariate plot?  If so, just bin the data and toss it into the lowest bin.


Vectorized method to find matching values between two columns

you could use set to get the intersection (it has a complexity logarithmic in the size of the sets a and b)

 a = set(df['A'])
 b = set(df['B'])
 a.intersection(b)


Issues related to the code for ROI pooling from the feature map

For the implementation of ROI pooling kindly refer to the following article...

https://medium.com/xplore-ai/implementing-attention-in-tensorflow-keras-using-roi-pooling-992508b6592b

https://deepsense.ai/region-of-interest-pooling-explained/


Problem to classify multilabel dataset while using random forest algorithm

From the documentation:

  Warning: At present, no metric in sklearn.metrics supports the multioutput-multiclass classification task.

See here and also this issue. Multiclass-multioutput doesn't seem reliably implemented at this point.


What could be the value of having a business analyst in a data science team?

I believe the integration of Business Analyst(s) into data science team makes great sense.

For a number of reasons:

there will be a lot of interaction between business proper and the data science team. BAs are used to acting as an interface layer between IT and business functions. This is much harder than it seems, as IT and (for example) Sales & Marketing people are by nature extremely different. It takes a lot of patience and experience to get them both on the same page. Hint: when talking to Sales or Accounting do not start with an icebreak about the virtues of Open Source!
BAs usually have better domain knowledge than DSs. They can be of great help in initial phases of the project as they tend to have a more natural "feeling" for the business (e.g. which outcomes are possible, and which not, are we forecasting a rare event and some under/over sampling is in order, what sort of results are no news and what kind are so surprising that they should be brought to board attention immediately etc.). Data Science types are good at finding the correct answers once a problem is formulated, but asking the right question is arguably more important. And it is a task a BA can help with.
Business Analysis is around for longer, so there is a deep and broad pool of experienced people around. Data Science is young but sexy and looks nice in a resume, so it will be easy to persuade a smart guy/gal with strong understanding of the underlying business and good relationship with the key players in the company to become a project member. It is a win win situation.


Sklearn StratifiedKFold code explanation

A KFold split will take the data and split it however many times you designate.   StratifiedKFold is used in order to ensure that your training and validation datasets each contain the same percentage of classes (see sklearn documentation for more). The function StratifiedKFold takes two arguments, the array of labels (for binary classification this would be an array of 1's and 0's) and the number of folds.  They have designated the number of folds as 1./eval_size where eval_size = 0.10.  So this is a 10-fold validation.

train_indices, valid_indices = next(iter(kf))

This line derives the indices in order to split the data into the train/validation data sets.

Now that we have the indices, we use those indices to actually split the data.

It is interesting that they wrote next(iter(kf)) and then input the indices into the dataset when they could use sklearn.cross_validation.train_test_split.  train_test_split is just a wrapper for next(iter(kf)), but it is more readable and it's already a function in sklearn.

Hope this helps!


Why this calculation of weight vector in linear regression is only for small dataset?

$X$ is your design matrix (data), so can get large, both in number of data points and number of predictors. The issues are indeed:

Forming $XX^T$ may be computationally expensive
Computing it explicitly may be inaccurate due to floating-point roundoff, as you're adding lots of products
Inverting a matrix can be expensive here if the number of predictors is large as the Gramian is large
With lots of predictors and a big Gramian, it's more likely to be ill-conditioned and suffer from numerical instability


An abstract idea for the performance diffs between SLP and MLP

More layers/more neurons does not necessarily mean you will get better performance. If your data is too simple or the number of observations is not that high, then adding more parameters (more layers/neurons) may result in overfitting the data. During training, the network will try to represent the training data as closely as possible. When there are a good number of units in the network, then the representation that is learned by the network will mainly model the general trend of the data faithfully. But, if there are too many neurons, then it is possible that some of the neurons will simply model noise in the training data which does not generalise to unseen data, resulting in worse performance.

Also if you are using a saturating nonlinearity as an activation function (i.e. sigmoids or tanh) then adding too many layers may result in vanishing gradients which will cause your network to train very slowly, or perhaps not at all.

Designing neural networks that work well is not very easy, and the best way to get an intuition for what will work is to experiment. One thing to try is to evaluate a range of sizes for your hidden layer(s). If you are using sigmoid or tanh as your activation function, I suggest you try using rectified linear units as well.


K-means clustering to separate temperature vertical profiles

here is my 2 cents: k-means is an unsupervised method and therefore it's best suited if you wanted to explore the optimal number of clusters the temperature can be split, this could be useful to see that maybe there is more regions other than east or west were the temperature is different like north and south and you could use a silhouette score or elbow curve to find the optimal number of clusters.
But, what you describe here is a supervised classification problem where you already have the labels(east and west) and want to separate the temperature accordingly, depending on the type of data I would recommend a classification algorithm such as k-nearest neighbors
you could use k-means with a fixed number of clusters, in this case 2, but it would not be as effective as a supervised algorithm


How do I get similarity with autoencoders

You can calculate the cosine similarity between two encoded vectors you would like to compare. The cosine similarity between two vectors is defined as follows:


How to retrieve Twitter username with UUID

It is not possible with the current version of the dataset, the User.txt ids are hashed. But, the pre-processed features are in UserFeature.mat which you can load with something like

import scipy.io
mat = scipy.io.loadmat('UserFeature.mat')

mat['X'][0] will then be the feature vector for the first user id in the text file.

There are no descriptions of the feature names either (or at least I didn't find any)


Entropy can be calculated using formula:-

$Entropy = -p\log_2 p − q\log_2 q$

Here $p$ and $q$ are probability of success and failure respectively in that node.
Entropy is also used with categorical target variable. It chooses the split which has
lowest entropy compared to parent node and other splits. The lesser the entropy,
the better it is.

Steps to calculate entropy for a split:

Calculate entropy of parent node
Calculate entropy of each individual node of split and calculate weighted
average of all sub-nodes available in split.


Naives Bayes Text Classifier Confidence Score

First there is a simple theoretical reason why relying on the probability provided by a NB model is not a good idea: what NB predicts is the posterior probability, i.e. the conditional probability $p(C_i|d)$ for every class $C_i$ for a given document $d$. The class $C$ which is predicted is just the one which obtains the highest probability $p(C_i|d)$. The sum of all the posterior probabilities for a particular $d$ is 1 since it's a conditional, and this means that a class is always predicted relatively to the other classes: a high probability for $C$ doesn't mean that the document is very likely to be $C$ in general, it only means that it is much more likely to be $C$ than any of the other classes. If it actually belongs to another class unknown to the model, there's no way the model can represent this.
There is also a practical reason why this is not a great idea: experimentally it is known that NB tends to systematically assign extreme probabilities (and to overfit easily but that's not the point). It does this whether or not the class is correct, and this can easily be checked with your data: if you plot the predicted probability for every instance with a different colour for correct/incorrect prediction, you're likely to see that even the incorrect instances were predicted with a very high probability. It's tempting to interpret the predicted probability as a confidence measure, but it's usually a mistake.
But there is hope: the proper way to deal with potential unknown "classes" (these are not classes technically) is not with regular classification methods, which always rely on a "closed world" assumption (only the classes seen during training exist). There are also open-set classification methods: these are much less common but they are not restricted to the known classes. I think the most common method is probably one-class classification, in particular with one-class SVM.


Why is the code provided in this book mostly commented out?

Welcome to the site!
It looks to me that the code you reference is just showing some example usage for the functions defined above it. There are several reasons someone might do this--sometimes it's so auto-documentation generating code (e.g., doxygen) can pick up the example code for downstream formatting, other times it's so that the reader can identify the code they might need to modify for their own use case. I suspect the latter here. If you're interested in trying to just run the code as written, you could copy it into a text editor like TextMate or VSCode and do a vertical select (usually shift+option+click/drag) to highlight and delete the comment characters!


How do I deal with missing values in a data set?

If the features are categorical, just fill the NAs with "Missing" as a new category. If they're continuous there are a number of approaches you could try. As a starting point, you could simply fill them with the mean value (or median if you have outliers significantly skewing things) and add a new binary feature that flags the value as being missing in the original feature. So...

| Feature A |
| --------- |
|    14     |
|    nan    |
|    23     |

becomes

| Feature A | Feature A Missing |
| --------- | ----------------- |
|    14     |          0        |
|   18.5    |          1        |
|    23     |          0        |

I've never found a hard and fast rule for the "best" way of doing things; I suggest trying the above and cross-validating your model to make sure its performance is within acceptable limits.

Discretizing continuous variables into High/Medium/Low/Missing may also be helpful; again, try it and cross-validate to see if it improves performance. You should try it both with and without the original continuous variables too.


Clarification on K Means Clustering Algorithm and its Analysis

K-means is an iterative algorithm. So what you have in the picture may be one of the early iterations and therefore the sum of squared distances (between points and clusters) has not been minimised yet. In further iterations, the circled point would indeed be assigned to cluster centroid C1.

Now, having said that, and this is purely speculation, but if the diagram in your picture is not simply 2 dimensional as we interpret it to be, but is actually a 2-d projection from a higher dimensional feature space, then indeed the circled point may be closer to centroid C2 in that higher dimensional space (although it doesn’t look like it in 2-d).


How to interpret agglomerative coefficient agnes() function?

Not sure if you are facing a concrete problem. Anyway, here it goes a definition of the agglomerative coefficient, from Finding Groups in Data: An Introduction to Cluster Analysis:

  Generally speaking, the AC describes the strength of the clustering
  structure that has been obtained by group average linkage. However,
  the AC tends to become larger when n increases, so it should not be
  used to compare data sets of very different sizes.

Also, if you are familiar with the silhouette,

  The AC can also be compared to the silhouette coefficient (SC)

First sight, the coefficient you get points to a pretty reasonable cluster structure in your data, since it is closed to 1: the coefficient takes values from 0 to 1, and it is actually the mean of the normalised lengths at which the clusters are formed. That is, the lengths you see when you look at your dendogram.

EDIT after comments: indeed it can be misleading that the text in the book I referenced ("Finding Groups in Data...") refers in my copy-paste to the group average linkage. However, the coefficient refers to the banner (which is like a kind of simpler or binary dendogram) and just validates the structure that the clustering produces, regardless of the method.

This is another reference to the agglomerative coefficient with no reference to the method used (although of course always in the context of hierarchical clustering).

EDIT 2 after comments: I explain below my understanding on how the coefficient is calculated, always using the same reference.

The formula for the coefficient is

$$AC = \frac{1}{n} \sum_{i} l(i)$$

i.e. the mean of the l(i), calculated as follows:

  For each object i, we look at the line containing its label and
  measure its length I(i) on the 0-1 scale above or below the banner.

Although I could not make the plot with 0-1 scale, it is just a normalization of the lengths.

The banner is a binary representation of the dendogram, where:

in the left side, when all objects are separated in the initial stage of the agglomerative algorithm, the banner is just blank
as soon as 2 objects get together in the same cluster, they appear in the banner
the l(i) are marked by the points in the x axis in which the objects get together, and are the lengths of the horizontal bars of the plots

Always clearer with an example, actually taken to a similar one in the same book.
Let's imagine 5 objects in a space of 4 dimensions; 4 of the objects very closed together, and one of them far away.

df <- data.frame("c1"=c(2, 2, 2, 2, 10), "c2"=c(2, 2, 2, 2, 10),
                 "c3"=c(2, 2, 2, 2, 10), "c4"=c(2, 2, 2, 2, 10))

Let's do the clustering,

library(cluster)
r <- agnes(df, method="ward")

The bannerplot actually looks a bit ugly:

but anyway, in there the 4 objects that are at the same position are joined at the beginning in a cluster, and the 5th is at distance 20. Personally, it is much easier to look at the dendogram:

however the information is pretty much the same.

Now the AC calculation. The lengths of the bars are 1 (normalized) for the first 4 objects, and 0 for the last one, thus the AC is 0.8:

> r$ac
[1] 0.8

In the dendogram, one should imagine the bars rotated 90 degrees, and they go from the root of the tree to the point in which the groups are formed.


Assign new point to a class using spectral clustering

I think you do not, since I was able to find several papers that proposed algorithms for the same:

Incremental spectral clustering by efficiently updating the eigen-system
Incremental kernel spectral clustering for online learning of non-stationary data
A Fast Incremental Spectral Clustering for Large Data Sets

Such algorithms are called sequential, incremental, streaming, or online. Armed with this knowledge, you can find more papers on your own.


How to calculate change in Loss Function w.r.t Weights for a fully connected NN

I doubt there is any ready-to-use facility but you can achieve this very easily with some custom code.
You can do -

Train your model(to the extent you want)
Get output using predict method on a dataset
Calculate the Loss using Loss function, y_true and last step output
Get the weight of every layer using Keras get_weights() API
Update weight as per your experiment using Keras set_weights()  API
Calculate the new Loss using the same process as mentioned in step#3

Keras weights API -
You can get weights as a simple array and can update accordingly.
May traverse in the loop on Layers Or access directly using array index.
Some code snippets(See reference links for detail)
Layer.get_weights()
Layer.set_weights(weights)

for layer in model.layers:
    print(layer .get_weights())

weights = model.layers[1].get_weights()[0]
biases = model.layers[1].get_weights()[1]

weights = [np.random.rand(*w.shape) for w in model.get_weights()]
# update
model.set_weights(weights)

References
Keras.IO
Stackoverflow
Stackoverflow
Stackoverflow


How do we know a neural network test accuracy is good enough when results vary with different runs?

So, the question is about how to report test accuracy, etc, when you see variation over executions.
As @Nikos M. has eluded to, you typically train and test the model at least 3 times and then report the average test evaluation metrics (along with the standard deviation to show the level of variation). This is why you are seeing 'discrete' values reported in papers, this is because they are averaged over x train/test runs.


Question and Answer Chatbot for Customer Support

This was intended as a comment.

I think this question is not receiving enough attention because it is too difficult to answer without trying several approaches first.

Your third idea is a nice one, but LSA is most likely not going to be able to help you choose clusters with the necessary granularity. For example, you may get a cluster of "broken things" but not to the specificity that is required to be helpful to the customer. A similar approach would be using Hierarchical Dirichlet Processes, but again, you may not get the clusters that you want.

I wouldn't be so eager to try your fourth option because it requires a lot of work and I'm not sure you're going to get great results unless your customer support deals with very well-defined questions and in that case, you're going to have to build your own ontology from scratch.

In your situation, I would be curious if I can get information about the inner workings of Watson (obviously, the purpose is only to have a general idea of some steps that are sensible enough to try.) Here is a descriptive image that could be helpful from Wikipedia:

The main idea is to segment questions into its components (using a regular chunking), then come up with several possibilities (hypothesis generation) of what the question might refer to (maybe this step can be implemented as a clustering). The third stage is rating the previous hypothesis based on available information. Finally, you choose the more likely hypothesis. There is a feedback involved in this process using sample questions and their answers. It would be interesting to see if the rating stage can be modeled in terms of a Bayesian or an incremental approach to allow for a feedback.


How to reapply rescaling when you want to predict your data

Adding to jrouquie's answer, the formula would be:

For each column,

rescaled_feature = (input_feature - shift)  * inv_scale


How do I choose a discount factor in Markov Decision Problems?

This is the typical value function of Reinforcement Learning. The discount factor evaluates the importance of the accumulated future events in your current value. The smaller the number, the less important are the future events in the current action.

Usually this number is selected heuristically. I usually select 0.9. If I don't want any discount then I would select 1.


How to design n-dimensional feature descriptor similar as the input image?

The problem was with the tf.reshape. Instead I replaced it with pd_input = Reshape((2, 2, 256))(p_encoder). Finally I got the desired output which is same as input (265, 256, 3).


Dimensionality reduction without select components

It would be helpful to know a bit better what you're trying to achieve and why the selection of a specific number of eigenvalues bothers you.
From the generic information you gave it seems you're aiming at training a model on a compressed/dense representation of several features, in which case I would suggest to train an autoencoder (or something similar), on top of which you could then train whatever classifier you need.
Otherwise if the problem relies only on the amount of features you have you could try with some feature selection strategies


Appending DataFrames to lists in a dictionary - why does it seem like the list is being referenced by each new DataFrame?

The problem is when you create the dictionary models using models = dict.fromkeys(target_labels, []), you actaully only created one empty list, and all keys point to that list. Everything you append goes into that same list.

For example,

models = dict.fromkeys('abcd', [])
print(models)
print(models['a'] is models['b'])
models['a'].append(3)
print(models)

will return

{'d': [], 'a': [], 'b': [], 'c': []}
True
{'d': [3], 'a': [3], 'b': [3], 'c': [3]}

Notice that models['a'] is models['b'] is True.

You can create the dictionary instead using

models = dict([(key, []) for key in 'abcd'])
print(models)
print(models['a'] is models['b'])
models['a'].append(3)
print(models)

Now it returns

{'d': [], 'a': [], 'b': [], 'c': []}
False
{'d': [], 'a': [3], 'b': [], 'c': []}

Notice that models['a'] is not models['b'].


what is the best to do with highly correlated features?

No I don't think this is good: this means that you replace both variables with only an estimated value of C2 based on C1 (or of C1 based on C2, it's not totally clear to me).
Assuming you really want to avoid having two highly correlated features, you could instead:

just arbitrarily pick one of them (which is similar to what you're doing but "cleaner")
simply use the average of the two values (which might be similar to what you intended to do?)

Over a whole set of features you could use more advanced methods such as feature extraction.
But it's important to make sure you actually need to do this: correlated features are a problem for some ML methods, but not all of them. For example having correlated features with Decision Trees wouldn't cause any problem. In any case it's worth experimenting with/without the feature in order to observe what actually happens with your data.


How to remove rows from a data frame that are identical to other df?

You can try this:

cond = df1['Email'].isin(df2['Email'])
df1.drop(df1[cond].index, inplace = True)

>>df1
    First   Last    Email
2   Joe     Max     email3@email.com
3   Will    Bill    email4@email.com


how to get the number of columns that have the same value between 2 rows

If your records are in form of a pandas dataframe, then you can use df.columns.duplicated() to know the columns which are duplicated.


Muti-Output Decision tree with classification and regression in output

I check and found this in scikit-learn  documentation and I'm sure that is the answer I was looking for!

  Multioutput classification support can be added to any classifier with MultiOutputClassifier. This strategy consists of fitting one classifier per target. This allows multiple target variable classifications. The purpose of this class is to extend estimators to be able to estimate a series of target functions (f1,f2,f3...,fn) that are trained on a single X predictor matrix to predict a series of reponses (y1,y2,y3...,yn).


There are pseudo-codes in this paper, Section 3.1 shows an example of incremental mining by this algorithm. However I can't find any projects or working source codes with this algorithm implemented.


Should one hot vectors be scaled with numerical attributes

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values if you are doing it for any reason to prepare other columns.

The effect of doing so will depend on the model class, and type of normalisation you apply, but I have noticed some (small) improvements when scaling to mean 0, std 1 for one-hot-encoded categorical data, when training neural networks.

It may make a difference too for model classes based on distance metrics.

Unfortunately, like most of these kind of choices, often you have to try both approaches and take the one with the best metric.


RNN LSTM input conflict due to generator

get_embedded_line(row[1]) has to return a matrix with size (Batch size, 255, 100) but it is yeilding (100, 1) matrix. please check the implementation of "get_embedded_line" function.


OK, first regarding the real time part.

Machine learning (particularly supervised learning) has two parts

Training
Predicting

Training is a complex and slooooow process, you need to figure out a good model, then train it with the correct data... not simple and computationally expensive.

Predicting however is MUCH simpler, as normally consist (in the world of neural networks) of a simple forward pass in the neural network, and a forward pass is, at its core, a bunch of matrix multiplications. So once a model has been built and train, predicting something using the existing model is quite quick, also one can retrain the model with new data too.

Now, regarding with the text searches and how to deal with them. There is a very cool idea called word embeddings, it simply consist on transforming words into vectors. Imagine a 2d grid (like the x,y coordinates we studied in school), now imagine that the vertical axis (y) denotes how powerful something is, the hifher the y value, the more powerful it is. And now imagine that the x axis denotes the gender of something. (lets assume negative x means male and positive x means female).

And now thing of the word : King. King is male (x=Negative value) and powerful (y=Positive value). With those two coordinates, and starting from position 0,0 you could have a vector.

Now think of Queen, in this case the coordinates will be x=Positive value and y=Positive value, now you have another vector.

And now think of the word Spartacus (he was a slave in ancient rome), so that word will have x=Negative and y=Negative.

Think of this, suddenly not only you can represent the words as numbers, you can also represent them with vectors, which allows you to possibly figure out that King and Emperor are both SIMILAR words.


Good observation, and yes, they are in fact equivalent ways of computing the entropy of a bernoulli random variable.

To begin, you have to notice that in the openai code, we do not have the value of $ p $ passed to the function, instead we have $ logit(p) $ which is defined to be:

$ logit(p) = log\frac{p}{1-p} = log(p) - log(1-p)$

Also, two formula to keep in mind. The first is the sigmoid function which is:

$ sigmoid(x) = \frac{1}{1+e^{-x}} $

And the second is the sigmoid of the logit which is:

$ sigmoid(logit(p)) = \frac{1}{1+\frac{1-p}{p}} = p$

(i.e. the inverse of the logit function is the sigmoid function)

Now we are ready to go from the first equation (used in the code) to the second equation (the generic equation to compute the bernoulli entropy)

$ entropy = (1 - sigmoid(logit(p)))*logit(p) - log(sigmoid(logit(p))) $

$ = (1 - p)*logit(p) - log(p) = (1 - p)*(log(p)-log(1-p)) - log(p) $

$ = (1 - p)*log(p) - (1-p)*log(1-p) - log(p) $

$ = -p*log(p) - (1-p)*log(1-p) = entropy $

Thus the formula used in the code is equivalent to the second formula you provided.