How much data to use for feature selection?

Have you considered an approach that wouldn't require you to do feature reduction? For example, if you were to use a neural network, the need for feature reduction is greatly reduced. Now, obviously I haven't seen your data so I don't want to suggest that neural networks should be your only answer. But with that many features I would be more concerned with how I'm actually going to reduce them rather than the number of data points that I need to collect.


Is there a standard data science epistemology?

Interesting question.

The way I see it, the justification comes from both mathematics and experimental science.

If you suggest a new algorithm, you should state your assumptions and provide a proof, just like in math. However, you should also demonstrate it on real data, showing that your assumptions are reasonable and that your algorithm can deliver value.

If you suggest a model, you should justify it based on its performance on the observations, just like in experimental sciences. This is true no matter how did you derive the model. However, examining the model logic itself and the way it was derived will help to gain confidence in it and trace where it needs modification.


Keras Model question for Pre trained model extension

Short answer - Yes.

The pretrained Resnet model has trained parameters (weights and biases). When you add extra layers on top of it, the parameters of these layers are randomly initialized. Therefore the final model will give random results.

You need to train the new model after adding extra layers on your new dataset. You don't need to train all the previous layers, just few of them.

For further reading search for - transfer learning. That's what you're trying to do.


The main distinction to be made here is between a parameter and a hyperparameter; once we have this clarified, the rest is easy: grid search is not used for tuning the parameters (only the hyperparameters), which are tuned with with gradient descend.

Now, roughly speaking, a parameter is something that changes during training; in a neural network, the only parameters are the weights and the biases, and they are tuned with gradient descend.

A hyperparameter can be thought as something "structural", e.g. the number of layers, the number of nodes for each layer (notice that these two determine indirectly also the number of parameters, i.e. how many weights and biases there are in our model), i.e. things that do not change during training. Hyperparameters are not confined to the model itself, they are also applicable to the learning algorithm used (e.g. optimization algorithm, learning rate etc). A specific set of hyperparameters defines a family of models, which differ among themselves in the exact values of their parameters; in contrast, a specific parameter set (e.g. weights & biases in a NN) define a unique model.

Having clarified the above, it should be easy to see that a in your example above is a hyperparameter, and not a parameter; as such, it would normally be optimized using grid search.

  To me, the second option is better since it can lead to the optimal point

Not so fast; gradient descend is not applicable everywhere: there are mathematical prerequisites for a function in order to be eligible for optimization using gradient descend, i.e. to be continuous & defferentiable. This is indeed the case for the loss as a function of the weights & biases, but not clear if it is also the case for the loss as a function of your parameter a. And if it isn't, we simply cannot use gradient descend for tuning a.

On the contrary, techniques like grid search do not have such prerequisites, thus they can be used for a much wider range of cases.

  in grid search, how do we pick the range?

As we pick almost everything else in ML & DL: empirically, and with trial & error (there is actually not much theory behind it).

UPDATE (after comment):

  Now assume that the prerequisites are met for a so that we can use gradient descent. Which one is better now? Grid search or gradient descent?

Sorry, I didn't realize that this was a "Superman vs. Batman" question... Well, gradient descend is a disciplined mathematical operation, which is guaranteed to find the global minimum for convex functions (although NN loss functions are not convex), while grid search is actually just an ad hoc, quick-and-dirty procedure which does not guarantee anything (as you have already correctly suspected), so...

Beware though, especially if this is part of any homework or exam: the question, despite its phrasing, is not actually asking you which one is "better" in an abstract & hypothetical context, but which one is better in a very specific context, i.e a neural network model; and if something is not even applicable, it cannot be actually better, right?

In other words, if someone is actually testing your ML & NN knowledge here, they are certainly more interested in the exposition above, rather that which one is "better" in general; and choice (2) would be a huge mistake (after all, assume that the NN loss is a continuous & differentiable function of a is not part of the question)...


Is there wights of voice or audio for VGG or Inception?

As far as I know, VVGish is the VGG adapted to audio processing. I can remember using it with mfcc, not Amp-Time input tho.


The Policy Gradient theorem states that the gradient of the expected reward is equal to the expectation of the log probability of the current policy multiplied by the reward. Notice that to compute the integral of the expectation we can use Monte Carlo method. For this you would need sampled trajectories. Have you looked at a derivation? The whole theorem comes from mathematical derivations. Check in that page the Assumptions and the Likelihood Ratios (REINFORCE) sections. Hope it helps!


Difference between impurity and misclassificaton

Is Gini index just a fancy name for misclassification?

No.

Note that Gini index definition doesn't involve predicted values, and also it involves some probabilities, which are not dependent on classifier.

Also in context of decision trees, Gini impurity corresponds to each region, and is not a single value, such as missclassification rate (technically you could also count missclassification rate per region, but then you'd also ).

See this notebook for a concrete example.


There is an answer on this question: https://stats.stackexchange.com/questions/259290/theoretical-minimum-number-of-support-vectors

Also, follow the links there to see more...


What does Logits in machine learning mean?

Logits interpreted to be the unnormalised (or not-yet normalised) predictions (or outputs) of a model. These can give results, but we don't normally stop with logits, because interpreting their raw values is not easy.

Have a look at their definition to help understand how logits are produced.

Let me explain with an example:

We want to train a model that learns how to classify cats and dogs, using photos that each contain either one cat or one dog. You build a model give it some of the data you have to approximate a mapping between images and predictions. You then give the model some of the unseen photos in order to test its predictive accuracy on new data. As we have a classification problem (we are trying to put each photo into one of two classes), the model will give us two scores for each input image. A score for how likely it believes the image contains a cat, and then a score for its belief that the image contains a dog.

Perhaps for the first new image, you get logit values out of 16.917 for a cat and then 0.772 for a dog. Higher means better, or ('more likely'), so you'd say that a cat is the answer. The correct answer is a cat, so the model worked!

For the second image, the model may say the logit values are 1.004 for a cat and 0.709 for a dog. So once again, our model says we the image contains a cat. The correct answer is once again a cat, so the model worked again!

Now we want to compare the two result. One way to do this is to normalise the scores. That is, we normalise the logits! Doing this we gain some insight into the confidence of our model.

Let's using the softmax, where all results sum to 1 and so allow us to think of them as probabilities:

$$\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}} \hspace{20mm}   for \hspace{5mm} j = 1, …, K.$$

For the first test image, we get

$$prob(cat) = \frac{exp(16.917)}{exp(16.917) + exp(0.772)} = 0.9999$$
$$prob(dog) = \frac{exp(0.772)}{exp(16.917) + exp(0.772)} = 0.0001$$

If we do the same for the second image, we get the results:

$$prob(cat) = \frac{exp(1.004)}{exp(1.004) + exp(0.709)} = 0.5732$$
$$prob(dog) = \frac{exp(0.709)}{exp(1.004) + exp(0.709)} = 0.4268$$

The model was not really sure about the second image, as it was very close to 50-50 - a guess!

The last part of the quote from your question likely refers to a neural network as the model. The layers of a neural network commonly take input data, multiply that by some parameters (weights) that we want to learn, then apply a non-linearity function, which provides the model with the power to learn non-linear relationships. Without this non-linearity, a neural network would simply be a list of linear operations, performed on some input data, which means it would only be able to learn linear relationships. This would be a massive constraint, meaning the model could always be reduced to a basic linear model.
That being said, it is not considered helpful to apply a non-linearity to the logit outputs of a model, as you are generally going to be cutting out some information, right before a final prediction is made. Have a look for related comments in this thread.


You could use the same reward function that Openai's Inverted Pendulum is using:

$costs=-(\Delta_{2\pi}\theta)^2 - 0.1(\dot{\theta})^2 - 0.001u^2$

where $(\Delta_{2\pi}\theta)$ is the difference between current and desired angular position performed using modulo $2\pi$. The variable $u$ denotes the torque (the action of your RL agent). The optimal is to be as close to zero costs as it gets.

The idea here is that you have a control problem in which you can come up with a quadratic 'energy' or cost function that tells you the cost of performing an action at EVERY single time step. In this paper (p.33 section 5.2) you can find a detailed description.

I have tested RL algorithms in this objective function and I did not encounter any problems for convergence in both MATLAB and Python. If you still have problems let us know what kind of RL approach you implemented and how you encoded the location of the pendulum.

Hope it helps!


Take for example the Euclidean distance L2, defined by:
$$L_2(x,y) = \left(\sum_{i=1}^{d} (x_i-y_i)^2\right)^{1/2}$$
where $d$ is the vector dimension. You can easily add a term $\alpha \in (0,1)$ and put more weight on the first term, for example:
$$L_2(x,y) = \left(\sum_{i=1}^{d} (\alpha x_i-(1-\alpha)y_i)^2\right)^{1/2}$$
That will certainly be non-cummutative and basically it can be interpreted as putting more weight on the 'reference' vector.
Now, if you ask me, what would be a good use of it, I hardly understand why you would use a non-commutative distance or non-symmetric distance. First of all, it will not be a distance in mathematical sense, since symmetry is required by definition, secondly, many algorithms works with symmetric distances and can't be adapted easily.


what is the representation/meaning/implication in real life of $P(\text{+})$ in the wiki Drug testing Example about Bayes' theorem

$P(+)$ represent the probability that the test will produce positive result.

This is computed using law of total probability in the denominator.


I don't get that (or any) error (after correcting your paramators to parameters).

That said, if you only want to use one solver, you might as well pass it directly to the estimator and leave it out of the grid search.


Do I need to encode samples during inference?

Yes the encoders need to be rerun during inference time. You basically need to run the exact same data preprocessing steps prior to training on the unseen observations. Without them your model will not be to recognize unseen data and hence fair poorly.


Measuring Difference Between Two Sets of Likert Values

If you are trying to see whether the survey had made any positive effect on the people, modeling a relationship between before and after with correlation is a fair approach while avoiding some edge cases. Correlation looks whether $X's$ and $Y's$ are linearly correlated or how one variable is influenced by other. But I guess what you might be looking for is just something simpler like average increase in rating after the activity. For example

import numpy as np
from scipy import stats
ratings_before = np.array([1,2,1,3,4,5])
ratings_after = np.array([5,5,5,5,5,5])
print "P.Corr: ",stats.pearsonr(ratings_before, ratings_after)[0]
print "Avg.Increase: ",np.mean(ratings_after-ratings_before)

which gives

Nan
2.333

And assume that your activity went so good that everyone rated you a 5-star. Then correlation turns as $NaN$ as denominator goes to zero as no $Y$ can be explained by $X$.

Hope this helps.


Your definition is correct. For the reference you can compare it with the probabilistic model from Tensorflow "Vector Representations of Words" tutorial:

  $$
\begin{align}
P(w_t | h) &= \text{softmax} (\text{score} (w_t, h)) \\
\\
           &= \frac{\exp \{ \text{score} (w_t, h) \} }
             {\sum_\text{Word w' in Vocab} \exp \{ \text{score} (w', h) \}}
\end{align}
$$

It's the same as yours, but they generalize the condition $h$ as a history.

In case of skip-gram, the target word $w_t$ is any context word ($w_{-t}$ in your notation) and the history is the center word ($w_{t}$ in your notation). You specify $\theta$ explicitly in the condition, but that is usually omitted, because there is only one vocabulary in a given problem.


How do multiple linear neurons together allow for nonlinearity in a neural network?

The phase

"To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

is NOT correct. More precisely, it should be:

"To learn non-linear decision boundaries when classifying the output, we need a non-linear activation function."

To understand why, imagine you have a network with many layers and nodes (multiple neurons in your question). If you don't have a non-linear activation function such as ReLu or sigmoid, your network is just a linear combination of bias and weights. Your network won't be useful for classifying non-linear decision boundary. But if your inputs can be linearly separable, you don't need neutral network...

That's why all neutral networks almost always have a non-linear activation function. ReLu is the most popular, but there are other possibilites. When you pipe up a dozen of non-linear outputs like in neutral network, your network will be able to classify a non-linear decision boundary. The more your have, the better it can perform (but also easier for overfitting).


Why does not log transformation make the data normalized?

Log transformation leads to a normal distribution only for log-normal distributions. Not all distributions are log-normal, meaning they will not become normal after the log transformation.

EDIT:

As you have commented, if you are trying to convert an arbitrary distribution to normal, methods like QuantileTransformer can be used. But note that these transformations make a distribution normal by changing (destroying) some information from the original data.


I've tried this one, and is very straight forward and easy to follow.

Java is very similar to python in some sense so it shouldn't be a problem for you if you have done OO.

If you want to get deeper inside natural language understanding and processing I recommend this book.


How to find similar points to a positive set when you don't have any negative set?

To summarize, you have labeled data in one class (positive) and unlabeled data. You want to find the positive examples in the unlabeled data. The general name for this setting in machine learing is one-class classification, which is a fairly broad field.

A sub-area that is particularly relevant is positive-unlabeled learning, which is the problem of training a classifier when one has just positive and unlabeled data.

Also note that you have all the examples that need to be predicted at training time. Therefore you can use a transductive learning algorithm. Particularly, if you have a notion of which companies are similar, you could construct a graph by connecting similar companies by edges. You could then run a graph propagation algorithm that would assign scores to the unlabeled items.

Finally, here is a similar question where the answer suggests a method of positive-unlabeled learning.


as i understand, you have the wrong backprop gradient implementation. Here you should take into account, that rnn's hidden state h has its previous state in the equation: h[time-1]. This is also must be extracted via chain rule. For more information suggest to refer this post.
It also contains Python rnn implementation.


in my opinion, you have 4 possibilities:

You may treat the pdf directly using tabula
You may convert the pdf to text using pdftotext, then parse text with python
You may use an external tool, to convert your pdf file to excel or CSV, then use required python module to open the excel/CSV file.
You may also convert pdf to an image file, then use any recent OCR software (which reconstruct table automatically from the picture) to get data

This answer comes from:

https://stackoverflow.com/questions/47533875/how-to-extract-table-as-text-from-the-pdf-using-python/53050405

Your question is near similar to:

https://stackoverflow.com/questions/27927880/extracting-tables-from-a-pdf
https://stackoverflow.com/questions/17591426/extract-table-from-a-pdf
https://stackoverflow.com/questions/25125178/how-to-scrape-tables-in-thousands-of-pdf-files
https://stackoverflow.com/questions/29868541/pdf-data-and-table-scraping-to-excel
https://stackoverflow.com/questions/17217194/extracting-table-contents-from-a-collection-of-pdf-files/26110587#26110587

Regards


Data transformations in hierarchical classification

It depends on the dataset, but generally fit again

why? If you dont fit again on the second level when classifying 1.1 and 1.2 you are introducing bias that you got from the first level when you classified between classes 1 and 2.

why it depends? if information is intertwined between all of the parent and children classes and you will use these models again in the future, you could be loosing important information when fitting again, in other words you will be only over-fitting on the current train (classify 1.1 1.2)


It is just a matter of modelling.

Note that $w \cdot x = \sum_{i=1}^n w_i x_i$.

For example in linear regression (ignore intercept term for now, we can always insert the intercept term by including $1$ as a feature), we try to fit a model of the type of $y(x) = \sum_{i=1}^n w_ix_i$, however, the problem with this when the dependent variable is that $y$ can take values beyond $0$ and $1$.

Hence, we consider another model $F(p) = \sum_{i=1}^n w_ix_i$ where $F(p)$ can take any real value and $F$ is a function which takes value from $(0,1)$ and be able to map it to $\mathbb{R}$. $F(p)$ can be chosen to be $\log \left( \frac{p}{1-p} \right)$, which is an increasing function which map $(0,1)$ to $\mathbb{R}$.


Is it possible to detect a person using facial recognition on the fly?

If you need to re-train the model to classify new faces, this will not scale well to registering new people routinely. You may also suffer from glitches in accuracy during new registrations unless the training routines are carefully monitored.

Instead, recognition systems that need to register new items typically don't re-train. They are trained on the general task of separating objects - identifying whether they are different or what is different about them.

One common way to achieve this separation is to use the NN to map images of faces to a descriptive vector, and match each new image according to distance to stored vectors of profile pictures. The distance, even to the closest stored vector, should be small in order to consider it a successful match. Registering a new user is then a matter of saving a new vector embedding calculated from the neural network. Matches can be done with database lookups - they are still limited by how fast you can do the distance calculations looking for a match, but you only need run the NN forward once. The distance calculations can be fast and batched - it should be easy to use e.g. tensorflow or torch to calculate 1000 potential matches in a fraction of a second.

To ensure that faces are well categorised according to their differences, ignoring lighting, pose, hairstyle etc, the network needs to be trained with this objective in mind. One way to do this is to train with triplet loss using two pairs of images, one that should match, another that should not.

Andrew Ng's course on CNNs covers this approach very well, starting from this lecture.


Found the solution - access the index you want on colnames():
colnames(sports_by_sex)[2], for example.


How to understand conv layer to another same conv layer in VGG16?

If I've got the meaning of question, first convolution accepts inputs of size 224*224*3 means that height and width are both 224 and the size of depth or number of activation maps, channels here, is equal to three. The output of this layer will be activation maps with equal height and width as the input, 224, because it is same convolution. The number of activation maps for this layer which is going to be passed to the next layer is equal to 64 because you have 64 filters in this layer. The point is that each filter is of size 3*3*3 to fit to the input. The output of each filter is an activation map of size 224*224*1. The output of filters come together and construct output of size 224*224*64 which means the input of the next layer will have 64 channels, actually depth here. Consequently, the filters of the second convolution layer is of size 3*3*224 to match the entire input of the previous layer. In other words, you will have 64 filters of size 3*3*64 and each will have an output of size 224*224*1. Take a look at here which definitely can help you.

The purpose of writers of VGG net was to make a network which was just deep enough to perform well on ImageNet data-set. CNNs have lots of hyper parameters which setting them is not based on well behaved math stuff. I mean there is no proof to show which hyper parameter is better than the others. They are found based on experience. They are gained practically. Writers of this paper tried to show that if you use same hyper parameters for convolution layers and just make the network deep and deeper, it was much deeper than AlexNet, you will gain good performance without caring about different set of hyper parameters. For understanding what ConvNets do, there is already an answer here which may help you, it contains the interpretation of different layers in CNNs. They have reached to this setting of convolution layers and architecture by experience and there idea was too use just a deep net without complicated hyper parameters.


How can we use Neural Networks for Decision Making intead of Bayesian networks or Desicion Trees?

One of the central challenges to using neural networks for decision processes is the transparency of the decision mechanism. We tend to grant more trust and credence to algorithms that we can see some of the intermediate reasoning steps for.

Neural networks are, notoriously, black-boxes algorithms in which the intermediate steps are largely obscured from the developer and/or user.

To that end, using a method to explain the decision mechanism of the neural network such as lime or some proxy model approach, to express the decisions made by the network via an alternative model type such as decision trees or linear models, which lend themselves better to interpretation allows us to address this property of neural networks.

That said, there's no direct barrier to using neural networks as the basis of a decision system. Any function which takes data and outputs a prescriptive decision could meet this criterion.

The real trick is ensuring to yourself (as the developer of the algorithm) that the decisions the network is making are in line with the decisions you want it to make and, separately, building confidence through evidence to the stakeholders of the algorithm that:

The engine is making the decisions they need it to
That they have a sufficient understanding of how that decision is being made to trust it to make the decision correctly going forward
That the mechanism the algorithm is using is "fair"-- in that it's not going to adversely affect one group or another in ways that could have serious ethical, regulatory, or legal consequences-- and other risks posed by the implementation of the alorithm for its intended purpose are expressed. This instance comes to mind as as a specific risk for self-driving car decisions. Self-driving cars have potentially huge benefits, but it would be irresponsible (at least) to ignore the ways the AI that powers them could go wrong.

It's worth saying that the last two points above are often not well-understood by stakeholders and therefore fall to the developer, as the person who should have the knowledge of risks related to the approach they are using, to proactively address.


when to use dot product and when to use the common product In neural networks?

Neuron-level operations:

First, let's describe the output of each neuron in the network individually.

For the first hidden layer:

$$ A_2^1 = f_2(Z_2^1) = f_2(X \cdot W_1^1)$$
$$ A_2^2 = f_2(Z_2^2) = f_2(X \cdot W_1^2)$$
$$ A_2^3 = f_2(Z_2^3) = f_2(X \cdot W_1^3)$$

Let me explain the notation a bit: $A_2^1$ is the output of the first neuron of the second layer. $f_2$ is the activation function of the second layer. $W_1^2$ is line connecting the second neuron of the weights you denote $W1$ in the figure.

Second hidden layer:

$$ A_3^1 = f_3(Z_3^1) = f_3(A_2^1 \cdot W_2^{11} + A_2^2 \cdot W_2^{21} +  A_2^3 \cdot W_2^{31})$$
$$ A_3^2 = f_3(Z_3^2) = f_3(A_2^1 \cdot W_2^{12} + A_2^2 \cdot W_2^{22} +  A_2^3 \cdot W_2^{32})$$
$$ A_3^3 = f_3(Z_3^3) = f_3(A_2^1 \cdot W_2^{13} + A_2^2 \cdot W_2^{23} +  A_2^3 \cdot W_2^{33})$$

Again $A_3^1$ is the output of the first neuron of the third layer and $f_3$ is the activation function of the the third layer. This time $W_2^{12}$ corresponds to the weight $W2$ that connects the first neuron of the previous layer to the second neuron of the current layer.

For the output layer:

$$ \hat y = A_4 = f_4(Z_4) = f_4(A_3^1 \cdot W_3^1 + A_3^2 \cdot W_3^2 + A_3^3 \cdot W_3^3)$$

The notation follows the rules above.

Note that all the above operations are between simple numbers, meaning that all $\cdot$ symbols you saw above are common multiplications! Because this way of representing Neural Networks isn't effective, instead of describing each neuron individually, we usually describe the operations of layers with vectors or arrays. This is where your question lies (about the dot product and the transposing).

Layer-level operations:

First hidden layer:

$$ A_2 = f_2(Z_2) = f_2(X \cdot W_1) $$

In the above equation $W_1$, $Z_2$ and $A_2$ are all $1 \times 3$ arrays. While $X$ remains a simple number (you can think of it as a $1 \times 1$ array).

Second hidden layer:

$$ A_3 = f_3(Z_3) = f_3(A_2 \cdot W_2) $$

Again $A_3$, $Z_3$ and $A_2$ are $1 \times 3$ arrays. This time $W_2$ is a $3 \times 3$ array.

Output layer:

$$ A_4 = f_4(Z_4) = f_4(A_3 \cdot W_3) $$

Here, $A_4$ and $A_4$ are numbers and $W_3$ is a $3 \times 1$ array.

Every $\cdot$ in the above is a basic matrix multiplication. Because this example wasn't the best to answer your question, I'm going to change it a bit. Imagine if the second hidden layer had $4$ neurons instead of $3$. In order for us to get from a $1 \times 3$ array to a $1 \times 4$ one we would need a weight array with a shape of $3 \times 4$.

$$
 \begin{pmatrix}
  A_2^1 & A_2^2 & A_2^3
 \end{pmatrix}
\cdot
 \begin{pmatrix}
  W_2^{11} & W_2^{12} & W_2^{13} & W_2^{14}\\
  W_2^{21} & W_2^{22} & W_2^{23} & W_2^{24}\\
  W_2^{31} & W_2^{32} & W_2^{33} & W_2^{34}
 \end{pmatrix}
=
 \begin{pmatrix}
  Z_3^1 & Z_3^2 & Z_3^3 & Z_3^4
 \end{pmatrix}
$$

This is summarized as $A_2 \cdot W_2 = Z_3$. Because in matrix multiplication the order of the two matrices counts, we need to make sure that the operation is possible. If they are in a reverse order we need to transpose the weight array. This would result in the following equation: $W_2^T \cdot A_2 = Z_3$.


Dropping one category for regularized linear models

When you do linear regression you have to leave out one column as it's a singular matrix and hence columns are linearly dependent and we cannot calculate the inverse.
But when you do regularization it take cares of singularity. The matrix is almost surely nonsingular. Hence we don't need to drop a column and if you drop different columns from each feature it could lead to different prediction as it would lead to bias.
Refer this.


How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity?

I'm going to just answer about the difference between the two functions qcut and cut because it's a very important difference:

the first qcut is indeed about quantiles, which means that it's about dividing the data into bins, each containing an equal number of points. For instance if you use deciles it means that there is the same number of people in the first decile as in the last (or any other), so the useful information is the range of values for each bin. For example if the first decile ranges from 0 to 500, then 10% of the people are paid less than 500.
the second cut is not about quantiles, it simply creates bins which all cover an interval of values of equal length. For instance the first interval would be 0-500, the second 500-1000, etc. In this case the number of points in every bin is usually different, and that's the useful information. Typically this is how a histogram is built: equal intervals but possibly different frequency (number of data points) in every bin.

[edit following OP's comments]

Data points are reported salaries for each company role (actually, it is an average value reported )

Ok so the data that you have contains a distribution of the salaries for every role.

The X axis, simply the id corresponding to each role

It's not a very good idea to transform an ordinal/categorical variable into a continuous one, in a case like this you could instead plot a bar for every role (with the name of the role on the X axis). Even more precise, you could plot a boxplot for every role, keeping the order of the roles by mean salary. This way it would be possible to visualize not only the mean (usually replaced by median in a boxplot) but how much variations there are for each role.
Another visualization idea would be to plot the full distribution of salaries with a difference color for every role. In this case X would usually be the salary, Y would be how many people are in this interval (bin) of salary, and stacked bars with different colors show the proportion by role. This would show not only the link between salary and role but also how many people are in each category.

how to bin the distribution (qcut or cut)

I suspect that this is the wrong question, because apparently you're not very familiar with these concepts and how to represent them. So I would suggest that you first play with the visualization ideas I mentioned: normally these don't require you to do any binning yourself, the library should take care of it (see examples in Python).
Knowing whether you need cut or qcut depends on what you want to do with the result. As I said above, quantiles are useful for solving questions such as "80% of the employees have a salary lower than X", whereas cut directly represents "X% of the employees have a salary between 100 and 150". Ultimately the two represent the same information, just under different perspectives. If the difference is still confusing, I'd suggest you start with regular bins with cut because quantiles are slightly more complex to interpret.


You could treat this as a spelling error identification problem. The "Name" column should be a set of unique keys. You can calculate the Levenshtein distance, which finds the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other, between each key. Then set a similarity threshold. Any two keys that are greater than the similarity threshold are merged together.

You can write your own based on Peter Norvig's Spelling Corrector or use Python's FuzzyWuzzy package. The code would be something like this:

from fuzzywuzzy import process

names  = set("BT104 SW104 LBLB-F104 LBLB104".split())
threshold = 85

for name in names:
    leave_current_out = names - {name}
    for match, score in process.extract(name, leave_current_out, limit=1):
        if score >= threshold:
            print(f'Potential misspelling: {name} with {match}')


What does anneal mean in the context of machine learning?

Annealing is short for simulated annealing. Simulated annealing is the process of slowly decreasing the probability of accepting worse solutions as the solution space is explored. Over the course of the experiment, γ value was slowly lowered to balance exploration and exploitation. γ is a machine learning hyperparameter so any hyperparameter search method would work (e.g., manually selection or cross-validation).


Do I need to convert strings before using LSTM?

You can use one-hot encoding to encode your domains as char arrays. Then your training samples should have dimension (samples, longest domain length, all chars used). Here is a code sample:

import pandas as pd
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LSTM

# make some fake samples
urls = ['https://datascience.stackexchange.com/',
        'https://github.com/',
        'www.google.com/']

labels = [1,0,1]
df = pd.DataFrame(zip(urls, labels),
                  columns=['domain', 'label'])

# Make it all to a long string
concat_domains = '\n'.join(df['domain']).lower()

# Find all unique characters by using set()
chars = sorted(list(set(concat_domains)))
num_chars = len(chars)

# Build translation dictionaries, 'a' -> 0, 0 -> 'a'
char2idx = dict((c, i) for i, c in enumerate(chars))
idx2char = dict((i, c) for i, c in enumerate(chars))

# Use longest name length as our sequence window
max_sequence_length = max([len(name) for name in df['domain']])

# build dataset with domains as one-hot encoded chars
X = np.zeros((df.shape[0], max_sequence_length, num_chars), dtype=np.bool)
y = df['label'].values

for i, sequence in enumerate(df['domain']):
    for j, char in enumerate(sequence):
        X[i, j, char2idx[char]] = 1

# build a model with input dim: (length of longest domain name, number of unique chars found)
model = Sequential()
model.add(LSTM(64, input_shape=(max_sequence_length, num_chars)))
model.add(Dense(units=1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam')

# train for 10 epochs
model.fit(X, y, epochs=10)


Check my answer to this question. Nowadays there're many pretrained embedders to choose from. They'll give you fixed-size numerical vector of features. You don't even have to go DNN way, xgboost will work just fine.


deep learning output data in keras fit method

If you are talking about time-series analysis, your test data should be the three columns you want to predict, but shifted by some amount. So the first batch (e.g. first 5 rows) predicts the 6th row. Here is a small diagram showing what I mean:

Notice that the timestamp column not in the target, and that we predict the three features in the following row, given an input of 5 rows.


I'm not sure if I fully understand what you mean, but numpy.expand_dims can be used to expand an array at the given index, allowing you to convert a 2 dimensional array to a 3 dimensional array.

import numpy as np

array = np.array([[1,2],[3,4]])
print(array.shape)
# (2, 2)
array = np.expand_dims(array, 0)
print(array.shape)
# (1, 2, 2)


Choose points to maximize volume of convex hull

Genetic algorithms are often a good option for optimization problems, assuming it's not a requirement for the solution to be the best one but only to be "close enough".


How to do predict a new sms to be spam or not?

You are refiting the CountVectorizer during prediction. Just remove the line

cv.fit([new])

and it should work.


As far as I understand, you want to combine two datasets to use both for training??

If yes, you need to append one dataframe to the other. You need to make sure, that all columns/features needed for training are present in both DFs. After that you can simply append the DFs.

Python example:

>>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))
>>> df
   A  B
0  1  2
1  3  4
>>> df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))
>>> df.append(df2)
   A  B
0  1  2
1  3  4
0  5  6
1  7  8

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.append.html


Appending to numpy array for creating dataset

I think its because of the way you are appending the datasets in list and then converting it to numpy array.

Solution 1

One quick solution is to reshape your array as -

data_new = data_new.reshape(data_new.shape[0]*data_new.shape[1], data_new.shape[2])

So, your data of shape (2,201,4) will become (2*201,4) = (402,4).

Solution 2

Another solution is to append the arrays in the function you have defined, instead of returning np.array(dataX), use -

return np.append(x, y, axis = 0)

So, you don't have to use dataX anywhere.


It is not really possible to alter input feature array size per example on normal CNNs. Instead this is fixed when the model is built for the first time, before you start training.

Depending on your goal, it might be possible to work around that using some kind of pipeline that worked with image patches (taking multiple slightly randomised patches to augment the training data can improve results and doing the same with prediction inputs can also drive up classifier accuracy). Or a more complex variant using RNN/CNN hybrid to consume an image as a sequence of parts, which might also be used for multi-object recognition.

However, these solutions are complex, and state-of-the-art results in image classification can be achieved by simpler techniques such as taking a centre crop and/or padding. Provided your training data is also treated the same way, and aspect ratios are not extreme this can work adequately.


You can use Rexster to expose a REST api and then you can use this api via Requests module (there are many other modules to do this task - though Requests is what I prefer)


In Python, why subsetting with or without square bracket is different?

Assuming, you have a pandas dataframe, .loc is strictly label based. Since you're using [] it accesses the column you're specifying inside the brackets and that is the reason you're getting shares 120. Read documentation for better explanation.

Here is another link that has answers similar to your question.


In softmax classifier, why use exp function to do normalization?

It is more than just numerical. A quick reminder of the softmax:
$$
P(y=j | x) = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}}
$$

Where $x$ is an input vector with length equal to the number of classes $K$. The softmax function has 3 very nice properties: 1. it normalizes your data (outputs a proper probability distribution), 2. is differentiable, and 3. it uses the exp you mentioned. A few important points:

The loss function is not directly related to softmax. You can use standard normalization and still use cross-entropy.
A "hardmax" function (i.e. argmax) is not differentiable. The softmax gives at least a minimal amount of probability to all elements in the output vector, and so is nicely differentiable, hence the term "soft" in softmax.
Now I get to your question. The $e$ in softmax is the natural exponential function. Before we normalize, we transform $x$ as in the graph of $e^x$:

If $x$ is 0 then $y=1$, if $x$ is 1, then $y=2.7$, and if $x$ is 2, now $y=7$! A huge step! This is what's called a non-linear transformation of our unnormalized log scores. The interesting property of the exponential function combined with the normalization in the softmax is that high scores in $x$ become much more probable than low scores.

An example. Say $K=4$, and your log score $x$ is vector $[2, 4, 2, 1]$. The simple argmax function outputs:

$$
[0, 1, 0, 0]
$$

The argmax is the goal, but it's not differentiable and we can't train our model with it :( A simple normalization, which is differentiable, outputs  the following probabilities:

$$
[0.2222, 0.4444, 0.2222, 0.1111]
$$

That's really far from the argmax! :( Whereas the softmax outputs:
$$
[0.1025, 0.7573, 0.1025, 0.0377]
$$

That's much closer to the argmax! Because we use the natural exponential, we hugely increase the probability of the biggest score and decrease the probability of the lower scores when compared with standard normalization. Hence the "max" in softmax.


What should be the main concerns when generating extra data?

It's as you said there is a chance of bias in the dataset. To avoid this you'll have to conform to an algorithm to generate data. Right off the bat, the solution I would propose is parametric methods. Find the statistical distribution of data and depending on that distribution fill in data accordingly.

If you want to know a bit more about parametric methods you can look at my answer here.

What does it mean for the training data to be generated by a probability distribution over datasets

For some machine learning methods look at this answer I posted here.

What are the best way to handle missing values

There is never one single correct way to do what you're asking, but there are good ways and it depends on the nuances of your data.


Neural Networks getting stuck at local optima

Ok all sorted - Bit embarrassing but forgot to normalise the data!


If you haven't figured out what's going on already then you could try a couple of things.

1) Since you mentioned that your input values are very small, you could try normalizing them. I agree that very small (in your case, really small) values might mess up the weight updation. When you do, make sure you normalize first and then do training-testing split

2) Can you tell me how your loss looks during training? Do you see overfitting? If you do, did you use dropout? Did you try L2 regularization on the weights? (only use them on weights, not on biases)

There could be a million things going on and a lot of room for improvement may be there in the network, but I need to have a look at how your loss decreases as the training proceeds. Also, you need to elaborate on how bad your results look? (I assume that your training is quite successful but your testing error is high?). Also , if you're not bound to use CNN, did you try out with an ANN with a few hidden layers and see how that model does?


Can absolute or relative contributions from X be calculated for a multiplicative model? $\log{ y}$ ~ $\log {x_1} + \log{x_2}$

One way I can think of is the following (beware that relative importance can mean different things and be counted in different ways):
The relative contribution of log-log model, seen as a linear model, is found by setting everything else to zero and taking the ratio:
Sp:
$$1/r_{x_1} = \frac{\beta_1 \log x_{1}}{\log y}$$
or:
$$\log y \approx r_{x_1} \beta_1 \log x_1$$
so the relative contribution of the multiplicative model can be derived from:
$$y \approx  x_1^{r_{x_1} \beta_1}$$
$$m_{x_1} = x_1^{r_{x_1} \beta_1}/y$$
One can even use the original variable and its exponent as:
$$m_{x_1} = x_1^{\beta_1}/y$$
Still another way to quantify importance for multiplicative models (inspired by geometric means) is:
Given $y = ax_1^{\beta_1}x_2^{\beta_2} \cdots x_n^{\beta_n}$, then:
$$m_{x_k} = \frac{x_k}{\sqrt[\beta_1 + \beta_2 \cdots + \beta_n]{y}}$$
So if $y = x^2$ then $m_x = 1$ since $x$ has $100$% importance on $y$.

Is there a standard way to make the relative contributions add to
unity? Or alternatively, to make the absolute contributions to add to
y?

When one wants to determine the percent a certain variable affects another variable, one uses a ratio between the two variables. Only for linear additive models do the percents add up to the original value. For multiplicative models these two properties of relative importance diverge and require different approaches (what I explicitly mentioned in the beginning).
So a percent of $x_k$ over $y$ is always some ratio like $\frac{x_k}{y}$ on the other hand for multiplicative models these ratios no longer can be added and don't add up. So one will have to modify their requirements, for example instead of adding up to original value, maybe the product equals the original value.
In this sense one can do a variation of the above proposals (eg the geometric mean inspired formula) and use:
$$y = x_1^{\beta_1} \cdots x_n^{\beta_n}$$
$$m_{x_k} = \frac{x_k^{\beta_k}}{y^{\frac{\beta_k}{\beta_1+\cdots+\beta_n}}}$$
With this definition of relative importance one still uses a ratio (so percent of importance of some value on some other value) while at the same time the product of all $m_{x_k}$ equals $1$, or:
$$m_{x_1} \cdots m_{x_n} = 1$$
Some references which follow another route:

Looking for relative contribution of log and standard variables

How do you find contribution of an independent variable in the given log-log regression model?

Measures of relative importance:

Quantifying the Relative Importance of Predictors in Multiple Linear Regression Analyses for Public Health Studies
Quantifying relative importance: computing standardized effects in models with binary outcomes
Relative Importance Analysis: A Better Way to Communicate Multiple Regression Results
Investigating the Measures of Relative Importance in Marketing Research


What are some best Text Representation techniques in NLP

There's no simple answer to this question. As far as I know in general the choice depends mostly on the type of classification:

Bag of Words (usually with tf-idf weights) is a simple but quite efficient representation for classification based on the text topic or similar, assuming the classes are reasonably distinct from each other.
Word embeddings are a more advanced option for semantic-based classification. They can handle more subtle semantic relations but require being trained on a large training corpus. Using pre-defined embeddings can be a solution but then there's the risk that the original training data isn't perfectly suitable for the dataset.
N-grams models can be used in many different ways but are often chosen when the classification involves syntax and/or writing style. Note that the higher the value $n$, the larger the training corpus needs to be, this can also be taken into account in the choice.

  I might have around 40 categories and then around a same number of sub-categories upto 4 levels.

It depends on the data but 40 classes is already a very challenging classification task. For the sake of simplicity let's assume a uniform distribution over classes: a random baseline accuracy would be 1/40 =  2.5%. Of course it depends on the data and a good classifier will do better than that, but don't expect too much...

Now 4 levels of 40 sub-categories means 40^4 = 2.5 millions classes! Even assuming you have enough data (say around 10 instances by class in average, that is 25 millions instances!), it's very unlikely that a classifier will be able to predict anything useful from such a huge amount of classes.


PCA, SMOTE and cross validation- how to combine them together?

To avoid data leakage, it's important to do any transformations after you split your data - so yes, you want to fit PCA to your training set/fold, but apply that transformation to both training and test data (when you want to consider generalisation).

Scikit-learn makes this process straightforward with sklearn.pipeline.Pipeline and there's a great example where 5-fold CV is done on a chain of PCA to logistic regression which should give you something build on.

To address your second question, there's nothing stopping you from including SMOTE in the mix after doing your train-test or CV split, but without knowing more about the data and situation I won't comment further.

Finally, it is not necessary to nest a train-test split within k-fold CV, since as you say, k-fold CV incorporates a test set. However, for final model selection, a more rigorous approach would be to split off a final validation set before you do anything else. Carry out your k-fold CV experiments on that training data to choose whatever model and parameters, and use the validation results as a final estimate of generalisation.

To implement this, the scikit-learn example above shows how you can evaluate the effect of different parameters on the classifier. What it does not show is final model selection and evaluation, but all that needs to be done is split off a reasonable amount of data prior to everything in the example, and then take the best model(s) and evaluate their performance on that validation set.


What does a conservative technique mean in the context of neural networks?

In this layered graph structure, the
relevance conservation property can be formulated as follows: Let
j and k be indices for neurons of two successive layers. Let Rk be
the relevance of neuron k for the prediction f (x). We define R j←k
as the share of Rk that is redistributed to neuron j in the lower
layer. The conservation property for this neuron imposes
∑j R j←k = Rk.
Likewise, neurons in the lower layer aggregate all relevance coming
from the neurons from the higher layer:
Rj = ∑k R j←k
These two equations, when combined, also ensure a relevance conservation property between layers
Refer to this paper Methods for interpreting and understanding deep neural networks by Grégoire Montavon for more details


error when input function called with shape (2,2)

x, y that goes in tf.estimator.inputs.numpy_input_fn() must be either an array or a dictionary of arrays. Quoting from the link (https://www.tensorflow.org/api_docs/python/tf/estimator/inputs/numpy_input_fn) - x: numpy array object or dict of numpy array objects. If an array, the array will be treated as a single feature.

So in your case it is treating 'feature' as a single feature. You can try this:

x1 = np.array([-1,-2])

x2 = np.array([1,2])

features = {'x1': x1, 'x2': x2}


order of features for model tuning vs model fitting

Ah, I was too quick, and misinterpreted your question!  At the bottom of this post I'll leave my old answer, answering why the test set needs to have the same column order.

As for column order of the data in hyperparameter selection vs. training a final model, no, I suppose there's no real reason these need to be the same.  In a tree model with column subsampling, you're right (in your comment) that the columns will be selected randomly anyway, so the original order doesn't matter.
 Even if you don't use column subsampling, and even for other models: a model generally won't be using the column order as informative; if anything, it's used as a fallback tiebreaker.  (Time series are an obvious exception, but in that case maybe the data isn't tabular in the same way.)

That said, it's still perhaps best practice to use the same pipeline, so that the column order will be the same anyway.  sklearn's hyperparameter tuners make this easy, with refit=True by default just refitting the model pipeline on the best hyperparameters found.

Since sklearn operates on numpy arrays and not pandas dataframes (one of the first things in most sklearn steps is conversion to arrays), you need to make sure columns arrive in the same order as the training data.  Otherwise the model will mistake values of some features as being different features!  Hopefully this will actually break things (wrong feature type, e.g.), but perhaps it will silently make very bad predictions!

This shouldn't be hard if you use pipelines.  The make_column_transformer (and all other steps) will be applied to the testing data in the same way as your training data, so the array after these steps will have columns in the right order.  (Alas, if you want to dig into the results, attaching names to the columns after the preprocessing parts of the pipeline may be a hassle.)


Testable hypotheses construction; minimum predictive strength vs. significance

Good question.  By convention, the null hypothesis is always framed to say there is no statistical relationship between one or more independent variables and a dependent variable.  This is typically tested at the p < 0.05 level.  The level of the relationship beyond it being significant is not addressed in a null hypotheses.  (See, for example, Encyclopedia of Research Design, edited by N.J. Salkind, 2010).  As for testing the "70% confidence" result, there is no way to do that.  That is the statistical result.  The overall validity of a logistic regression model can be assessed using Pearson's chi-squared, R-square, and classification accuracy.


What is meaning of different symbols in image?

In general in probabilities the symbol $\sim$ means "follows the distribution ..", for example:

$$X \sim N(3,7)$$

would mean that the random variable $X$ follows a normal distribution with mean 3 and std. dev. 7. In your example the symbol is used to define the notations apparently: the training data follows the distribution of the real data denoted  $p_{data}$ and the generated samples (whatever these are) follow the distribution calculated by the model denoted $p_{model}$.

The point seems to be about the distinction between the real distribution of the data and the distribution obtained by the model: naturally one wants the latter to represent the former as accurately as possible, that's the job of the estimation algorithm.

An alternative very common notation would be $p(x)$ for the true (i.e. data) distribution and $\hat{p}(x)$ for the estimated (i.e. model) distribution.


K means library designed for large dataset in python

http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html

Although k-means is not to hard to code.


To address your points:

There are no significant autocorrelations
The correlation is low (~0.25), but there are significant autocorrelations.

The data is random & most of the correlations (except for 2 lags) fall within 95% confidence limits
The confidence intervals are used to show which autocorrelations are significant. As you rightly observed, a couple peaks jump out of this region and this tells us that these few correlations are statistically significant, the rest is random. This post may be helpful here.

This timeseries is not worth forecasting
As per the previous point, there are a couple of statistically significant weak correlations in this dataset. But they are not strong, so a periodicity based forecasting model probably wouldn't be very accurate.


Calculate future GDP % using machine learning

I think you should consider time series modeling instead of observation based classification models. In the letter, you are propagating error in each prediction year.

I would use ARIMA, LSTMs, maybe semi-supervised models and motif discovery techniques.


This is a bit unusual because you are throwing away information on the sign: The vectors (-2.998, 8.0, 2.5) and (2.9998, 8.0, 2.5) yield the same result.

Therefore I do not think there is a standard term for this procedure.
A clear comment in your code could be 'Take the absolut value and normalize by the sum of all values' or, if you are feeling mathematical today, 'Take the absolute value and normalize by the $L^1$-norm'.


Automating the training and deployment of the model as more data becomes available

This is a big topic with a lot of possible solutions depending on your context, I'll just provide some of my personal experience from projects I've built.
I would say that it is common in industry to retrain models in an ongoing fashion as new data comes in. Many times this happens at the daily level using either a basic cron-like solution or an enterprise orchestration solution (e.g. Airflow/Kubeflow). Cron is easy to set up and manage for small/personal projects, the others require more engineering set up to manage but they also are more powerful/fully featured.
Once you have a model pipeline which trains in an ongoing basis, you would then need to decide whether to promote the new model to production or keep your existing model running. This could be done automatically given certain conditions (e.g. accuracy of new model > old model) or could require a manual step to evaluate the new model and promote after review. Many times the answer comes down to data quality/stability. For instance, if the input data has intermittent quality issues, you would be less likely to retrain and promote automatically without manual oversight.
Potentially helpful links:
Cron for ML: https://towardsdatascience.com/why-crontab-is-the-ultimate-data-science-back-end-tool-e3f212f2b13d
Airflow for ML: https://towardsdatascience.com/10-minutes-to-building-a-machine-learning-pipeline-with-apache-airflow-53cd09268977
Kubeflow Pipelines: https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/


Reward dependent on (state, action) versus (state, action, successor state)

Your intuition is correct. In the most general case (Sutton's definitions), the model of the environment consists of the state transition distribution and the reward distribution. The latter one is rarely considered, as many times the reward is being assigned by the modeler, dependent only by the selected action from the current state and being deterministic. As you mentioned it simplifies a lot the coding implementation.


Yes, this is a well-studied problem: rank aggregation. Here is a solution with code.

The problem is that the quantity you are trying to estimate, the "score" of the item, is subject to noise. The fewer votes you have the greater the noise. Therefore you want to consider the variance of your estimates when ranking them.


Dimension reduction techniques in R that do not use the full distance matrix

You can try using an auto-encoder which also is a non-linear dimension reduction technique.  It uses a neural network framework to find the most efficient transformation from $p$ dimensions down to whatever you choose.  It then finds how well it can reconstruct the origianl $p$ variables, and keeps tuning until it can optimally recreate the original values as "good" (lowest MSE) as possible.

In this case the input has 5 variables, and it reduces it down to 2 variables using non-linear transformations.  Once you get it down to the reduced variables (the light blue nodes) it then tries using non-lienar transformations to recreate those original five inputs using a similar nn structure.  The right-five red nodes are then compared to the left-five original inputs to see how much information is encoded in those 2 middle light blue reduced variables.

I've found the h2o implementation is pretty easy to use, but finding the correct parameters (number of layers, nodes, learning rate, etc) is still tough to select depending on your data.

Heres a little blog post that goes through the h2o autoencoder with the MNIST data set.


how to see the correlation graph when there is lot of variables?

You are trying to get a look at ~8 millions coefficients at once, I am afraid there isn't a single technique that would really help that.

However you might want to investigate your correlation structure. The first step to do that would be to do some correlation based clustering. In Python, you might want to plot your dendograms : https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.dendrogram.html This will allow you to observe sub-clusters of correlation. Further investigations will depend on what you are trying to achieve.


Detecting unique icon on video in real time

Well, the first thing you need to consider about bar codes/QR Codes is that they are usually black with white background.

If T-Shirts or Unique tags are been used, they should follow this pattern (or any one). You should be able to easily detect these. Take for example this image processing class at the end there is a bar-code example, which here I reproduce:

0 - Given a image:

1 - You can binarize by converting to grayscale and then performing adaptive threshoulding, which is basically a clustering algorithm for pixels:

2 - Then you can filter blobs by eccentricity

3 - Then by major axis:

4 - And finally by orientation

This is a basic algorithm for segmenting bar-codes on hard surfaces. A T-shirt will render distorted bar-codes, but if you can narrow down the search region by detecting probable bar-codes in binary images you probably will get a large speed-up for then use of more complex algorithms such as the ones you propose.

Suggestions:

1 - Add a FCN to regenerate image distorted by T-Shirt surface before step (3) in the previous algorithm

2 - Train a CNN to detect bar-code blobs on binary images (substituting steps 2,3 and 4)

Update

I will add another answer since you changed a bit the question by giving examples. Your unique identifiers seen to be compose of a fixed center (equal to all identifiers) with concentric arcs of 3 different radius $r_1 < r_2 < r_3$ and the arcs with radius $r_2$ and $r_3$ are punctured with holes that are supposed to make them different from in each identifier.

You can use threshoulding and Hough transform to identify the circles, you can use the center as reference to calculate the angular distance between the holes. Then you can differ an identifier from another by:

Counting holes
and then
Analyzing their angular distance

You can also use the arcs to predict/calculate the transformations suffered by the identifier by image distortions (rotation, shifting, non-linear transformation) and correct them.


Normal equation for linear regression is illogical

Hello Oleksii and welcome to DSSE.
The formula you are asking about is not for a pseudoinverse.
$\theta = (X^TX)^{-1}X^Ty$
Where

$\theta$ is your regressor
$X$ is a matriz containing stacked vectors (as rows) of your features/independent variables
$y$ is a matriz containing stacked vectors (or scalars) of your predictions/dependent variables

This equation is a solution to a linear set of equations:
$Ax = B$ that occurs in trying to minimize the least squares loss.
The reason why you see this pinv() on the code is because if X has not enough linearly independent rows, $X^TX$ (also known as $R$, the autocorrelation matriz of the data, and it's inverse is called the precision matriz) will result in a singular (or near singular) matriz, which inversion might not be possible. Even if this singularity only happens because of the working precision of your computer/programming language.
Using pinv() is usually not recommended, because even if it allows to compute a regressor it will overfit the training data. Alternative solutions to working with singular matriz is adding $\delta~I$ to $R$
where $\delta$ is a small constant (usually 1 to 10 eps) and $I$ is the identity.


What features are extracted from pre-trained model of CNN Keras?

Here is a good post about how features are extracted using CNN: https://towardsdatascience.com/cnn-application-on-structured-data-automated-feature-extraction-8f2cd28d9a7e

"I don't know what features are extracted from that."
It depends on what dataset was that CNN pre-trained on. E.g. if the model was pre-trained on a dataset consisting of dogs and cats, the features should contain useful information about dogs and cats.

In practice it turned out that models pre-trained on ImageNet dataset extract features which are a good starting point for training on other custom datasets.


Determining size of FC layer after Conv layer in PyTorch

Hello and welcome to Stack Exchange!

The answer to your question is quite simple: you did not use the correct formula.

The formula you used is (assuming we are working with square inputs)

$$
W'=\frac{W-F+2P}{S}
$$

but the correct formula is

$$
W'=\frac{W-F+2P}{S}+1
$$

Now if we redo your calculations starting with $(1 \times 28 \times 28)$ inputs:

$$
W^{(1)}=28-4+1=25\\
W^{(2)}=\lfloor\frac{25-2}{2}+1\rfloor=12\\
W^{(3)}=12-3+1=10\\
W^{(4)}=\lfloor\frac{10-2}{2}+1\rfloor=5
$$

Considering that the second convolution layer has 16 output channels (or feature maps), you can indeed then calculate the number of inputs as $16\cdot5^2=400$.


Understanding of naive bayes: computing the conditional probabilities

Your formula is correct for one $w_i$, but if you want to classify a document, you need to compute $P(c | w_1,\ldots,w_N)$.

Then you have $$P(c | w_1,\ldots,w_N) = \frac{P(c)\cdot P(w_1,\ldots,w_N|c)}{P(w_1,\ldots,w_N)} = \frac{P(c) \cdot \prod_{i=1}^N P(w_i|c)}{P(w_1,\ldots,w_N)} \neq \prod_{i=1}^NP(c|w_i)$$

where the second equation holds because of the naïve Bayes assumption.

For classification purposes you can ignore $P(w_1,\ldots,w_N)$ because it is constant (given the data). The formula is still simple ("naïve") but doesn't simplify quite as much.

  The last part of the equation looks a bit suspicious to me as it seems way too simple to compute for a rather complex probability.

Keep in mind that while Naïve Bayes is a decent classifier for many applications, the generated probabilities are usually not very representative.


So, yes and no.
First, there's not a layer that does this in the standard Keras API. It might be possible to write a custom layer to do it, but I'm not comfortable enough doing that to guide you.
What you COULD do is create three different layers, each of which accepts the whole thing as input, and let each of them figure out what parts they're responsible for. More on that in a minute.
Second, to do it requires you to use the Functional API. The Sequential API is just that - sequential. But functional is pretty easy once you understand what's going on.
Ok. Now for the above approach.
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Flatten, Concatenate
from tobias_code import get_my_data, compiler_args

data = get_my_data() # obviously, this is a stand-in for however you get your data.

input_layer = Input(data.shape[1:])
hidden = Flatten()(input_layer)

This is where the Functional API differs from the Sequential API. Because the Sequential API assumes each layer attaches to the previous layer, you don't have to manually do so. The Functional API allows you to connect layers however you'd like! But to allow for that, it requires you to manually connect each layer to the previous. I typically reuse 'hidden' to chain my layers together, deviating only for the input, output, and any unusually connected layers.
hidden = Dense(192, activation='relu')(hidden)
main_output = Dense(192, activation='relu')(hidden)

Here's where yours would end. You'd call model = tf.keras.Model(input_layer, main_output), do whatever compiling you had to do, and begin training. But we're going to continue on. (You probably wouldn't use relu for your output layer, but eh...)
# I'm going to build each individual parallel set of layers separately
branch_a = Dense(96, activation='relu')(main_output)
branch_a = Dense(48, activation='relu')(branch_a)
output_a = Dense(24)(branch_a) # This will be one of my outputs, so I want a linear activation

branch_b = Dense(96, activation='relu')(main_output) # note that it is main_output again
branch_b = Dense(48, activation='relu')(branch_b)
output_b = Dense(24)(branch_b)

branch_c = Dense(96, activation='relu')(main_output) # and again. 3 layers are all sharing that output.
branch_c = Dense(48, activation='relu')(branch_c)
output_c = Dense(24)(branch_c)

Now, the Functional API can handle multiple outputs. You can define separate loss functions for each output and separate metrics for each output. But I haven't done a lot with that, so rather than tell you incorrectly how to do it, we're going to concatenate your outputs into one, which is probably what your output looked like in the sequential model.
final_output = Concatenate()([output_a, output_b, output_c])
model = tf.keras.Model(input_layer, final_output)
model.compile(**compiler_args)
model.summary()

Now, I haven't run it, and I make no claims as to its efficacy; I pretty much typed it here and am hoping for the best. But it should get you where you want to be, and hopefully if it's not exactly what you want it will set you on the right path.


Repeated features in Neural Networks with tabular data

Strictly theoretically it makes no difference on accuracy.
Here is why: We already know mathematically that NN can approximate any function. So lets say that we have Input X. X is highly correlated, than we can apply a decorrelation technique out there. Main Thing is, you get X` that has different numerical representation. Most likely more difficult for NN to learn to map to Outputs y. But still in Theory you can Change the architecure, Train for longer and you can still get the same Approximation, i.e. Accuracy.
Now, Theory and Praxis are same in Theory but different in Praxis, and I suspect that this Adjustments of Architecture etc will be much more costly in reality depending on the dataset.


metric learning and information retrieval

Conditional Random Fields (CRFs) can be used for segmenting/labeling sequential problems. Try CRF++: Yet Another CRF toolkit,  a simple, customizable, and open source implementation of Conditional Random Fields (CRFs)

You can label and create a tagged training corpus and use CRF++

You also need to create a feature template

Refer : http://taku910.github.io/crfpp/ for more details.

Check the example from data for CoNLL shared task (PoS tagging).


What approaches could be used to teach a model to make appropriate brush strokes to reproduce a painting?

Main Idea:

The main idea is the you could measure how good is the output and the input of your model simply representing it by some mathematical object, for example a vector in a high dimensional space and then using some error function to measure the fitness (how good your model is performing).

A possible formalization

The scenario your describing fits in a more general optimization function, the most general optimization scenario is;

$$ min_{x \in X} f(x) $$

where $X$ is the domain. In the case of image processing and computer vision $x$ is a matrix of pixels, this matrix of pixels can be represented by a vector $x \in \mathbb{R}^{m \times n}$. In the case you describe you could also represent the input (painting) as a vector $y\in \mathbb{R}^{m \times n}$.
Then $f$ coul be RSME between some $x$ and $y$, i.e. $f(x) = {\| y - x \|}_2$. Notice that $y$ is given and it does not change.

Now the key question here is how to represent a stroke, or more formally a function the values of $x$. A very basic idea can be represent the stroke by the following tuple $(o, e, t, c)$ where $o$ and $e$ are two dimensional points, the origin and the end of the stroke; $t$ is the thickness of the stroke and $c$ is the color of the stroke. Let $S$ be the space of all the strokes, assuming the thickness and colors are finite, S is finite. The possible pairs of $o$ and $e$ are finite namely all the pairs of possible points in $m \times n$. Now let $S^k$ be the composition of $S$ with itself $k$ times.

Finally the problem can be represented the following way:

$$\min_{s \in S^k}{\| y - t(s, x) \|}$$

Where $t$ is the function that applies $s$, that is a number of different strokes to an empty canvas $x$, by empty canvas I mean a matrix full of zeros.

Summarizing the ideas expose above are a possible formalization of your problem, of course is not the only one. The main issue is that the robot does not "learn" is simply computes the optimal set of strokes to apply.

Other ideas

Another possible approach is to use reinforcement learning, this learning paradigm does not needs labeled training samples, it relies in a reward function $R$ to guide the learning process. Also if defines a set of states and actions the agent can be or do. The states could be the strokes and the actions add another stroke, the reward function can be a variation of the error function. Also I am far from being an expert in reinforcement learning I guess is worth looking into it.

A proven approach

The problem you describe has been demonstrated in real life. See this paper: Feedback-guided Stroke Placement for a Painting Machine. The paper demonstrates a system that describes your scenario, also see this site for more work of the authors or this for the site of the project.


The answer to this question really depends on the insights you are after with these statistics.
Generally speaking however, it is wise to investigate per class in such a situation. Taking your examples this would entail comparing average string lengths for both classes. This will help you understand how both classes are (dis)similar given the dataset.


Derivation of the cross-entropy equation in Michael Nielsen's book

The author has taken activation function as sigmoid in this case. The derivative of this function can be re-written like

$$\sigma(z) = \frac{1}{1+e^{-x}}$$

whose derivative $$\sigma'(z) = \frac{e^{-x}}{(1+e^{-x})^2}$$ which can rewritten as $$\frac{1+e^{-x}-1}{(1+e^{-x})^2} \rightarrow \sigma(z) * (1 - \sigma(z))$$. This is subtly mentioned in eq.3 of chap1.


https://paperswithcode.com/task/time-series-classification/latest -> You can find all the state-of-the-art papers along with code for Time Series Classification.
https://arxiv.org/pdf/1809.04356.pdf -> This paper provides an in-depth review of some deep learning architectures for Time Series Classification.
Hope it helps!


prediction() returning mistakenly false positives

The slot "fp" counts how many false positives there are at each choice of classification cutoff (which can be found in the "cutoff" slot). The cutoff represents at what value you set the threshold to binarize the numerical values into classes. Your output already appears to be binary classes, so the concept of a threshold doesn't really make much sense, but the package still tries, setting the potential thresholds at Infinity, 0, and 1. When you set the threshold at 0, everything gets classified positive, including the 2 actual negative samples - when the classification threshold is 0, you get 2 false positives.


How can we use the current rewards as a system input in the RUN time when working with Deep Q learning?

My friend - Thushan Gave me this answer . So I will post it . I think it will describe it . Sometimes in the run time it's not desirable to work with immediate rewards . It can mess up the agent. I will quote it here .

  Your argument makes sense, but not always. And this is a design decision
  you make in your problem formulation. However to highlight why this
  won't work in some situations, I'll give an example.

  Say you're playing a shooting game with a RL agent. And at some
  specific location you get killed by getting shot from an enemy.
  However if you include the reward with the state, this might suggest
  to the algorithm, that this location is always bad and will try to
  avoid it (which is wrong). We should't be avoiding locations, but
  enemies.


About neural network called "Self Organizing Maps"

Lateral connections exist so that the update of a neuron forces the neighboring neurons also to be updated but to a lesser degree.

Think of each neuron that has n inputs as a vector, a point in an n-dimensional space. It's coordinates are the values of the weights on its inputs.

When the network receives an input - also an n-dimensional vector, each neuron calculates the distance between its weights and the input vector. The neuron whose weights are the closest to the input is the winner, and is allowed to update its weights. It updates them by moving one step towards the input vector. The size of the step is equal to the learning rate.

While it moves, it uses the lateral connections to pull its neighbors (or push some of them away, depending on their distance to the winner and on the form of the function that you use for the lateral connections). As a result, the neighbors also move but they move less than the winner.

Only the winner and its neighbors are allowed to update their weights. All other neurons don't move. The neighbors are those who are closer to the winner than a certain radius. This radius is a hyper-parameter.

This type of learning allows to chart the space of the input vectors. The result is the weights of the neurons are distributed more or less uniformly in the set of the input vectors. After training, each neuron can be thought of as a representative of some region of the input space.


How to handle non-numeric user id's in a recommendation model

I'm not sure why you would include user ID as a predictive variable, unless there is specifically some part of the user ID that gives you information. For example, maybe the first three letters tell you that the user is a premium user and then you could feature engineer a category from that.


Regression using gradiant boosting - smoother predictions

Another approach could be to train several models and then simply take the average of their predictions - essentially using an ensemble. It will be necessary to initialise the models differently and ensure there is some randomness/stochasticity during training such that the models are indeed different.

This approach should allow your grupo of models to tolerate one or two models giving odd predictions in each new prediction, while still giving the most sensible result.

Studies show that this can dramatically smooth predictions. Have a look at this comparison of use cases and various ensemble technique from Armstrong. There are some guiding principles suggested, depending on saveral factors such as domain knowledge and variance of predictions.


How to quantitatively compare two or more complex data sets

One option is to calculate a distance metric between the data sets.
Choosing a distance metric depends on the properties of the data. If the data is binary, the Hamming distance can work. If the data are sets, the Jaccard distance can work. Other data typee require other distance metrics.


How to build recommendation system that also takes time as a feature

Use a diminishing sample weight where the newest sample = 1 and older samples get lower weights. This way if a user changes his interests over time, your recommendations will change accordingly.


How to train a model to predict if 2 samples refer to the same thing?

If I understand correctly, it sounds somewhat like an NLP problem. For each row, and each column, you will compute the similarity between entries across dataframes e.g.
Your two dataframes:
Row182, ColumnA, DataFrame1 = XYZ_ABC_123
Row182, ColumnA, DataFrame2 = XYZ_ABC_152

New data frame:
Row182, ColumnA = 0.8

Repeat for all rows and columns. You would then set some threshold at an acceptable similarity rate / accuracy of your similarity metric matrix. To highlight the most severe cross-overs.
You could have a look at https://www.nltk.org/, but also depending on the data entries you could just do a pure pythonic solution e.g. 80% match between string entries.
I'm guessing at what your data looks like, so some examples there could help


scalable tools to build kNN graph over sparse data

I think what you might be looking for is,

L2Knng: Fast Exact K-Nearest Neighbor Graph Construction with L2-Norm Pruning

They have multiple runtime options specifically for different kinds of datasets (including sparse data). The link for the same is : http://glaros.dtc.umn.edu/gkhome/node/1162


Sampling Big Data for Predictive Analytics in Python

This is what I do in projects :

Pre-process data in DB / Data Lake. aim is to :
A. Form batches (might require a new table with shuffled indices)
B. Create a copy with Normalization and other feature related tasks

After this, try algorithms that support batch learning. Neural networks support batch learning and few other algos (https://sklearn.org/modules/scaling_strategies.html#incremental-learning).

With batch learning, you can plot loss for each batch and see if algo is working or not.


Distinguish real building pictures from rendered ones

To apply machine learning, you need to find thousands of labeled examples of both real and rendered images.
After you have labeled data, most likely a Convolutional Neural Network (CNN) will learn the differences between the labels.


Is it common to add noise to Time Series data before training a model

I'll go through your questions one by one:

  I once read about somebody who added noise to their time series before training a model. They didn't write why they did it though.

No, noise is not adding any value to your dataset. The reason is that noise, by definition, cannot be learned by any statistical/ML model.

  Is this common practice?

No it's not.

  why do people do it ie. to prevent over-fitting?

It's not a technique that helps you in preventing overfitting. There are several techniques to do it (ensemble modeling, cross validation), many of them depend on the kind of model you want to implement (such as recurrent dropout for RNNs).


Modeling uncertainty from Logistic Regression

I see many facets of your question and in what follows will present my top 2 :-)

Facet 1 -- assessing the uncertainty in estimated coefficients

In logistic regression, assessing the uncertainty in the estimated coefficients is virtually the same as for least-squares regression .

In both logistic regression and least-squares regression, the regression coefficient table will include a column for the regression coefficients followed by a column of standard errors, then by a column of test statistics, and finally a column of p-values. The table below shows the example coefficient table output for the some regression problem (where the   probability of buying some magazine for kids is estimated).

Note that the test statistics are labeled “z value” and the p-values are labeled “P(>|t|)” in the table above.
The standard errors can be used to construct confidence intervals for the regression coefficients. Roughly speaking, going plus or minus 2 times the standard error from the regression coefficient gives approximately a 95% confidence interval for the coefficient.

For example, for Residence Length, the regression coefficient is 0.024680. The next column gives the standard error of the regression coefficient which is 0.013800. Thus an approximate 95% confidence interval for the Residence Length regression coefficient is:

$$
0.024680 \pm 2 \times 0.013800 = 0.024680 \pm 0.0276\
$$

This means that the regression coefficient for Residence Length could be anywhere from -0.00292 to 0.05228 (with 95% confidence).

As is well known, we often use the odds-ratio, which is the exponential of the regression coefficient (i.e., $\exp(\beta)$ ), to help to interpret the meaning of the regression coefficient. The odds-ratio for the Residence Length coefficient, as shown in the coefficient table, is 1.0250. This means that there is a 2.5% increase in the odds of buying the magazine associated with each additional year of residence.

We can also compute the odds-ratios corresponding to the ends of the confidence interval. These odds-ratios will give us an equivalent confidence intervals for the odds. So continuing the example using Residence Length, the odds ratios corresponding to the ends of the confidence interval are

$$
 \exp(-0.00292)=0.99708
$$
 and
$$
 \exp(0.05228)=1.05367.
$$
Thus, the interval [0.99708,1.05367] is an approximate 95% confidence interval for the odds ratio. This means that there could be anywhere from a 0.292% decrease to a 5.367% increase in the odds of buying the magazine associated with each additional year of residence.

Facet 2 -- modeling data with measurements uncertainty

Here again many options are known. Take a look at this to get started: https://pdfs.semanticscholar.org/0a7a/a5e3d407b24e2f5c24287bfdda20e573bd05.pdf

I will just cite the source:

  in some datasets, the outcome of interest is measured with imperfect
  sensitivity and specificity. It is well known that the
  misclassification induced by such an imperfect diagnostic test will
  lead to biased estimates of the odds ratios and their variances.

  In this paper, the authors show that when the sensitivity and specificity
  of a diagnostic test are known, it is straightforward to incorporate
  this information into the fitting of logistic regression models.

  An EM algorithm that produces unbiased estimates of the odds ratios and
  their variances is described. The resulting odds ratio estimates tend
  to be farther from the null but have greater variance than estimates
  found by ignoring the imperfections of the test. The method can be
  extended to the situation where the sensitivity and specificity differ
  for different study subjects, i.e., nondifferential misclassification.
  The method is useful even when the sensitivity and specificity are not
  known, as a way to see the degree to which various assumptions about
  sensitivity and specificity affect one's estimates.

Hope this helps!


Still have the same error after reshaping

Dont reshape X. It was fine in the first place.
Try y = y.reshape(-1,1)
You were missing the assignment back to y, since reshape returns a new object.


How many neurons a deep learning model (specifically a CNN) does have?

I hope you are talking about the parameter count in a convolutional neural network.
Below is basic understanding about calculating the parameters. After getting this, it would be self explanatory.
kernel count = k
kernel size = (a, a)
depth of input = d
Total number of parameters: k*(a*a)*d + k

Now, for below scenario which has one input layer with one con2d layer.

To cross check the number of parameters, we need k,d,a:
 k = 64, d = 3, a*a = 3*3

Total parameters: 64*(3*3)*3 + 64 = 1792

Hope its clear now.


Explain forward filling and backward filling (data filling)

Yes you are correct. Forward filling and backward filling are two approaches to fill missing values.
Forward filling means fill missing values with previous data.
Backward filling means fill missing values with next data point.

You can refer the below url for more on missing values:

https://www.kaggle.com/juejuewang/handle-missing-values-in-time-series-for-beginners

https://www.kaggle.com/dansbecker/handling-missing-values

https://www.kaggle.com/meikegw/filling-up-missing-values


MLP Parameter tuning - gridsearchCV cannot fit?


How to extract sub sentences from sentence mentioning a particular subject?

If you would know the target(s) (pasta, pizza) and the sentiment features (good, bad, etc), you could try to catch the feature that is closest to the target in a sentence.

But to say more, it really is necessary to see more of your sentences to understand the structure.


What is the difference between filling missing values with 0 or any othe constant term like -999?

First, DeviceType and DeviceInfo don't sound like naturally numeric values.  If you're going to need to encode them anyway, then it doesn't matter: just encode "missing" as another level (or the baseline all-zeros).  And if the non-missing values are nearly-unique, they may not be very useful anyway; perhaps just the fact that they exist is informative?

Tree models can deal with missing values implicitly (splitting the non-missing data into two subsets, then examining which side the rows with that feature missing should go to); however, not all implementations allow for that.  E.g., sklearn doesn't yet (but working on it?) allow missing values at all; xgboost and lightgbm do what I've mentioned above; catboost only sends in one fixed direction (https://github.com/catboost/catboost/issues/588).  Quinlan-family trees actually send missing values along all possible paths, and return a result that's a weighted sum of the possible results, weights coming from the proportion of the training data in the node that went along each path (https://stats.stackexchange.com/a/98967/232706).

In a tree-based model, imputing with $-999$ (or any value less than all your data) is the next best choice: it allows you to split the rest of the data however the tree normally would, and just always sends the "missing" rows to the left.  You may want to test imputing with a very large value as well (which will instead always send the missing rows to the right); again, see the catboost github issue above.

For a linear model, imputing with anything will distort the distribution and the model.  However, adding a missingness indicator and imputing (with anything) takes care of the model: the coefficient on the imputed feature can fit to the "real" slope, while the coefficient on the indicator prevents the imputed value from pulling that slope away from its true value.  An indicator variable may also help in a tree-based model, though that's not as certain.


GPU performance is about 50% slower than benchmarks

My guess is that the reduction in performance is due to differences in versions PyTorch. The published benchmark uses torch==0.4.1 (even though the README states something different). You are using PyTorch 1.0.1.

PyTorch 1.0 speed is sometimes slower than lower than 0.4. PyTorch 1.0 can be speed-up by adding torch.backends.cudnn.benchmark = True and compiling from source.

Often times published benchmark are heavily tuned from the "off-the-shelf" version.


Visualizing F-score differences in information extraction

Yes, I think that's a sound approach and a good way to compare different systems.

A ROC curve comparison is usually more informative than the raw performance scores, but it's still quite general. In case you want to observe even more detail, you could also try to look at specific groups of instances. One way to do that is to count for every instance how many systems correctly classify it: an instance almost always correctly classified is "easy", and conversely an instance which is almost always misclassified is "hard". It's often interesting to look at what happens specifically for the "hard" instances with the different systems. You could take a subset of "hard" instances and calculate the performance or ROC curve only on those, in order to distinguish more precisely the best systems.

For the record, if it makes sense for your task you might also want to consider more flexible scoring methods for text spans: currently it seems that your evaluation considers an answer correct only of the exact span is predicted. You could consider counting the fact that a span is partially correct, for instance by counting the number of tokens correctly annotated.


principles of time series analysis by neural network models

According to the Efficient-market-hypothesis, you are right: the price movement of today does not affect the price movements of yesterday. However, this hypothesis describes an abstract assumption, which can be very useful for macroeconomic modeling but should not be considered as a comprehensive description of reality (The same applies to other economic assumptions, like that of the Homo economicus for example.).

In fact, there is a lot of criticism regarding the efficient market hypothesis. The Cryptocurrency-Bubbles and the Dot-com bubble are most probably the best examples that price movements in financial markets are not pure random walks in many cases, but interfere with behavioral psychological effects.

From my experience, I can assure you that many indicators are quite consistent: if you're trading in a growing market, the probability of an increase in prices is slightly higher than that of a decline (even though there will be a tipping point sooner or later). In addition, there are other indicators, such as volatility, which are relatively stable. Of course, you could try to manually determine all these indicators and include them as features in an ordinary DNN, but in practice, using an LSTM usually turns out to be easier and more accurate.


State of the art approaches for Information retrieval tasks based on deep learning

Deep learning extends the ability of information retrieval (IR) systems.

Deep learning has proved to be a powerful tool for feature engineering. This improvement in feature engineering, over manual construction, has improved the quality of any machine learning system, including information retrieval.

Specifically, word embeddings create a dense, vectorized representation of words that encodes the semantic relationships based on co-occurrence. Both documents and queries can be projected into the same latent space and nearest-neighbors can be found. Word embeddings can be used to extend the posting index to by finding related terms. They can also be used for query-expansion / query rewriting.


Why a sign of gradient (plus or minus) is not enough for finding a steepest ascend?

Using only sign of gradient is a way to go, but might result in slow convergence. Nevertheless it is a valid variation of the method.

The Geometry of Sign Gradient Descent

Sign-based optimization methods have become popular in machine
learning due to their favorable communication cost in distributed
optimization and their surprisingly good performance in neural network
training. Furthermore, they are closely connected to so-called
adaptive gradient methods like Adam. Recent works on signSGD have used
a non-standard "separable smoothness" assumption, whereas some older
works study sign gradient descent as steepest descent with respect to
the ℓ∞-norm. In this work, we unify these existing results by showing
a close connection between separable smoothness and ℓ∞-smoothness and
argue that the latter is the weaker and more natural assumption. We
then proceed to study the smoothness constant with respect to the
ℓ∞-norm and thereby isolate geometric properties of the objective
function which affect the performance of sign-based methods. In short,
we find sign-based methods to be preferable over gradient descent if
(i) the Hessian is to some degree concentrated on its diagonal, and
(ii) its maximal eigenvalue is much larger than the average
eigenvalue. Both properties are common in deep networks.

Properties of the sign gradient descent algorithms

The aim of this article is to study the properties of the sign
gradient descent algorithms involving the sign of the gradient instead
of the gradient itself and first introduced in the RPROP algorithm.
This article provides two results of convergence for local
optimization, a first one for nominal systems without uncertainty and
a second one for systems with uncertainties. New sign gradient descent
algorithms including the dichotomy algorithm DICHO are applied on
several examples to show their effectiveness in terms of speed of
convergence. As a novelty, the sign gradient descent algorithms can
allow to converge in practice towards other minima than the closest
minimum of the initial condition making these algorithms suitable for
global optimization as a new metaheuristic method.


The Area Under an ROC Curve (AUC) vs Confusion Matrix for classifier evaluation?

A confusion matrix can be used to measure the performance of a particular classifier with a fixed threshold. Given a set of input cases, the classifier scores each one, and score above the threshold are labelled Class 1 and scores below the threshold are labelled Class 2.

The ROC curve, on the other hand, examines the performance of a classifier without fixing the threshold. Given a set of input cases, the classifier scores each one. The ROC curve is then generated by testing every possible threshold and plotting each result as a point on the curve.

The ROC curve is useful when you want to test your classifier over a range of sensitivities/specificities. This may or may not be a desirable thing to do. Perhaps you want very high sensitivity and don't care much about specificity - in this case, the AUC metric will be less desirable, because it will take into account thresholds with high specificity. The confusion matrix, on the other hand, could be generated with a fixed threshold known to yield high sensitivity, and would only be evaluated for that particular threshold.

A confusion matrix evaluates one particular classifier with a fixed threshold, while the AUC evaluates that classifier over all possible thresholds.


Should softmax cross entropy with logits always be zero if logits and labels are identical?

Logits are the pre-transform values in a layer, and are not compared directly to the labels when calculating the cost function. In fact, with softmax layer, setting the matching logit to 1, and the rest to zero isn't even that good, because (using $z_i$ as logit $i$ and $a_i$ as activation $i$):

$$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$$

Substitute in your 0 and 1 values for $z_i$ and you will see the $a$ values are not that close to 0 and 1 (they would be 0.0853 and 0.2319 respectively for 10 classes). It is these activation values that feed into the loss calculation. So with typical multiclass loss function $l = -\sum_{\forall i} y_i log(a_i)$, you would score a loss of 1.4611 for such an example.

To get a very low cost, instead set the logit matching the true label (or 1) to a higher value, e.g. 10, and the logits matching the false value (or zero) to a low value e.g. -10. This would lead to

$$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$$

Where

$$z_i = \{10\; if\;1\;\; |\;\;-10\;if\;0\}$$

And varies between 10 and -10 for values between 1 and 0.

That should significantly reduce the calculated loss (1.855e-08 for those values in a 10 class prediction). The logit allows us to 'stretch' the values for correct and incorrect predictions.

Getting an actual 0 loss when using softmax or sigmoid outputs and logloss is not technically possible. You might see it due to rounding in computation. However, neither sigmoid nor softmax equals exactly 0 or 1, except in the limit as input values tend to $\pm \infty$