What is an interpretation of the $\,f'\!\left(\sum_i w_{ij}y_i\right)$ factor in the in the $\delta$-rule in back propagation?

The equation that you show calculates (the negative of) a gradient to the objective function. The value $\delta_j$ is the rate of change of the objective function for an isolated change of output of a neuron before applying the activation function for the neuron. The factor of $f′(net_i)$ is required because it is correct - it is due to the chain rule applied twice - once due to the linear relationship linking weights between layers, and again to get the gradient pre-activation function in the next layer down.

An interpretation of $\delta_i$ is "the (negative) gradient of objective function with respect to a pre-activation value of a neuron the network". If, somehow, the pre-activation value could be changed - ignoring how it could be changed - then $\delta_i$ tells you the linear scale of the impact on the objective function, at least for very small changes.

This value is not normally used directly in weight updates, but is fed deeper backwards into the networks layers if there are yet lower layers to update, because it can be used for weight updates in those layers using the same rule.

The property of low gradients associated with strong activations is because you previously chose tanh as an activation function, and is not because of the way you calculate the gradient. Once you chose tanh, you were not really free to choose a different backpropagation gradient calculation. Or rather, there is not much to gain - and potentially a lot to lose - by doing so.

  What is the purpose of this for weight updating?

The value shown is not used directly, but used to derive $\frac{\partial J}{\partial W_{ij}}$ gradients for weights in deeper layers (where $J$ is value of your objective function).

In terms of your equations

$$\frac{\partial J}{\partial W_{ij}} = -\delta_j y_i$$

This is very simple, so it has been folded in to your delta update rule. However, doing that is hiding something important. Out of all the intermediate values you calculate during back propagation, $\frac{\partial J}{\partial W_{ij}}$ is the key one. It is the gradient of the objective function with respect to parameters that you can change.

Once you have a set of gradients for  $\frac{\partial J}{\partial W_{ij}}$, this can be used in various ways.  Using this gradient directly for weight updates using the delta rule (as in your equation $\Delta W_{ij} = \eta \delta_j y_i$) is a basic approach. The purpose of this rule is to reduce the value of the objective function by taking a small step in the direction that you have just calculated will reduce the value of objective function. The step size needs to be small (multiplied by a factor $\eta$) because you have only measured the gradient at one point, and don't know for what step size the relationship still holds - complicated by the fact that you are making a similar step in multiple dimensions at once.

The delta update rule is a relatively weak update method compared to e.g. Nesterov momentum, Adagrad, Rmsprop optimisers which will use the calculated gradient as input, but adjust the update weights based on history of previous gradients. These optimisers can deal with low but consistent gradient values and make large updates.

  How would the networks behave differently
  if one would remove or change this factor?

This is not advisable as effectively you would be optimising against a different network architecture than the feed-forward network you had constructed. The most likely way they would behave is to either diverge or settle on a larger value of your objective function than you would otherwise see. Making small changes (such as a minimum absolute gradient) can work, but usually such a tweak is best done consistently, by changing the activation function as well.

In a few edge cases, you might consider "lying" in the gradient calculation. A trivial case could be considered the ReLU activation function ($y = 0 (if x \lt 0), y = x (otherwise)$) where the gradient is technically undefined at $x=0$, but can safely be treated as either 0 or 1.

An alteration to updates that is more usually done (by e.g. momentum variants, Adagrad, RMSProp et al) is to use the gradient as-is, but then multiply update amounts by various factors based on history or estimates of higher-order derivatives, because you can guess that the gradient will not switch sign if you update a weight by a smaller or larger amount. These approaches replace the delta update rule with update mechanisms that are more robust when gradients become small.


Tips for checking data integrity / data sanity?

Here are a few things to check in a data set, there could be a lot more but i'm sharing what i did myself.

Check the NULLs in your columns, count them to see if there is a column that contains too many NULLs and that you might want to eliminate
Plot your data distribution each column at a time to see the range of values taken by your data, to see if there is something particularly remarquable about its behavior.
And I'd recommend this famous kernel on Kaggle where a good exploratory data analysis is done.


can we learn a model to pre-process text?

Based on the details given in the comments, it looks like what you need is a binary classifier which predicts whether or not to remove a "numerical point". This means that each instance is an individual "numerical point", and you would need to annotate a sample of instances to use as a training set.

A simple option is to use a bag of words representation, i.e. represent every instance as a vector over the vocabulary where each cell contains the frequency of the corresponding word (or other variants, e.g. binary or TF-IDF weights). Then many classification algorithms can be used: decision trees, SVM, Naive Bayes, etc.


Does Decision tree classifier calculate entropies before transforming categorical features using OneHotEncoder or transformation should be done

It's not totally clear to me what is your main question but here are a few points which might help you:

Like any regular classifier, a decision tree classifier does not transform the features. It's up to the user if they want to apply any transformation before calling the classifier.
Standard decision tree classifiers can deal with features which have multiple values.

Afaik training a decision tree model using a library classifier is not going to help you much understanding how the model is calculated, it's just going to give you the result. You could try to code your own implementation if you want.


Generative Adverserial Networks can be tolerant of this type of perturbation. In fact, if you look at figure 8 of the DCGAN paper, the authors demonstrate how the GAN learned to encode perspective directly such that the authors are able to use the GAN to rotate the camera on an image by applying a "turn vector".


Andrew Ng word2vec @1:29. In the video, his choice of window size is more than one. In his example, if "orange" is the center word, the target word could be "juice", "glass", "my", which are all within the window size of 5.

In your example, "cat sat on the mat", if you choose a window size of one, you can get 7 training samples:

(cat, sat),
(sat, on),(sat, cat),
(on, sat), (on, the),
(the, on), (the, mat)

If this is the entire corpus, p(cat|sat), probability of sat occurs as the center word, cat being the context, is 0.5, p(sat|cat), cat being the center word, sat being the context is 1. The distributed word representation of "cat" and "sat" being the center word should be different.

"it seems like for the same center word we can get different outputs:" That's because they are predicting more than one context words.

@31:35 224 2017 lec2, the slide says "predict surrounding words of radius m for every word". That is, the problem is to find a distribution of context words given the center word, so for one center word, the model will have more than one output words (unless you set m to 1 and specify the direction as well).

"I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer."

@39:37 224 2017 lec2 The professor did a dot product between the matrix output and the word representation. It's also shown on the top of the note: "$u_o ^T v_c$"

Multiply the hidden layer (d * 1) by Weights 2 (V * d) will get a V*1 vector. That's $u_o$, output vector representation. Then we do $u_o ^T v_c$ to get the output for each context word. After that, we apply softmax and compare that value with the context word's one-hot representation ($y$ vs $\hat{y}$), three comparisons in the prof's example.

That sounds like the idea of the Continuous Bag of Words model.


How to run a model stored as a .Rda file?

I suppose you come from Python. R is quite different as it does not require to save to external files your model, you just need to have it in your workspace. Once you have a model object, you just have to use the predict function and specify the unseen dataset. Here's an example of a tree model:

model<-tree(Y~X1+X2, data=dataset) #estimate of the tree model
predictions<-predict(model, newdata=unseen_dataset) #prediction on new data

Be sure variable names between datasets correspond.


cross validation on whole data set or training data?

Yes, it's called overfitting. Your model is beginning to memorize the training set, but not performing well on any validation or test set. If your question is why is this happening, I'd like to refer you to another answer I wrote explaining this phenomenon in more detail.

One interesting question that could be made is why is the performance on the cross-validation folds worse than on the test set?

This is a bit more tough to answer, because I don't have all the details. Some possible explanations could be that the since the training set is larger than each fold, the model was trained better, or that simply the test set examples happened to be easier.


Gradient boosting how can accuracy increase when we lower the depth of tree?

In boosting, a low biased & low variance model is ensembled by additively combining high biased & low variant models. Here, biases are gradually diminished while keeping variances low. Therefore, a tree with a single depth, that is highly biased, isn't an issue because additively errors will be reduced.


What is the difference between SVM and logistic regression?

Both logistic regression and SVM are linear models under the hood, and both implement a linear classification rule:

$$f_{\mathbf{w},b}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T \mathbf{x} + b)$$

Note that I am regarding the "primal", linear form of the SVM here.

In both cases the parameters $\mathbf{w}$ and $b$ are estimated by minimizing a certain function, and, as you correctly noted, the core difference between the models boils down to the use of different optimization objectives. For logistic regression:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i \log(1+e^{-z_i}),$$

where $z_i = y_if_{\mathbf{w},b}(\mathbf{x}_i)$.

For SVM:

$$(\mathbf{w}, b) = \mathrm{argmin}_{\mathbf{w},b} \sum_i (1-z_i)_+ + \frac{1}{2C}\Vert \mathbf{w} \Vert^2$$

Note that the regularization term $\Vert \mathbf{w} \Vert^2$ may just as well be added to the logistic regression objective - this will result in regularized logistic regression.

You do not have to limit yourself to $\ell_2$-norm as the regularization term. Replace it with $\Vert \mathbf{w} \Vert_1$ in the SVM objective, and you will get $\ell_1$-SVM. Add both $\ell_1$ and $\ell_2$ regularizers to get the "elastic net regularization". In fact, feel free to pick your favourite loss, add your favourite regularizer, and voila - help yourself to a freshly baked machine learning algorithm.

This is not a coincidence. Any machine learning modeling problem can be phrased as the task of finding a probabilistic model $M$ which describes a given dataset $D$ sufficiently well. One general method for solving such a task is the technique of maximum a-posteriori (MAP) estimation, which suggests you should always choose the most probable model given the data:

$$M^* = \mathrm{argmax}_M P(M|D).$$

Using the Bayes rule and remembering that $P(D)$ is constant when the data is fixed:

\begin{align*}
\mathrm{argmax}_M P(M|D) &= \mathrm{argmax}_M \frac{P(D|M)P(M)}{P(D)} \\
 &= \mathrm{argmax}_M P(D|M)P(M) \\
&= \mathrm{argmax}_M \log P(D|M)P(M) \\
&= \mathrm{argmax}_M \log P(D|M) + \log P(M) \\
&= \mathrm{argmin}_M (-\log P(D|M)) +  (-\log P(M))
\end{align*}

Observe how the loss turns out to be just another name for the (minus) log-likelihood of the data (under the chosen model) and the regularization penalty is the log-prior of the model. For example, the familiar $\ell_2$-penalty is just the minus logarithm of the Gaussian prior on the parameters:

$$ -\log\left((2\pi)^{-m/2}e^{-\frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2}\right) = \mathrm{const} + \frac{1}{2\sigma^2}\Vert \mathbf{w} \Vert^2$$

Hence, another way to describe the difference between SVM and logistic regression (or any other model), is that these two postulate different probabilitic models for the data. In logistic regression the data likelihood is given via the Bernoulli distribution (with $p$=sigmoid), while the model prior is uniform (or simply ignored). In SVM the data likelihood is modeled via some $\mathrm{exp}(-\mathrm{hinge})$ distribution (not sure it even has a name, but I hope you get the idea that undoing the minus-logarithm would always bring you back to $P(D|M)$, up to a constant), and the model prior is the Gaussian.

In practice, the two models have different properties, of course. For example, SVM has sparse dual representations, which makes it possible to kernelize it efficiently. Logistic regression, on the other hand, is usually well-calibrated (which is not the case with SVM). Hence, you choose the model based on your needs (or, if you are unsure, on whatever cross-validation tells you).


Using vgg16 or inception with wights equals to None

The advantage of using a pre-trained model without loading the weights (which would mean you are only use the model, not a pre-trained version) is that you can easily use an existing model architecture and applying it to your problem. This can save you quite some time since you don't have to build the model architecture yourself in tensorflow/keras/pytorch and can go straight to applying the model on your data.


Need explanation of a matrix multiplication

Let's take it step by step.

$$
f^{(1)}(x)=h=W^Tx  \tag{1}
$$

$$
f^{(2)}(h)=h^Tw \tag{2}
$$

We substitute h.

$$
f(x)=(W^Tx)^Tw   \tag{3}
$$

To make it work, we'll make a little trick.

  The transpose of a transposed matrix is the original matrix.

$$
(A^T)^T = A
$$

We substitute w.

$$
f(x)=(W^Tx)^T (w^T)^T  \tag{4}
$$

Now we use following:

$$
(AB)^T=B^T A^T
$$

  The transpose of two matrices multiplied together is the same as the product of their transpose matrices in reverse order.

et voila

$$
f(x)=w^TW^Tx
$$


Incremental modelling of kmeans in pyspark

There's dedicated lib for incremental / online learning. It's called Creme. You can find it here. It has kmeans implemented, though I don't know how well it's suited for spark data sets.


customer segmentation with unbalanced data

Split your data into zero and non-zero valued to get like 8 clusters.

That is by far the best clustering you can find in your data because of the data. I doubt you can find much more than that. Have you tested if any of the "interesting" patterns you found with k-means are better than these?


Using the trainbr function for classification in Matlab

The trainbr mode uses the Bayesian regularization backpropagation. This method was presented in 1, which presents a regression problem with the loss function
$$ E_D = \sum_{i=1}^n (t_i - a_i)^2 $$
where $t_i$ is the target and $a_i$ is the network's response. The paper proposes to add a regularization term, leading to a loss function $F$ of the form
$$ F = \beta E_D + \alpha E_W $$
where $E_W$ is the square of the sums of all network weights, i.e. $E_W = \sum_{i,j} \| w_{ij} \|^2$.
The two parameters $\alpha$ and $\beta$ control the weighing of the two parts $E_D$ and $E_W$: For $\alpha \ll \beta$, the network will minimize the loss, without really trying to keep weights low. For $\alpha \gg \beta$, the network will minimize the weights, allowing for some more error. In reality, this means a large $\alpha$ will stop the network from overfitting, which leads to a better generalization at the cost of a larger training error.

The key to find a train a model which generalizes well, but still has a low error rate, is the right setting of $\alpha$ and $\beta$. This is achieved by treating them as random variables and finding an optimal setting, using the Bayesian methods presented in 2. (I won't talk about the details on that here, you can find that in the two linked papers.) Finally, the paper presents an algorithm, which calculates the optimal $\alpha$ and $\beta$ in each training iteration. This makes this algorithm generalize really well, especially in the presence of noisy input signals.

However, as described, the loss function is a weighted sum between the MSE ($E_D$) and the regularization term ($E_W$). So, in short: you can only use it with the MSE, and not with cross-entropy. You'll need a different training algorithm, you can find a list in the MATLAB documentation, here.

References:

  1 F. D. Foresee and M. T. Hagan: "Gauss-Newton Approximation to Bayesian Learning", in Proceedings of the 1997 International Joint Conference on Neural Networks, June 1997. DOI: 10.1109/ICNN.1997.614194. [Link to PDF].

  2 D. J. C. McKay: "Bayesian Interpolation", Neural Computation, May 1992, Vol. 4, No. 3, pp. 415-447. DOI: 10.1162/neco.1992.4.3.415. [Link to PDF].


Histogram is extremely skewed to the left

I think there are a few "easy wins" here.

You might add more bins - you are already using the bin setting. Just add something high like 100 or even 1000 to get a first feeling for the data
You can define the range of your bins. For example you could set the range with a list of two entries range = [0, 5000] as an additional parameter
You can consider not showing all data- maybe movies with a few ratings or movies with a lot are not what you focus on. You can use the range argument from point two to limit the plot to the region interesting for you.
You can try a cumulative and normed plot cumulative = True, normed = True this can be very useful to get a better estimate on statements like "80% of movies have less than 100 ratings" - for cumulative plots I recommend a very large number of bins.

I hope this helps and let me know if something is unclear


I am having too many features.

No, you don't :).

First of all, it is highly likely that not all of them are important for the prediction that you want to make.

I would highly recommend using a CART Random Forest for regression of the variable of interest. It literally requires minimal coding if you choose to do it in python using the RF algorithm from the sklearn package.

It's big advantage is that it is straightforward to use and understand and, moreover, it provides you with the learnt feature_importances_ of all inputs
after training, so that you can exclude the least important ones and speed up the inference/training in the future.

-EDIT-

To understand the difference between Classification and Regression Decision trees, check this helpful link.

The decision tree implementations for regression are commonly the C4.5, the C5.0 or the CART algorithm. The one that is used by sklearn is CART, please take a look at section 1.10.6 in this link.

A good example of how to use the sklearn Decision Tree for regression is this.


Support vector data description (SVDD) finds the smallest hypersphere that contains all samples, except for some outliers. One-class SVM (OC-SVM) separates the inliers from the outliers by finding a hyperplane of maximal distance from the origin.

If the kernel function has the property that $k(\mathbf{x}, \mathbf{x}) = 1 \quad \forall \mathbf{x} \in \mathbb{R}^d$, SVDD and OC-SVM learn identical decision functions. Many common kernels have this property, such as RBF, Laplacian and $\chi^2$.

SVDD and OC-SVM are also equivalent in the case that all samples lie on a hypersphere centered at the origin, and are are linearly separable from it.

See Lampert, C. H. (2009). Kernel methods in computer vision (Chapter 5) for more detailed descriptions of these models.


Demonstrating and visualizing cluster centroid updates by using the first four iterations in Python?

You can use the plt.scatter() and plt.subplots() to achieve this as follows:
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
data = make_blobs(n_samples=200, n_features=8,
                           centers=6, cluster_std=1.8,random_state=101)

fig, ax = plt.subplots(nrows=2, ncols=2,figsize=(10,10))

from sklearn.cluster import KMeans
c=d=0
for i in range(4):
    ax[c,d].title.set_text(f"{i+1} iteration points:")
    kmeans = KMeans(n_clusters=6,random_state=0,max_iter=i+1)
    kmeans.fit(data[0])
    centroids=kmeans.cluster_centers_
    ax[c,d].scatter(data[0][:,0],data[0][:,1],c=data[1],cmap='brg')
    ax[c,d].scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], s=200, c='black')
    d+=1
    if d==2:
        c+=1
        d=0

This will produce:


Difference between Word Embedding and Text Embedding

A Text embedding is a vector representation of a text. A trivial way to construct a text embedding is to average the word embeddings of each word in the text.
However using this method, you will lose contextual information.


An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation is exactly as you would do with linear layer.

Why don't we just call it linear layer, then?

At theory level, an embedding layer performs a matrix multiplication to the input. However, in practice, the coded implementation is slightly different. This is due to the fact that the input, as a category, is incoded in a one-hot way, and the matrix multiplication by a one-hot encoded vector is as easy as a look-up, so there is no need to multiply the whole matrix.


recommender systems : how to deal with items that change over time?

You will have look for incremental, online, or dynamic versions of classic recommender system algorithms. Those are the terms associated with changes over time.
Another option is to reinforcement learning. The reinforcement learning framework can also model changes over time. The most work has been done in multi-arm bandits for recommendations.


How much neural network theory required to design one?

Here is a good way to get going:

C++ or c#
spend a week learning vectors and matrices.
Be able to code a "class vector" and "class matrix"
Get intuition on how matrices represent weights between 2 layers
Learn the chain rule and the "tree diagram" when computing composite (nested) functions
Spend a week learning backprop. Be able to compute derivatives for all layers, using pen an paper a toy example with 4 layers, 3 neurons each.
be able to make code that reads a .txt file character by character.
Learn about one-hot, to describe your character to the network (it only understands vectors)  - or see how your byte-problem might be used instead of characters.
Learn about hard-max softmax and their derivatives,  tanh and sigmoid
code fwd prop
code bkprop
test if it can predict the next char
debug
debug
DEBUG
Gradient checking


correlation coefficient between weather data and power consumption

I am not very sure how effective NN would be for this problem. The way I see it is that you have 48 entries in a time series and now you are trying to predict the next 4. Keeping aside the correlations for a second, it seems like there is very little data to "learn" from. I mean the power consumption is a seasonal thing (much like temperature) but you have only 1 years data so it should be difficult for NN to capture this seasonality.

I would like to raise another point about the way you are using correlations. If you are taking correlations using the actual temperature value then I do not think that to be a good idea. Think of it in this way, I would be using a lot of power at both very high and very low temperature. So instead of using the raw temperature values, I should normalize them with respect to some "comfortable" temperature value by taking the absolute difference between the actual temperature and the "comfortable" temperature.

For instance, let us say that "comfortable" temperature is 25C, then both 5C and 45C would be normalized to 20 and this looks more plausible as my power consumption should be high in both the cases. This might also explain the poor correlation that you observe for 3 months of data. What should be the "comfortable" temperature is an entirely different story altogether.

Do let me know if this line of thought makes sense.


Cross validation for anomaly detection using autoencoder

Autoencoders are used to reduce the dimensionality of the feature space. They can capture nonlinearities that other dimensionality reduction teqniques like PCA can not. Autoencoders are build by training the model to reproduce the input. In this case you can split the data set into three:

training
cross validation
testing

Train you model using the training set, check the performance by looking at your loss for the cross validation set, make some changes, and repeat. After you have a model you are confident in, then using the test data set as the final word in its performance.

I think the main point is that your loss should have something to do with the difference between an element of the data set, and the encoded-then-decoded version of that element. Therefore you cross validate against a separate chunk of data as you tune the hyper parameters of the encoder.


How to perform model selection for One-Class Classification?

Have you read Janssens' dissertation "Outlier Selection and One-Class Classification"? He has a chapter on evaluation which may be of use. Have you thought about artificial generation of negative instances?

I had to deal with OCC evaluation awhile back and never came upon an entirely satisfactory solution. As I recall, the basic problem boils down to the fact that you don't really have labelled instances of the negative class. Without these instances there is no way of computing precision = TP/(TP+FP) which relies upon the count of incorrectly accepted negative instances (FP). This just leaves you with recall = TP/(TP + FN) as a computable metric - which by itself is next to useless in evaluating classifier performance i.e. it's trivial to write a classifier that get's 100% recall, just predict the positive class for every instance in your test data... you will correctly identify every true positive instance, but you will also incorrectly predict every negative instance as a positive instance. What is typically required is a balance or trade-off between precision and recall.

You didn't mention a particular problem domain in which you are working, but perhaps you can leverage this domain knowledge as a work-around to the problem of OCC evaluation. This particular problem I will relate is technically cast as a PU Learning problem (learning from Positive and Unlabelled instances). I'll try to describe some solutions that I explored while tackling this challenge - I'll give you fair warning that they are not entirely satisfactory solution, but I believe they are logically sound and somewhat defensible. From a purely pragmatic perspective, they are definitely better than the alternative.

Implicit Negative Class

Scenario: imagine a membership-based incentive program, in which people can pay an annual fee to become a member of Foobar Inc. Members, but not non-members, are entitled buy widgets at Foobar's retail outlets.

You are in charge of a marketing drive to increase Foobar's membership. You have limited funds and reach, so you need to carefully select people from the general population that you believe will most likely become members if you target them with a promotional campaign. In this scenario you have a database of known Foobar member's (the labelled, positive class data, which we label POS) and a database of known non-members from the general public. This is your unlabelled data that contains people who may or may not become members if targeted. All we have is POS instances and UL instances -- there are no labelled NEG instances.

Depending upon your particular problem, context, and domain, you may be able to transform the OCC/PU problem into a more classical problem with standard performance metrics. In the situation I just described, there are at least three ways of creating an implicit negative class:

Use the spy technique to generate "reliable" negative instances
that can be placed in your test set;
If historical campaign data is available you could identify those
individuals who were targeted multiple times for campaign
membership, but who rejected it every time -- these could be
"consistently negative" instances;
If your known membership pool consists of members of varying
value to the business, you could divide members into good/bad, and
use the bad instances as a proxy for NEG instances.

One final area you may want to explore is evaluation in collaborative filtering research. Collaborative filtering on unary responses (e.g., Facebook "like")

  can be considered the matrix completion analog of the
  positive-unlabelled (PU) learning problem in classification and
  regression modelling. (Aggarwal, 2016)

Perhaps this field has developed some novel approaches to OCC evaluation that you can leverage. Without knowing the details of your particular case, however, its hard to provide a more detailed solution. Please update this thread with your eventual solution -- there is so much more research that needs to be done in this particular problem domain.


Forecasting with a Machine Learning Algorithm

I'm afraid your data is probably too complex and specific for somebody else to understand exactly what's going on.

The only idea I can suggest is to try to analyze manually the errors that your model makes:

Are there any patterns, like a kind of errors which happens quite often? For instance a country which tends to be overestimated, a month which tends to be under-estimated, this kind of thing.
If yes try to investigate why: for instance is there some additional feature that could help the model make better predictions for these cases?


I am pretty sure that the SMOTE package in python can also be used for multi-class as well.
Just you can check its implementation for iris dataset-

http://contrib.scikit-learn.org/imbalanced-learn/stable/auto_examples/plot_ratio_usage.html
-Please correct me if I am wrong.


What does embedding mean in machine learning?

In the context of machine learning, an embedding is a low-dimensional, learned continuous vector representation of discrete variables into which you can translate high-dimensional vectors. Generally, embeddings make ML models more efficient and easier to work with, and can be used with other models as well.

Typically, when I stumble upon jargon I'm not familiar with I first turn to Google, and if it can't be found I ping my colleagues and data science forums.


If an exhaustive nonlinear scan is too expensive and a linear scan doesn't yield the best results then I suggest you try a stochastic nonlinear search i.e. a random search for hyperparameter optimization.

Scikit learn has a user friendly description in its user guide.

Here is a paper on random search for hyperparameter optimization.


sklearn random forest and fitting with continuous features

To understand how a random forest treats continuous data it is imperative to understand how a random forest works. At the base of the random forest algorithm lays a tree construction. The default in sklearn is to split a tree based on the Gini coefficient (see sklearn documentation). This type of tree algorithm is referred to as CART trees. You can change the criterion to entropy to select ID3 and C4.5 trees. Without going to deep into the maths, the tree algorithm will seek to split the tree based on a cutoff that leads to the lowest Gini coefficient.

The random forest algorithm will build a large number of deep trees on your data and average over all the trained trees to give you the final prediction.

Depending on your requirements in terms of data size and necessity for parallelization I can highly recommend H2O. It is an open source machine learning software suite with APIs in Python and R. Their random forest implementation is very fast and leads to models with a higher AUC (see this page for a good comparison between different ML libraries).


Feeding back hidden state manually in tf.nn.dynamic_rnn (Tensorflow)

There's a lot going on inside an LSTM, so it's easy to get confused. I think you are confusing the state and the weights:

The weights are trained at training time and are not updated during prediction. This is like the weights of any other neural network.
The state updates as the model moved "forward" through the text. It is essentially what enables the LSTM to keep track of where it currently is in a sequence.

The weights have a fixed size, of course, but when generating text the LSTM acts only on the last character and the current state, which is why the sequence length is set to 1 (and if you're only generating one text at a time, the batch length is also 1. you could increase the batch size to generate multiple texts at the same time).


To be clear, RNN are not necessary as there exist other options. However, RNN are useful since they implicitly deal with temporal relationships between inputs.

Regular neural networks are idempotent, meaning that the same input can be passed multiple times and the same output will be computed (obviously if the network is not updated in the meantime).

This idempotence means that regular neural nets cannot implicitly "see" temporal patterns. To do so, they would need to be provided with temporal information as you suggest. This is actually what transformers achieve! By providing temporal embeddings on top of the input, it always a regular neural net to learn from temporal information.


Its "Precision at 1", or how often the highest ranked document is relevant:

http://ir-ratio.blogspot.co.uk/2012/03/precision-at-1-and-reciprocal-rank.html

Suppose you are looking for items about monkeys. Your query engine queries documents for "monkeys" and ranks by relevance. If the highest ranked document is indeed about monkeys, then that's a win for your query algorithm. But if the highest ranked document is ranked 1 because it has the text "Enough of your monkey business" then its a loss, because that's not really about monkeys.

Repeat over a bunch of search terms. The Precision-at-one is then the number of wins over the total number of search terms tried.


Knowledge of statistics To be a data scientist

Very good question!

Statistics is the main core. Not only theoretical understanding but practical use of statistical concepts are very important (Applied Statistics). Moreover working with statistical Softwares to have some ideas how they work and what are usual input-outputs and visualizations.

More important is the main fields of Data Science namely Data Mining, Machine Learning and Pattern Recognition. They are all built on statistics (try to get a course on Statistical Learning Theory to get some ideas).

After statistics, computer knowledge in general is important. Get to know the concepts of programming (what is a good program and why, which language is proper for which task, what is algorithm and how you should evaluate, produce or read them, etc.). In Data Science, reporting the results are important. Get to know some visualization techniques/libraries which are handy and commonly used.

Then you come to the concept of Database and even more general "What is data in real-world?" Try to get your hands dirty with some SQL-like query languages on some toy examples to learn what the data is and how you load, transform or extract them.

The last but not least are Machine Learning/Data Mining/Pattern Recognition skills. I recommend a MSc. in this field (however it's not necessary but my personal experience as an interviewer shows that usually people with this background stand higher).

These are the main things came to my mind. Please note that

This is just my opinion and based on my experience.
There are more details that I skipped to keep the answer as generally valid as possible.

Good Luck!


Feature selection by overfitting a small sample size

Congratulations! You have suggested independently the Wrapper method for feature selection. Yes, you can use this method.
However, consider that the wrapper method is slow since you have to train a model for each iteration. More than that, feature selection is a NP-complete problem so don't expect the optimal subset.


How to sort numbers using Convolutional Neural Network?

I have a solution however I use a densely connected layer at the output to simplify the reshaping. If you can manipulate the sizes of this model such that you have 4 output parameters this should work as well.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv1D, MaxPooling1D, Reshape
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

Preparing the data

We will generate some random lists containing integers between [0,49], we will take a random permutation of the list and then take the first 4 values. We will then set our targets $y$ as the sorted rows of $x$.

import numpy as np

n = 100000
x_train = np.zeros((n,4))
for i in range(n):
    x_train[i,:] = np.random.permutation(50)[0:4]

x_train = x_train.reshape(n, 4, 1)
y_train = np.sort(x_train, axis=1).reshape(n, 4,)

n = 1000
x_test = np.zeros((n,4))
for i in range(n):
    x_test[i,:] = np.random.permutation(50)[0:4]

x_test = x_test.reshape(n, 4, 1)
y_test = np.sort(x_test, axis=1).reshape(n, 4,)

print(x_test[0][0].T)
print(y_test[0])

  [ 44.  36.  13.   0.]  [  0.  13.  36.  44.]

The model

I tried different combinations of parameters. This worked out not bad.

input_shape = (4,1)

model = Sequential()
model.add(Conv1D(32, kernel_size=(2),
                 activation='relu',
                 input_shape=input_shape,
                 padding='same'))

model.add(Conv1D(64, (2), activation='relu', padding='same'))
model.add(MaxPooling1D(pool_size=(2)))
model.add(Reshape((64,2)))

model.add(Conv1D(32, (2), activation='relu', padding='same'))
model.add(MaxPooling1D(pool_size=(2)))

model.add(Flatten())
model.add(Dense(4))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

  Epoch 10/10  100000/100000 [==============================] - 6s
  56us/step - loss: 0.9061 - acc: 0.9973 - val_loss: 0.5302 - val_acc:
  0.9950

Results

So for a new list of values, I get the predicted output. Then I determine which value in the original list is closest to each of these and replace them. I could have just rounded the predicted values, however this caused so +/-1 errors due to rounding the wrong way.

test_list = [1,45,3,18]
pred = model.predict(np.asarray(test_list).reshape(1,4,1))
print(test_list)
print(pred)

print([np.asarray(test_list).reshape(4,)[np.abs(np.asarray(test_list).reshape(4,) - i).argmin()] for i in list(pred[0])])

  [1, 45, 3, 18]  [[  0.87599814   3.43058085  17.36335754  45.21624374]]
  [1, 3, 18, 45]

And for the sequence you suggested as a test case

test_list = [5,3,6,2]
pred = model.predict(np.asarray(test_list).reshape(1,4,1))
print(test_list)
print(pred)

print([np.asarray(test_list).reshape(4,)[np.abs(np.asarray(test_list).reshape(4,) - i).argmin()] for i in list(pred[0])])

  [5, 3, 6, 2]  [[ 1.85080266  2.95598722  4.92955017  5.88561296]] [2,
  3, 5, 6]


Exception: Data must be 1-dimensional appears when trying to make a submission

Python tells you that the data you give for the column "predictions" is not 1-dimensional (i.e. it's not a flat list). And indeed preds is not a 1-dimensional array, what you want to give is the corresponding labels that you collected in predictions.

So Instead of this:

results=pd.DataFrame({"image ID":filenames,
                  "Predictions":preds})

you probably want:

results=pd.DataFrame({"image ID":filenames,
                  "Predictions":predictions})


Python model persistence without pickling

One option is to export the model to Predictive Model Markup Language (PMML) format.

In scikit-learn, sklearn2pmml can be used.


Is there any option to represent 160 discrete variables each one with their own color in scatter plot?

The number of colours you can use in a scatter plot will vary depending on what software or programming language you are using.

The main issue is that scatter plots might not be the best option to use if you want to distinguish 160 different variables, particularly if you are only distinguishing them by colour.

If you can group the variables you might be able to make it work with less colours, or consider using a different type of graph.


Tuning the model parameters vs the parameter of optimizer for Deep Neural Networks?

The short answer, is that you want the fastest way to reach the performance you expect/desire. This would mean first playing with some hyperparameters like learning rate, initialisation strategy, trying different optimizers, etc.
Having said that, if you managed to write your training process such that it is trivial to swap out the model completely for another one, you can also give that a go.

If you have a model that is known to work well for a given type of task, but is really under-performing (e.g. not converging or a large difference to results in the literature), then I would first check for some common issues in the training process:

data quality: are the inputs scaled correctly? Should you add some
data volume: if you don't have many samples, can you get more or use augmentations?
loss function: are the outputs of the model being correctly consumed and the losses correctly computed? are you even using the right loss formulation?

This is essentially debugging your training setup.
I would convince myself as far as possible that none of these exist before building a different architecture or a radically different approach. The main reason for this, is that I assume you did some research into the problem and the type of models that are known to perform well - so if this doesn't work, what makes you think another architecture will?  :)


Dwell Time computation from Real-Time user location

We have a collection of real time user locations from our application. Each record contains longitude, latitude and the time at which the geo location was captured. How can I compute the dwell time at an user's stationary place from this data?

Things to consider when collecting the data:

If you are using a Library such as: react-native-mauron85-background-geolocation and the Ionic Framework you would set the stationaryRadius to define an area within which the device reporting its location is considered to be stationary.

When there are obstructions and reflecting surfaces multi-path reception is likely. The signals arrive at the receiver at slightly different times and, depending on which signal the receiver chooses to believe for any given sample, the distance from the satellite also differs. Multiply this behavior by several satellites, then factor it into the overall position calculation, and the net result is that the receiver perceives abrupt shifts in position. Sometimes these can be significant - say 40 or 50 feet.

Higher level software in the receiver tries to smooth this out. Generally speaking, it does a better job of this when the tracker is moving. This is because your true change in position per unit time is much larger than the random shifts caused by multi-path reception.

You can average the reported position over a period of time to get a circle within which the device is likely to be. Using secondary information like IP address and WiFi positioning can provide additional information to narrow down the guess faster. See also: Wireless Technologies.

Here is an example that shows you where you are using Mozilla's Geolocation API, click on "Geolocation Live Example" and give permission to share your location (not with us at SE, with Mozilla).
Once you have fairly accurate data it's simply a matter of defining a radius and minimum dwell time (which would be a moving average).

If they are moving fast then a shorter stay is a dwell, while at a slower speed a dwell wouldn't occur until a longer period of time.

Advanced mathematical modeling is necessary to analyze a better result and not get confused by meandering and backtracking.


I guess you are using the fnn package.

attr is a list of attributes which can be used for both nn.index and nn.dist. In this case, you are using index.

So, index returns an n x k matrix for the nearest neighbor indice.

And the definition of the nearest neighbor index is:

  The nearest neighbor index is expressed as the ratio of the observed
  distance divided by the expected distance.

Definition reference


Differentiate between positive and negative clusters

Maybe you don't have a positive and a negative class.
Your input are word vectors. Unless you trained your word vectors before with explicit positive and negative labels, it is very unlikely that your KMeans learned that difference.
If you used pre-trained word vectors, your KMeans could have learned an arbitrary difference between cluster 0 and cluster 1. Maybe it learned which reviews are from males and which from females, maybe which have the word "parachute" and which don't have the word "parachute", the options are endless.
What you can do, is access which labels your KMeans learned (model.labels_) and filter your input X per cluster. Then, count the occurence of each word in each cluster and order which words happen the most in each of them. This might help you understand the difference between cluster 0 and cluster 1.
Note: if the top words you get are words like: a, the, of, if, etc. Use a stop-word list, or filter those word with a max document frequency threshold.


practical improvements worth trying over plain LSTM in text classification?

A properly written LSTM network is quite powerful for NLP - why do you think that "combining" this with something else is the answer???

The project you're describing should be pretty easy to implement. The only barrier you have here is how much labeled data you can collect. Personally, I would focus on getting hundreds of thousands of examples, in a balanced dataset, and then feed that (with NLP approaches) into an LSTM network. You should be able to generate very good accuracy rates in a reasonable amount of time.


How to test dev set on Time Series data via forecasting

I feel your frustration. I am not sure if this is a complete answer, but I’ll try to add some value here.

Disclaimer: I am not familiar with the neural nets you mentioned but have used other times series nns. So my answers will be general.

First, there is nothing specifically wrong about retraining every day unless you are doing so at the expense of the number of training epochs. For example, you could conceivably need to train for 20k epochs to get the best predictive power. If this takes 20 hours, it may not feasible. So you may be tempted to train for 5000 epochs. This is probably a reason not to retrain every day. My recommendation would be to invest in more epochs and retrain less frequently. Hopefully the resulting model is more generalized and powerful in the long run.

If you DO still want to train every day, I would load the weights and biases from the previous training session and continue training for a smaller number number of epochs. Start with as many epochs as you can to produce the first model. Then add the additional row and continue training for a manageable number of epochs. TBH, one extra row should not really change your model weights and biases significantly unless the new y is pretty extreme. In fact, I doubt it would even be obvious.

Now, why does your accuracy drop with the new data? Several ideas:

Just stating the obvious. In case it’s not. You will need many training epochs. Each application is different, but sometimes people only train for a few epochs and don’t understand why the model sucks. You might need 200 epochs, you might need  2000000.
You are overfitting and the addition of the new data is generalizing the model a little more each day. Doubtful, but possible.
You are reinitializing your model with random weights and biases and do not have a seed set. So your model could be finding different local minimums each time you train. I would be surprised if it was always in the negative, but it may be that you have just flipped heads 25 times in a row. I would set a seed when initialize to take this out of the picture.

Hopefully there’s something of use in here.


Split large documents into many smaller ones?

I think it should be easy to detect a language based on a standard vocabulary (bag of words).

However, I would split the articles into small pieces (maybe not 100 but a bit more words, eg. 200-500), so you can train and test your model without problems.


LSTM doesnt find finer dependencies than the Random Forest model

Some (possibly stupid) ideas for combining the two models:

Make an average of the two forecasts
Instead of a naive average you use LSTM for estimating the coarse daily trend and the RF for the fine details, maybe like so:

a. Smooth (maybe with a loess or a kernel) the LSTM to get the daily trend

b. For each day, subtract the mean volume call from the prediction of the RF, to get the fine details

c. Add the forecast of two models

General suggestions for improving your model:

Instead of training a model on the whole series of call volume, you might be better off feeding your model one day at a time, e.g. you cut the time series between 8 am and 5 pm every day. After all, you do not want your model to spend parameters on learning that no calls are made at night or during the weekend.
Add the day of the week as a parameter in the model, as well as some information on the call volume in the past (like previous day) if you think that there might be some time correlation between consecutive days.
If you think that the day of the week has a strong effect on the call volume (e.g. if each day has a special, recurring pattern) then you might consider creating a separate model for each day of the week.


Why does a randomly initialised convolution kernel correspond to an edge detector?

Take this with a grain of salt, but I think this is simply not true. You can evaluate it with the code I just wrote:

Only 2, probably 3 of the 25 look like edge filters to me. The result of an edge filter looks like this:


To get equivalent results :
im_torch = im.reshape((1, 1, 3, 3))
filt_torch = filt.reshape((1, 1, 3, 3))

out = torch.nn.functional.conv2d(filt_torch, im_torch, stride=1, padding=0)
print(out)

import scipy.signal

out_scipy = scipy.signal.convolve2d(im.detach().numpy(), filt.detach().numpy()[::-1, ::-1], 'valid')
print(out_scipy)

There was a shaping issue (https://pytorch.org/docs/stable/nn.functional.html) and the application of the filter in torch differs a bit with how it is handled in scipy : https://discuss.pytorch.org/t/functional-conv2d-produces-different-results-vs-scipy-convolved2d/17762


You certainly need to add at least one other variable representing the time of year, because from your graph it's clear that the fare can't be predicted accurately using only the time until departure: for the same day you can have many points representing different fares. That makes sense, since the fares are going to be very different depending if the flight occurs during holiday season or not. There might also be other variables of interest but since it's always the same flight I can't think of any.

I would suggest plotting a graph where x is the time of year and the time before departure is represented for instance with colour or gradient, that should make things clearer.


What does "exaggeration" mean in the context of Boosting?

Not sure, but "exaggeration" could be an other way to talk about "overfitting".

Boosting are sequential model: each time you build a new tree, it will use the results of the precedent ones, and focus on residuals (where the precedent trees dont perform well). For this, the model exagerate the weights of the precedent mistakes.

If you build too much trees, the last ones will learn on noise, because
residuals will contains much noise than information. You will have an excellent error rate ("exaggerated error rates") for your train dataset, but if you apply your model on a new dataset, the error will be lower.


Does the network learn based on previous training or does it restart? Matlab, neuralnetworks

It trains again based on what it learned the first time you did OP = train(OP,inputsVals,targetVals). More generally, train uses your network's weights, i.e. it does not initialize the weights. The weight initialization happens in feedforwardnet.

Example:

% To generate reproducible results
% http://stackoverflow.com/a/7797635/395857
rng(1234,'twister')

% Prepare input and target vectors
[x,t] = simplefit_dataset;

% Create ANN
net = feedforwardnet(10);

% Loop to see where train() initializes the weights
for i = 1:10

    % Learn
    net.trainParam.epochs = 1;
    net = train(net,x,t);

    % Score
    y = net(x);
    perf = perform(net,y,t)
end

yields

perf =

    0.4825

perf =

    0.0093

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0034

perf =

    0.0028


With only 5 samples per class you probably should not be using any Neural Network. If you can find a way to summarize your long time-series into a couple of numerical features, then you can try with a simple Logistic Regression model. Exploring the data is the key here, as well as understanding the problem domain. What is the physics of the process for example, or how would experts describe the requirements to get a particular score.

Even so I would try to collect some 10x more data.


Why and how to match variables in logistic regression?

It appears (if I understand you correctly) that you want to model the causal effect of some confounders $X$ on some outcome $y$ by using Logit. If this is correct, and if you know the outcome $y$, you should be fine with using just Logit, because in this case (and if you are able to contol for all relevant confounders $X$), you can identify the marginal effect of $x$ on $y$.

I guess you refer to "Propensity Score Matching" (PSM) in your question. This technique is used to "predict" cases in which you do not know for sure if some $i$ would have received treatment ($y=1$ or $y=0$) and you try to "predict" this outcome. In other words: The propensity score is the conditional (predicted) probability of receiving treatment given some $x$.

However, in case you observe $y$, and in case you don't have rasons to believ that there is a bias in $y$, you should be fine with a normal Logit. Here it would be really important for you to clarify what your problem actually is (this is not clear from your question). So if treatment ($y$) is non-random, you may think about "correcting" this bias using PSM. If not, you can go with normal Logit.

PSM is used in econometrics and related fields. So there is no off the shelf Python implementation to my best knowledge. However, you may have a look at this post. Or if you don't mind using R, have a look at this.

Anyway: PSM has it's pros and cons. It also is a little outdated since matching has made huge progress in recent years. So the most important thing for you to do is - as it appears to me - to get a good idea if you really need to "adjust" the treated/non-treated $y$ (so if you have reasons to believe that the data generating process is biased or non-random).


Once model has been trained with Keras

It can be exported to TensorFlow model OR
With tf.keras it can be served as http service

Examples :

https://towardsdatascience.com/deploying-keras-models-using-tensorflow-serving-and-flask-508ba00f1037

https://medium.com/tensorflow/training-and-serving-ml-models-with-tf-keras-fd975cc0fa27

https://medium.com/@mr.acle/exporting-deep-learning-models-from-keras-to-tensorflow-serving-7d4a6e49ce3


Optimization based on validation and not training

Welcome to the site! The optimization function in your model creation is not the best place to do what you want. What you should be doing is:

Create the model
Run as many epochs as you want during model training
During the epochs create Keras callbacks that save the best model based on validation accuracy (as opposed to other metrics)

That will allow you to still create your model in an efficient way while ending up with an epoch training process that creates the model with the highest validation accuracy.


Create a binary-classification dataset (python: sklearn.datasets.make_classification)

Let's build some artificial data. There are many ways to do this. I usually always prefer to write my own little script that way I can better tailor the data according to my needs.

Let us first go through some basics about data. A lot of the time in nature you will find Gaussian distributions especially when discussing characteristics such as height, skin tone, weight, etc. Let us take advantage of this fact.

According to this article I found some 'optimum' ranges for cucumbers which we will use for this example dataset.

Temperature: normally distributed, mean 14 and variance 3. If a value falls outside the range $[10,18]$ then it is said to be not edible.
Color: we will set the color to be 80% of the time green (edible). 10% of the time yellow and 10% of the time purple (not edible).
Moisture: normally distributed, mean 96, variance 2. If the moisture is outside the range $[94, 98]$ then the cucumber is not edible.

Building the dataset

We will build the dataset in a few different ways so you can see how the code can be simplified.

A very verbose example

This example will create the desired dataset but the code is very verbose.

import numpy as np

# Number of samples
n = 100

data = []
for i in range(n):
    temp = {}

    # Get a random normally distributed temperature mean=14 and variance=3
    temp.update({'temperature': np.random.normal(14, 3)})

    # Get a color with 80% probability green, 10% probability yellow
    # and 10% probability purple
    color = 'green'
    color_random_value = np.random.randint(0,10)
    if color_random_value == 8:
        color = 'yellow'
    elif color_random_value == 9:
        color = 'purple'
    temp.update({'color': color})

    # Get a random normally distributed moisture mean=96 and variance=2
    temp.update({'moisture': np.random.normal(96, 2)})

    # Verify if the instance is edible (label=0) or not (label=1)
    label = 0
    if temp['temperature'] < 10 or temp['temperature'] > 18:
        label = 1
    elif temp['color'] != 'green':
        label = 1
    elif temp['moisture'] < 94 or temp['moisture'] > 98:
        label = 1
    temp.update({'label': label})

    data.append(temp)

Then we can put this data into a pandas DataFrame as

df = pd.DataFrame(data=data)
df.head()

A cleaner example

import numpy as np

n = 100
data = {'temperature': np.random.normal(14, 3, n),
        'moisture': np.random.normal(96, 2, n),
        'color': np.random.choice(['green', 'yellow', 'purple'],
                                  size=100,
                                  p=[0.8, 0.1, 0.1])}
df = pd.DataFrame(data=data)

Then we will get the labels from our DataFrame

def get_label(color, moisture, temperature):
    if temperature < 10 or temperature > 18:
        return 1
    elif color != 'green':
        return 1
    elif moisture < 94 or moisture > 98:
        return 1
    return 0

df['label'] = df.apply(lambda row: get_label(row['color'],
                                             row['moisture'],
                                             row['temperature']), axis=1)

Getting the data ready for applying a classifier

One of our columns is a categorical value, this needs to be converted to a numerical value to be of use by us.

This can be achieved using

df['color_codes'] =df['color'].astype('category').cat.codes

Now we are ready to try some algorithms out and see what we get.

Visualizing the data

The first important step is to get a feel for your data such that we can try and decide what is the best algorithm based on its structure.

I prefer to work with numpy arrays personally so I will convert them

X = np.asarray(df[['color_codes', 'moisture', 'temperature']])
y = np.asarray(df['label'])

Let's plot the data in 3D

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.scatter(X[:,0], X[:,1], X[:,2], c=y)

ax.set_xlabel('Color')
ax.set_ylabel('Moisture')
ax.set_zlabel('Temperature')

plt.show()

The blue dots are the edible cucumber and the yellow dots are not edible. We can see that this data is not linearly separable so we should expect any linear classifier to be quite poor here. I would presume that random forests would be the best for this data source.

Let's split the data into a training and testing set

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

Let's see the distribution of the two different classes in both the training set and testing set

import matplotlib.pyplot as plt
%matplotlib inline

n_classes = 2

training_counts = [None] * n_classes
testing_counts = [None] * n_classes
for i in range(n_classes):
    training_counts[i] = len(y_train[y_train == i])/len(y_train)
    testing_counts[i] = len(y_test[y_test == i])/len(y_test)

# the histogram of the data
train_bar = plt.bar(np.arange(n_classes)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training')
test_bar = plt.bar(np.arange(n_classes)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing')

plt.xlabel('Labels')
plt.xticks((0,1))
plt.ylabel('Count (%)')
plt.title('Label distribution in the training and test set')
plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2)
plt.grid(True)
plt.show()

Linear classifier

from sklearn import linear_model
clf = linear_model.SGDClassifier(max_iter=1000)
clf.fit(X_train, y_train)
clf.score(X_test, y_test)

  0.54545454545454541

Support Vector Classifier

from sklearn.svm import SVC
clf = SVC()
clf.fit(X_train, y_train)
clf.score(X_test, y_test)

  0.72727272727272729

K-Nearest Neighbors

from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X_train, y_train)
neigh.score(X_test, y_test)

  0.66666666666666663

Random Forests

from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_estimators = 100)
forest.fit(X_train, y_train)
print('Score: ', forest.score(X_test, y_test))
predictions = forest.predict(X_test)

  Score:  1.0

Well we got a perfect score. As expected this data structure is really best suited for the Random Forests classifier.


This paper seems interesting:
"Learning Deep Feature Fusion for Group Images Classification"


When inputting image rgb values to MLP, should I divide by 255?

The component values are often stored as integer numbers in the range 0 to 255, the range that a single 8-bit byte can offer,

Yes, If you divide by 255  the range can be described with a 0.0-1.0 where 0.0 means 0 (0x00) and 1.0 means 255 (0xFF).
Normalization will help you to remove distortions caused by lights and shadows in an image.

Refer to this Normalize RGB


Except for the OCR part, the right bundle would be pandas and sklearn.

You can check this ipython notebook which uses TfidfVectorizer and SVC Classifier.

This classifier can make one-vs-one or one-vs-the-rest multiclass predictions, and if you use the predict_proba method instead of predict, you would have the confidence level of each category.

If you're looking for performances and you don't need prediction confidence levels, you should use LinearSVC which is way faster.

Sklearn is very well documented and you will find everything you need for text classification.


assuming your products table is called products the following sql statement would achieve this

SELECT DISTINCT id, category as name, 1 as flag_for_ca FROM products
UNION ALL
SELECT DISTINCT id, prodct as name, 0 as flag_for_ca FROM products


Andrew Ng's course on Machine Learning on Coursera is terrific. I'm doing it right now, and he has wonderfully gauged student understanding. He helps you along and doesn't skip lots of steps. It apparently is not his goal to convince you of how smart he is - he's more concerned with explaining everything clearly. Highly recommended!


Polynomial regression (for nth degree polynomial) in statistics is a special case of linear regression. Lets give an example for square function:

1. y = w*x

  This is linear in terms of both weight (w) and data (x).

2. y = w*(x^2)    OR        y = w*z ; where z = x^2

  This is still linear in terms of weight (w) and still treated as a linear regression for the transformed data (z). While the modeled relationship between y and x is certainly non-linear.

As you can notice above: The commonality in (1) and (2) is the linearity with the weight/ coefficient of linear regression.


How are ANN's, RNN's related to logistic regression and CRF's?

No, it is not possible.

There is a fundamental difference between these graphs and neural networks, despite both being represented by circles and lines/arrows.

These graphs (Probabilistic Graphical Models, PGMs) represent random variables $X=\{X_1,X_2,..,X_n\}$ (circles) and their statistical dependence (lines or arrows). They together define a structure for joint distribution $P_X(\boldsymbol{x})$; i.e., a PGM factorizes $P_X(\boldsymbol{x})$. As
a reminder, each data point $\boldsymbol{x}=(x_1,..,x_n)$ is a sample from $P_X$. However, a neural network represents computational units (circles) and flow of data (arrows). For example, node $x_1$ connected to node $y$ with weight $w_1$ could mean $y=\sigma(w_1x_1+...)$. They together define a function $f(\boldsymbol{x};W)$.

To illustrate PGMs, suppose random variables $X_1$ and $X_2$ are features and $C$ is label. A data point $\boldsymbol{x}=(x_1, x_2, c)$ is a sample from distribution $P_X$. Naive Bayes assumes features $X_1$ and $X_2$ are statistically independent given label $C$, thus factorizes $P_X(x_1, x_2, c)$ as $P(x_1|c)P(x_2|c)P(c)$.

In some occasions, neural networks and PGMs can become related, although not through their circle-line representations. For example, neural networks can be used to approximate some factors of $P_X(\boldsymbol{x})$ like $P_X(x_1|c)$ with function $f(x_1,c;W)$. As another example, we can treat weights of a neural network as random variables and define a PGM over weights.


Convolution neural network with 11 million parameters unable to overfit on 100 image samples

You use a sigmoid activation in the last layer, which restricts your possible outputs to be between 0 and 1. This is typically used for binary classification, but it seems like you are not doing classification at all. If your image has values constrained to the range 0-1 then this is likely fine. Otherwise, this could be the source of your problem.

I suggest removing the sigmoid activation in the final convolution and leaving it with no activation.


How to shuffle only a fraction of a column in a Pandas dataframe?

I don't think there is any idiomatic way of doing this since it's quite unusual operation, normally the whole row or column should be shuffled. What you are doing looks like a good approach.

The error SettingWithCopyWarning you get is a common warning that you could be operating on a copy of the original data and not a view (the origianl). For more information I would recommend checking the answers here: https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas.

To avoid the error and make the code more compact you could do it as follows:

import random

fraction = 0.4
n_rows = len(df)
n_shuffle=int(n_rows*fraction)

pick_rows = random.sample(range(1, n_rows), n_shuffle)

df.loc[pick_rows, 'L2'] = np.random.permutation(df.loc[pick_rows, 'L2'])

Note that the use of loc here will make sure that no copy is created and everything is done on the origianl dataframe (i.e. this will not give the SettingWithCopyWarning warning).


Is rolling of biased dice random phenomenon?

Yes, you are correct. Rolling of dice is (almost always) a random phenomenon. In case of biased v unbiased dice, the difference is solely in the probability distributions. For unbiased the distribution must be uniform, for biased it can be anything we want.
The only case where biased dice stop being random phenomena is when we load them in such a way that they always produce a predetermined number (or a specific sequence).


GraphViz not working when imported inside PydotPlus (`GraphViz's executables not found`)

See: Graphviz's executables are not found (Python 3.4) and graphviz package doesn't add executable to PATH on windows #1666 and Problem with graphviz #1357 - it's a reoccurring problem (for that program) with the PATH environment variable settings. Installing particular versions, or in a particular order, or manually adding a PATH fixes the problem.

It's best if the Package sets the PATH correctly and removes it when you uninstall the Package (so you don't get too long a PATH - which usually won't happen). Setting it manually prevents future breakage and forces it to work, but you need to manually remove the extra text if you uninstall the Package.

Here's the advice from those three links:

pip install graphviz
conda install graphviz

or

You need to run

conda install python-graphviz

instead of

pip install graphviz

to get these bindings, which also work with conda's Graphviz package.

or

Download and install graphviz-2.38.msi (use the newest version) from
https://graphviz.gitlab.io/_pages/Download/Download_windows.html
Set the path variable

        (a) Control Panel > System and Security > System > Advanced System Settings > Environment Variables > Path > Edit

        (b) add 'C:\Program Files (x86)\Graphviz2.38\bin'


Understanding Exclusive-OR predictions in Elman network

I have watched This video and then realized that i missed the way the data is represented. So, in elman networks, the data is represented by 2-bit inputs at a time followed by 1-bit output. So, the first element is always unpredictable(so, 50% chance) and so is the 2nd term, but, once we get both 1st and 2nd term, we can for sure predict the third element as its a XOR gate.


Machine learning dataframe dimension concept vs NumPy dimension

The dataframe case refers to the linear algebraic notion of a dimension.  In the NumPy context, it just means the number of axes or rank.


How to visualize Ensemble Models ( Random Forest) with 1000 estimators

You can try SHAP which visually explains the output of (many) machine learning model(s) including LightGBM and XGBoost.

However, please note that it will not give you the entire Ensemble Model (Trees) as picture.
Further note that it doesn't work for RandomForest


Doing that is a very good idea. The problem is that doing that is very hard.
Feature selection is a NP-complete problem.
The practical meaning as that we don't know any fast algorithm that can select only the needed feature.

In the other direction, omitting features that don't have mutual information (MI) with the concept might cause you to throw the features you need most.
There are cases in which a single feature is useless but given more features it becomes important.

Consider a concept which is the XOR of some features. Given all the features, the concept is totally predictable. Given one of them, you have 0 MI.

A more real life example is of age at death. Birth date and death date give you the age. One of them will have very low correlation (due to increase in life expectancy).

In practice, omitting features with low MI is OK. Many learning algorithms are using MI so they won't be able to use the omitted variables anyway.
As for the selection itself, there are many algorithms, usually heuristics or approximation algorithm that are quite handy.


Attributes extraction from unstructured product descriptions

Left you a quick response on SO. The gist is that you can collect a lot of information from electronics shops and manufacturers' web sites, and lots you can annotate manually. If your goal is to only get training data, that's all you need:

My answer form the cross-post:
"Having developed a commercial analyzer of this kind, I can tell you that there is no easy solution for this problem. But there are multiple shortcuts, especially if your domain is limited to cameras/electronics.

Firstly, you should look at more sites. Many have product brand annotated in the page (proper html annotations, bold font, all caps in the beginning of the name). Some sites have entire pages with brand selectors for search purposes. This way you can create a pretty good starter dictionary of brand names. Same with product line names and even with models. Alphanumeric models can be extracted in bulk by regular expressions and filtered pretty quickly.

There are plenty of other tricks, but I'll try to be brief. Just a piece of advice here: there is always a trade-off between manual work and algorithms. Always keep in mind that both approaches can be mixed and both have return-on-invested-time curves, which people tend to forget. If your goal is not to create an automatic algorithm to extract product brands and models, this problem should have limited time budget in your plan. You can realistically create a dictionary of 1000 brands in a day, and for decent performance on known data source of electronic goods (we are not talking Amazon here or are we?) a dictionary of 4000 brands may be all you need for your work. So do the math before you invest weeks into the latest neural network named entity recognizer."


Convolution operator yields negative index of matrix

There are two approaches that can be taken:

Only use valid indices. Matrix C will then be smaller than matrix I, in your example it would be a 7x7 matrix (9 - 3 + 1 = 7). You may see this in neural network libraries as a convolution working with "valid" border.
Use synthetic values, usually just 0, for out-of-bounds indices in matrix I in order to calculate C. This produces the same size output as input, so some neural network libraries note this with "same" border.

Your formula suggests "same" border is being used*, because it would be unusual to work with a matrix C starting from an offset corner. If you do use "same" borders, it is important to normalise your input image data pixels to mean 0, otherwise the synthetic border will appear as a strong edge to the kernels.

If you are developing your own CNN code, I would suggest using "valid" border mode, because it is simpler, and a more common approach. It will work just fine for image classification tasks.

* Actually I think the formula you have posted is over-simplified or even wrong. I would expect to see the term $I(x+a,y+b)$ so that the top left index of C stays as (1,1) for a "valid" border. For a "same" border, I would expect to see a centered kernel.


Autoencoder: using cosine distance as loss function

Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being cosine proximity = -1*(Cosine Similarity) of the two vectors. This is done to keep in line with loss functions being minimized in Gradient Descent.

To elaborate,
Higher the angle between x_pred and x_true. lower is the cosine value. This value approaches 0 as x_pred and x_true become orthogonal.
And if the angle between x_true and x_pred is small the value approaches 1.

So it logically follows that the cosine proximity loss value approaches -1 as the model converges.

As for the usage, I would recommend using this loss only when the magnitude of the generated vectors isn't as important as the angle between them. And that really depends on what you are trying to solve.


(1) Data quality. The single best way to improve your accuracy. Garbage in garbage out. You already said your data is suspect. Some data was mis-classified; data only has single label, when multiple labels are possible. This is the biggie - this will improve your accuracy more than any other technique: improve the quality of your training data. One way to go about this. Recruit expert labellers (these could be workers in your own company who do this task, or they could be external people, e.g., mturkers who have been trained to do this labelling task). In general, the more labellers, the better. At a minimum I would have 3 labellers, label all n emails in your train/test set, using either majority label wins, or require all-three consensus depending on your confidence needs. You will want to measure your inter-rater reliability to ensure the integrity of your data using Cohen's kappa or similar. Importantly, this data will become your gold standard data. That is to say the average agreement upon labels provided by your original labellers is the ground truth to which your algorithm attempts to approach.

Obviously, this approach to data quality is costly in terms of resources (time to label all the emails; cost of hiring mturkers, etc.). But depending upon your specific needs it needn't be... basically it depends upon the needed level of "expertise" to do the task. Sometimes, it is safe to consider you (the researcher/developer) as the "expert" (e.g., you yourself said that some emails were mis-classified... how do you know that? You must have some sort of expert knowledge.). However, using a single labeller for the task makes your data less reliable due to individual processing differences introducing a bias into the data.

(2) Including stop words. Yes, you read that correct including stop words. Grammatical function words (e.g., the, and, of, etc.) can often be highly indicative of specific text types or genres. It is premature to eliminate stop words without first ensuring that they are an uninformative feature. Similarly, try including punctuation, try including capitalization features. Time and time again, I see people blindly pre-processing text without empirically determining whether these features are significant predictors of class.

(3) Capture spelling variation. As the OP stated, the original texts have many "spelling mistakes." You can improve accuracy by reducing this variance - basically map variants to their canonical form before extracting your word features (e.g., "colour" -> "color"). I often rely on external corpora to rapidly create these spelling maps.

(4) Balance your training data - The OP mentions that his classes are severely imbalanced. Try over- and/or under-sampling to balance your classes.

(5) Describe your problem. It is also important to understand what the underlying purpose of your classifier. If your classifier is intended to run on historical data, perhaps to flag incorrectly human-classified emails, then you would probably want to keep auto-response and correspondence thread info. On the other hand, if the goal is to classify new customer emails so that they get properly routed, for example, then you probably want to remove these correspondence features from your training and test data.


Computation of a column-stochastic matrix with target row sums

This problem, being more of a combinatorial rather than statistical nature, can be though of as a max-flow problem in a bipartite network. Each column of your matrix corresponds to a "source node" in the left part of the bipartite graph $BP(L, R)$; each row of the matrix corresponds to a "destination node" in the right part $R$. Capacity of each source node is the corresponding column's sum; similarly, capacities are defined for the destinations via row sums. Value $a_{ij}$ in the matrix describes flow of mass from $j$'th source to $i$'th destination along an existing edge $(j \to i)$ of $BP$. Your initially given 0-1 matrix describes which edges are present (e.g., the matrix in your example prescribes the edge $(3, 2)$ to be absent from the network, along with all self-loops). If you denote $F$ to be the sum of all column sums (or, alternatively, the total capacity of all sources), then your problem narrows down to finding a max-flow from $L$ to $R$ in $BP$ and checking whether the volume of the discovered flow equals $F$.

Note 1 (Feasibility): Before plunging into finding best flows, you may want to check your row sums: a necessary condition for your problem to have a solution is that the sum of all column sums (which is $N$ in your particular case of a column-stochastic matrix) should be equal the sum of row sums, as both numbers describe the same -- the sum of all elements of the target matrix. (For example, for $N = 3$ and the vector of row sums from your example, the problem is unfeasible.)

Note 2 (Upper bound): Since $a_{ij}$ describes the flow from $j$'th source, and the capacities of all sources are 1 in your setting, then each source cannot send more than 1 unit of mass to a single destination. Thus, each $a_{ij}$ will not exceed 1.

Note 3 (Lower bound): To make $a_{ij}$ for each existing edge $(j \to i)$ strictly positive, you can augment the max-flow problem with lower bounds on edge capacities. In a general setting, edge capacity bounds are $0$ and $+\infty$. You may want to replace the lower bound with some positive number, small enough not to interfere with the discovery of the best flow and large enough to satisfy your edge saturation needs.


How has data generation evolved over the years (and centuries)?

You could search on Google Trends all the mentions to generative models. Similarly, you can do something like this on ArXive papers and/or Google Scholar.


I think first of all, if you want to get aggregated data, you need to group it by day/week, like this:
library(dplyr)
library(lubridate)

x <- strptime(data$Timestamp, format = "%Y-%m-%d")#assume you need only days/month , assign to a variable, because dplyr has problems with with date type.

data$month <- month(x) #get month from date obj.

month_summ <- data %>% group_by(month) %>%  #group by month and calculated sold items per month
  summarise(
    total_sales = n()
  )

library(ggplot2)

ggplot(data=month_summ, aes(x=month, y=total_sales)) +   geom_bar(stat="identity") #plot the histogram

This should do the work for you or act like a starting point. Here is a good reference for ggplot2 bar plots
Hope this helps!


Sadly no, there is not a T-SNE implementation for WEKA.

If you can install python packages in your environment, then you can use the wekaPython package (in WEKA's package manager) to run scikit-learn's T-SNE implementation on data you have loaded into WEKA. Use this code in the 'CPython Scripting' panel (which appears after successfully installing wekaPython):

X = py_data.iloc[:, :-1]
y = py_data.iloc[:, -1]

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

tX = TSNE().fit_transform(X)

plt.scatter(tX[:, 0], tX[:, 1])
plt.show()


What algorithm is best suited to derive the best match between two people in a data set?

Given your data sample, unless you have a more subtle way to measure similarity between different values for a given genre (e.g. some other resource indicating that football is closer to basketball than to tennis for instance), it seems that the only similarity measure that you can use is to count how many tastes two users have in common.

The similarity score can be only 0, 1, 2 or 3, so I don't think you really need clustering. You can simply build a map where the key is the concatenation of the 3 "genre" columns, and the value is the set of users which have these tastes. Some users might not have an exact match (3 identical genres), so you do the same process but for only 2 genres in common, and then for only one.


Embedding a Machine Learning Model into a Web Application

I am not sure you would get any such tutorial regarding the same. It is more like asking how to build a software.

For example, consider you have to build an ML model for doing Sentiment Analysis before a post comes onto your feed, this is how you need to build the system:

The ML model would constitute the backend of the software, as it is just another algorithm we are using for building the software.
The plots or the feed you want to display to the end user would constitute the front end, which would include the templates and the views.

The above can be achieved through an MVC framework. The data can be sent back and forth the server and the client through a protocol.

Useful link

In case, you want to integrate it with an external system, then create an API for the above system which you have built and let the other systems use your software/app by calling your api.

Now, create a simple SlackBot which can interact with your API with some certain commands, and display the relevant results in the group.

Example of such a system


What is the difference between word-based and char-based text generation RNNs?

Here is what I learnt recently.

Obviously, when talking about text generation RNNs we are talking about RNN language models. When asking about word/char-based text generation RNNs, we are asking about word/char-based RNN language models (LM).

Word-based LMs display higher accuracy and lower computational cost than char-based LMs.

  This drop of performance is unlikely due to the difficulty for character level model to capture longer short term memory, since also the Longer Short Term Memory (LSTM) recurrent networks work better with word-based input.

This is because char-based RNN LMs require much bigger hidden layer to successfully model long-term dependencies which means higher computational costs.

Therefore, we can say that

  one of the fundamental differences between the word level and character level models is in the number of parameters the RNN has to access during the training and test. The smaller is the input and output layer of RNN, the larger needs to be the fully connected hidden layer, which makes the training of the model expensive.

However, char-based RNN LMs better model languages with a rich morphology such as Finish, Turkish, Russian etc. Using word-based RNN LMs to model such languages is difficult if possible at all and is not advised.

The above analysis makes sense especially when you look at the output text, generated by char-based RNNs:

  The surprised in investors weren’t going to raise money. I’m not the company with the time there are all interesting quickly, don’t have to get off the same programmers.

While simple char-based Maximum Likelihood LM with a 13-character window delivers this:

  And when she made many solid bricks. He stacked them in piles and stomped her
  feet. The doctor diagnosed him with a bat. The girl and her boyfriend asked
  her out.

Of course I cherry-picked the example (actually most ML LM examples looked better than any RNN generated text I've read so far) and this tiny ML LM was trained on a simpler corpus but you get the idea: straightforward conditional probability generates better texts than far more complex char-based RNN.

Char-based RNN LMs can mimic grammatically correct sequences for a wide range of languages, require bigger hidden layer and computationally more expensive while word-based RNN LMs train faster and generate more coherent texts and yet even these generated texts are far from making actual sense.


Why do we need to concatenate in a U-Net?

These types of connections between layers are called skip connections, searching for this will give you a way to find more in-depth information. Broadly speaking, there are two advantages to using skip connections in this case:

They allow gradients to more freely flow through the model, mitigating the issue of vanishing gradients
They allow features from the encoder side of the network to the decoder side of the network, adding extra information that might be lost because of the downsampling on the encoder side of the network.


Is there any PageRank-like method on weighted graph?

Weighted versions of Pagerank do exist, and it is easy to incorporate edge weights into the PageRank algorithm (just multiply each edge's probability by its weight vector, and then normalize to make the edge probabilities to add up to 1)

The difficult question is how to set these weights. Backstrom and Leskovec propose an algorithm to learn these weights in their WSDM paper on Supervised Random Walks which they propose for use in link prediction and link recommendation.


Predicting invoice data of 12 month using only 1 month data

Since Invoice analysis comes under trend analysis. It's better to take a look at the data from a statistical point of view. Plot graphs, manually check the relation and so on. if there is a trend which is not monotonic, or if it's recurring weekly it's better to take a look at (Recurrent Neural Network)RNN's to predict based on the day or week.


Conditionally replace dataframe cells with value from another cell

Aha! For some reason, when I named a variable that pointed to a value in a dataframe, and then tried to use the variable to put it back into the table replacing other cells, it changed the dtype. That's why I was getting NaNs. I just had to convert it to a usable type using tolist.

As for the nearest-value identification, this seems to have worked:

Pfinal = df.iloc[(df['P']-Plithos).abs().idxmin()]

F_max = Pfinal[1]

F_max = F_max.tolist()

df.loc[(df['P'] < Plithos),['F']] = F_max


Neural network, Support Vector Machine or something else to classify into 7 groups

Welcome to the site, @intellion! I can recommend a couple of things for you. First, if you think you'll continue your interest in data science and will be working on problems like this in the future, it is worthwhile to start familiarizing yourself with the field. There are a variety of introductory textbooks and courses you can choose from. If you have a strong mathematical background, I'd recommend Tibshirani's "The Elements of Statistical Learning", which is a solid classic in the field. If you're approaching this from a more application-oriented angle, I'd recommend something written specifically for the language you'd like to work in. If you plan on using R, there's "Machine Learning with R"; if you plan on using python, there's "Machine Learning in Python." There are of course a variety of free online resources as well.

To your question, you could certainly solve your problem with Support Vector Machines (SVMs) or Neural Networks, but I don't think that's the first place to start. To me, this looks like a classic multiple regression application situation (for an overview for doing this in R, see this or this textbook). If you really want to try something in machine learning, perhaps decision trees are a good place to start, as they'll allow you to easily assess the influence of variables on the final model fit (e.g., using Gini Impurity). I've found the rpart library in R to be quite nice to work with, but it can be a little slow for larger data. You could also use sklearn in python, if that's your preferred language.


Welcome to the site!

As you have mentioned that you are an Instructor of Statistics I think you can directly start with different methodologies/techniques available.

As @Emre has suggested it would be better if you can work on different project available on Kaggle, Analytics Vidya. You would be exposed to most of the necessary things in Data Science field. Following R-Bloggers, will expose you to the latest packages. Ofcourse following Data Science Exchange.

You can doing small projects on different datasets on UCI. For experimenting using different techniques.

These are just couple of things which would help you to be updated.

Follow them for atlest next 2-3 months then you can access yourself.


When should we choose agglomerative clustering over K-means clustering?

You can see this comparison table in sklearn, which gives some intuition about where and when each algorithm is successful:

It might be a good idea to try both and evaluate their accuracy, with an unsupervised clustering metric, like the silhouette score, to get an objective measure of their performance on a specific dataset.
Some other major differences are:

K-means performs better for 2D & 3D spheres
Hierarchical clustering can have reduced performance on larger datasets
Hierarchical clustering is sensitive to outliers


Does produced data fall under the "data collection" category?

There are probably many ways to think about this, but in the statistical world view I think the word you are looking for is dependent variable.

Statistician see students (or any other object for that matter) as a stochastic process that produce outcomes (dependent variables) given traits that we can observe (independent variables).

Data scientists tend to call those things target outcomes and features respectively.

None of these terms however mean anything about who owns the data. Philosophically a dependent variable is produced by the stochastic process (the outcome is governed by the givens), and are in a sense a characteristic of the process. Policywise this doesn't carry much weight though. GDPR for instance is mainly concerned about data that can be used to identify persons. That isn't likely the case with outcome variables, but it can be!


Keras stateful LSTM returns NaN for validation loss

No idea about the tensor board stuff, but the NAN for val loss could be being caused by an unexpected very large or very small number. As your training dataset is very small there is a high chance that the validation data has a very different underlying distribution.
If the latent space of your NN isn't continuous and validation samples fall too far away from the distribution of the training samples, the NN will not deal with them well. In a classification problem this would result in misclassification (most likely). However, as you are generating a prediction, this could mean that the NN creates a really large number or a really small one. Then, in your loss function might be getting something divided by zero or by infinity.

I'd suggest looking at the the loss function you are using, if it has a divide, can the denominator be zero? One solution to this would be to use the clip function to make sure that your predicted tensor is between 0+epsilon and 1.

def clipped_squared_error_loss(y_true, y_pred):
    y_true = K.clip(y_true, K.epsilon(), 1)
    y_pred = K.clip(y_pred, K.epsilon(), 1)
    return K.sum(K.square(y_pred - y_true))

You may also want to look into Kullback Leibler Divergence as a means to make the latent space of your NN continuous.


What is the difference between Hadoop and noSQL

Hadoop is not a database, hadoop is an entire ecosystem.

Most people will refer to mapreduce jobs while talking about hadoop. A mapreduce job splits big datasets in some little chunks of data and spread them over a cluster of nodes to get proceed. In the end the result from each node will be put together again as one dataset.

Let's assume you load into hadoop a set of <String, Integer> with the population of some neighborhoods within a city and you want to get the average population over the whole neighborhoods of each city(figure 1).

figure 1

    [new york, 40394]
    [new york, 134]
    [la, 44]
    [la, 647]
    ...

Now hadoop will first map each value by using the keys (figure 2)

figure 2

[new york, [40394,134]]
[la, [44,647]]
...

After the mapping it will reduce the values of each key to a new value (in this example the average over the value set of each key)(figure 3)

figure 3

[new york, [20264]]
[la, [346]]
...

now hadoop would be done with everything. You can now load the result into the HDFS (hadoop distributed file system) or into any DBMS or file.

Thats just one very basic and simple example of what hadoop can do. You can run much more complicated tasks in hadoop.

As you already mentioned in your question, hadoop and noSQL are complementary. I know a few setups where i.e. billions of datasets from sensors are stored in HBase and get then through hadoop to finally be stored in a DBMS.


How to Calculate semantic similarity between video captions?

There's not exactly best solution, mostly depending on the experiment results. So maybe you need to try several different approaches to find out the best way. But one common way (to my knowledge) is using a word embedding (add an embedding layer or use GloVe, BERT etc.) to get the word vector, and then concatenate the word vectors in each sentence into sentence representation. You can calculate the similarity between the sentence representation.

Well actually I don't practically test this :p, but I have read it in the papers. Therefore this approach is for reference only.