Super Resolution CNN's are used to increase the resolution of an image. These architectures are not used to predict classes or to detect objects. They are an image processing technique.

There are several approaches but usually, in a first step, the image is upsampled using Bicubic interpolation. Afterwards, the quality of the upsampled image is enhanced using a CNN. The second step yields additional improvements in image quality since Bicubic interpolation simply interpolates and a CNN, on the other hand, is able to recognize structures.

(Note: Bicubic interpolation is not a definitive requirement. This is just an example to explain why a CNN is used for resolution improvement.)

The image above is from a paper of the authors of SRCNN. As you can see, a low-resolution image is used as input for the Super-Resolution CNN. The SRCNN then generates a high-resolution image.

This short video has additional information about Super Resolution CNN's and should give a good intuitive understanding.


use machine learning to predict a next day of visit for customer

You can approach that problem as time-series prediction problem.
Below is one of articles or tutorials about time-series forecasting/prediction.

https://machinelearningmastery.com/time-series-forecasting/


Stata: Trying to merge two data sets with common year observations

Have a file open in your current stata session. in your case let's say the file opened is M&D Research Project one.
On the top option bar mentioning 'File', 'Edit' etc. go to 'Data' -> 'Combine Datasets' -> 'Merge two Datasets'
In Type of merges choose 'Many-to-One' since your currently open dataset is 'M&D' having multiple year observations that have to merge with one observation per year in your 'cpi' dataset.
In 'Key Variables' mention the name of the column to match/merge the data with. In your case it's 'year'.
Choose the file 'cpi' under 'Filename of dataset on disk'.

You'll have your resultant data.

NOTE: You may have to make sure the order of the year on both datasets is sorted

Also if you want to merger cpi with M&D and not vice versa, open cpi dataset first and choose many to one relation. The rest of the steps are same


If you don't define the validation set/ validation split for your model, then it has no way to check for it's performance because you have not provided anything to the model on which it can validate its performance. In this case, the model will run through training examples, will learn a way to minimize the cost function as per your code and that's it. You get a model which has learned a hypothesis from your data but how good  the model is, it can only  be checked by making predictions for the test set.


Correcting for one of multiple strong batch effects in a dataset

My first try to this would be to cluster my independent variables and use the labels in a supervised learning predictor. But I think this can be a good research problem beyond the scope of a stackoverflow short answer. I would recommend you to start with the Exceptional Model Mining paper.

Hope this helps!


ML methods for prediction, using categorical variables and time

One way to frame the problem of predicting patterns of complex discrete states is probabilistic graphical models (PGMs). PGMs model joint (multivariate) distributions over large numbers of random variables that interact with each other.
PGM do not have a notion of linear time. They model state. Given the current state and previous states, how likely are other states in the next time step? In your problems, state is the type and location of alert.
Common examples of PGMs are Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs).


Interpreting Machine Learning Classification Metrics

I assume the results you show have been evaluated according to a train-validation-test split approach. With the information you have provided, it is possible to figure your confusion matrix. It has to be something like :
label=0 and pred=0 :  330.000
label=0 and pred=1 :  4800
label=1 and pred=0 : 11.000
label=1 and pred=1 : 7200
According to this, your model has some value but and is better that what random choices could give you. Thinking of a random choice between the two classes given the distribution of the classes, you would get something close to :
label=0 and pred=0 :  328.000
label=0 and pred=1 :  12.600
label=1 and pred=0 : 12.600
label=1 and pred=1 : 484
So, I think you model has some value. Although, I think you have to confront this performance with the use-case you are working on, the features and model you are using and obviously the level of performance you think acceptable.
To increase your model performance, it exists many approaches you can try. Here are some of them :

Try other classification algorithms (Naive Bayes, Logistic regression, SVM, tree-based technics...).
Clean your input variables if there are dirty and extract insights from it. This is called features engineering.
Diagnose your results and search if you are in an underfitting or an overftitting situation. Adapt your algorithm if needed.
Apply technics relative to unbalanced classes. See more here.


In order to make the explanation clear I will use the example of 34-layers:

First you have a convolutional layer with 64 filters and kernel size of 7x7 (conv1 in your table) followed by a max pooling layer. Note that the stride is specified to be stride = 2 in both cases.
Next, in conv2_x you have the mentioned pooling layer and the following convolution layers. Here the layers are normally grouped in pairs (trios in bigger architectures) because is how the residuals are connected (the arrows jumping each two layers). The first matrix:

\begin{equation}\begin{bmatrix}
  3x3, & 64 \\
  3x3, & 64
\end{bmatrix}*3\end{equation}

means that you have 2 layers of kernel_size = 3x3, num_filters = 64 and these are repeated x3. These correspond to the layers between pool,/2 and the filter 128 ones, 6 layers in total (one pair times 3).

Following, we have conv3_x:

\begin{equation}\begin{bmatrix}
  3x3, & 128 \\
  3x3, & 128
\end{bmatrix}*4\end{equation}

2 layers of kernel_size = 3x3, num_filters = 128 and these are also repeated but on this occasion times 4. These are the following 8 green layers in the figure.

This continues until the avg_pooling and the softmax.

Be aware that the stride is always 1 except when the filter size increases. This is discusssed in the paper:

  Plain Network: Our plain baselines are
  mainly inspired by the philosophy of VGG nets. The convolutional layers mostly have 3×3 filters and
  follow two simple design rules: (i) for the same output
  feature map size, the layers have the same number of filters;
  and (ii) if the feature map size is halved, the number
  of filters is doubled so as to preserve the time complexity
  per layer. We perform downsampling directly by
  convolutional layers that have a stride of 2.

  Residual Networks: The baseline architectures
  are the same as the above plain nets, expect that a shortcut
  connection is added to each pair of 3×3 filters.

That is why, each time the number of filters is doubled you will see that the first layer of a different colour specifies num_filters/2.


Using recurrent neural networks for modeling errors in IMUs

The only way to find out is to try it, but I doubt it.  A Kalman makes a specific assumption about the "priors", i.e., about the nature of how the signals are related over time.  This is based on knowledge about the physics of movement and how this affects the sensor values, combined with some assumptions about the likelihood of different kinds of movements.  Assuming the model and assumptions are accurate, this helps provide a more accurate estimate of the orientation.

In contrast, a recurrent neural network has no knowledge built in, and no useful "priors".  It has no model about the physics of movements and no assumptions.  In principle, this makes it more general.  But in practice, it means that, if you have a limited amount of data, it is likely to be less accurate.

In general, a strong prior and useful model can be very helpful.  In the limit, as you have an unlimited amount of data, you might not need a model.  But with a limited amount of data, the model can be helpful.  The Kalman filter provides a pretty reasonable model, and when trying to work with data from IMUs, we typically have only a limited amount of data.

The place where RNNs could be more effective would be if the assumptions made by the Kalman filter's model do not correspond to reality.  Then, you could imagine that RNNs could possibly learn the dynamics, because they don't assume a particular model, whereas Kalman filter is locked into a particular model.  I don't particularly expect this to happen in practice, but it's a possibility.

Overall, from first principles, I would expect/predict RNNs to be less effective than Kalman filtering.  But that's just speculation/guessing.  If you really want to know, the way to find out for sure is to try some experiments.


Which plot to use for data spanned on multiple years?

For time data, you should often use a simple Line Chart, with time on the X axis and the other variable dependent of time (here the number of accident) on the Y axis. Since you have several states to plot, you can simply draw one line for each state. to differentiate them, you can use a visualization variable : color, width, pattern ... Color is probably the most relevant here. You can as well write the name of the state near each line.


Problem with a feature (normal distribution + peak around 0)

Aren't you providing the answer? You can split the feature in two, namely, if feature_to_split is the feature you're talking about, you can create feature_to_split_ispresent which will take either 1 or 0 depending on the presence or absence of that specific characteristic, and feature_to_split_value which will take the actual value of that characteristic.


Looking for smallest set of rows that form a natural key in a data set

You could do this a few ways...

write a script that does the following.

pick a table,

get number of rows in the table, can do this as you go. (+=1 in loop)

select several fields(columns) in table that you think may form the key,

create empty Set()

burn through the file row by row, for each row grab the target fields and construct a string key str(field1)+'_'+str(field2)..etc.  Add this key to your set.

when you're done going through all rows of a table, check len(set).  If the key you picked is good then len(set) will equal # rows.

Another way, and this will depend on what you want to do with your data and how often you will access it...is to import it into a sql database.  I usually use mysql. Once your data is in:

select count(*) from table_name;  will give you rows in table

select count(distinct field1, field2, field3) from table;  gives number of distinct combos of field1+field2+field3

if the two selects give the same number then you have a key that works. It may not guarantee a join-key but it will be unique to that table and will help the linking process.

many people highly advocate the use of pandas. If you get the data imported into arrays you can create a dataframe and do some queries similar to SQL.  I'm not super familiar with pandas so I can't give an example.

If it's a one-off, I'd make a python script.  But if you think you'll be exploring the data further, spending the time to put the data in a database could be useful.


Where the following propagation model arise in practice

Imagine each node in your graph as a user, and each edge as an action such as 'share'. It may also be a bidirectional relationship 'share' and 'view'.

Some social scientists and engineers estimate the probability that a message is 'shared' and 'viewed' given that a particular user decides to share it. This process is called "information diffusion", "information propagation", "cascading", etc.

If you are interested in the details of how these calculations are performed, check out these papers:

http://cs.stanford.edu/people/jure/pubs/netrate-netsci14.pdf

http://cs.stanford.edu/people/jure/pubs/cascades-www14.pdf

http://cs.stanford.edu/people/jure/pubs/infopath-wsdm13.pdf

Similar topics:
http://snap.stanford.edu/papers.html


i'm using GridSearchCV to find parameter C for SVC() classifier present in sklearn.svm . I'm not getting the optimal result desired

You didn't do any cross validation of data in GridSearchCV as you didn't mention any cv parameter. So what GridSearchCV did is it found the best estimator on the train set which possibly made the model overfitted on training data. You should use cross validation to find the best best estimator.
abc = GridSearchCV(clf,params,cv=5)


Can RNN learn for each `t` in time from a whole new dataset (many entries)

So, from what I understand you are trying to predict the target column given all your feature columns and the information from the past. This feels like a time-series forecasting problem to me too (You can check variations online).

Vanilla RNN tends to vanish or explode the gradients especially given that you have such a large number of timestamps. I'd suggest you to try an LSTM in order to avoid it as much as you can. Ideally yes, it is supposed to remember what happened earlier and learn from that.


Find correlation between grades from two raters

You can treat this as a type of inter-rater agreement problem and use Cohen's Weighted Kappa or a similar measurement. Weighted kappa takes into account the distribution of ratings for each round and the difference between grades.
Three matrices are involved: the matrix of observed scores, the matrix of expected scores based on chance agreement, and the weight matrix. Weight matrix cells located on the diagonal (upper-left to bottom-right) represent agreement and thus contain zeros. Off-diagonal cells contain weights indicating the level of separation between ratings. Often, cells one off the diagonal are weighted 1, those two off 2, etc.
source:Wikipedia
The equation for weighted κ is:
$$\kappa = 1 - \frac{{\sum_{i=1}^k}{\sum_{j=1}^k}w_{ij}x_{ij}}{{\sum_{i=1}^k}{\sum_{j=1}^k}w_{ij}m_{ij}}$$
where k=number of codes and $w_{ij}, x_{ij}, m_{ij} $ are elements in the weight, observed, and expected matrices, respectively. When diagonal cells contain weights of 0 and all off-diagonal cells weights of 1, this formula produces the same value of kappa as the calculation given above.
In practice, you may want to use an implementation in Python, R, or a statistical software package rather than manual calculations.
Here is some intuition from a similar example on why to use weighted kappa (round 2 grades are converted to round 1 scale):
| dog             | round one | round two |
| --------------- | --------- | --------- |
| Dog1_criteria1  | A         | C         |
| Dog1_criteria2  | A         | B         |
| Dog1_criteria3  | A         | A         |
| Dog1_criteria4  | B         | B         |
| Dog2_criteria1  | A         | A         |
| Dog2_criteria2  | A         | A         |
| Dog2_criteria3  | A         | A         |
| Dog2_criteria4  | C         | C         |

You could look at % agreement and give a score of 6 of 8, or 0.75. That seems good, but suppose the judges just gave everyone an A. That would also be 0.75. So we need to factor in the frequency of the grades and the probability of agreement between each combination. That's where the expected matrix comes in.
Then there is the degree of agreement between two ratings. You usually want to assign more agreement to an A/B or B/C than to an A/C. And the differences may not be linear at all. The weight matrix allows you to account for the differences.
The observed matrix is simply the count of observations for each possible pair of ratings.
A final note: there are several variations of Kappa, such as quadratic weighted kappa. Most of them should work well for comparing grades.


Question Regarding Multi-label probability predictions

A question is a bit broaden as you do not specify if you do not know how to do it in theory or how to tackle it with a ML method.

Some of ML methods:

LogisticRegression in sklearn handles multiple class

lr = LogisticRegression()
lr.fit(X, y)
class_probabilities = lr.predict_proba(X)  # outputs the probabilities

You might also want to consider Support Vector Machines.

Theory:

You can do a "one vs rest" when you train a single classifier per class taking the sample of all other classes as negative example. (see wiki article for that)


Which performs better in time series forecasting, LSTM or SVR?

So far I know, Recurrent Neural Network(RNN) is best for time series problem. LSTM(type of RNN), always consider long term dependencies and evaluate new value after understanding the whole series pattern.
Whereas SVR, consider each row as a sample for training data and predict the outcome and will not consider the previous patterns.

  For example: 10->20, 20->40, 30->60, and so on ...

  LSTM will try understand the whole series and then predict the next
  value, whereas for SVM each row is a individual training sample divided into
  Features/Target and will predict value based upon the different values
  it learnt so far.

Thats the reason RNN is better for timeseries.

Please refer:
https://www.researchgate.net/post/how_I_can_do_sequence_to_sequence_prediction_using_SVM

For SVR explanation
https://machinelearningmastery.com/time-series-forecasting-supervised-learning/


Feature Selection Algorithm for Attributes with Logical Relationships (like "AND")

Emergent pattern mining could be useful for you. In classification, emergent patterns are subsets of features which differentiate one class from the others. Look at Example 1.1 in the paper. Two feature subests, denoted $X$ and $Y$, are EPs because $X$ has large support in one class and small support in the other, and vice versa for $Y$. The nifty thing about EP mining is that it considers subsets of features instead of individual features.

Mining patterns in this way could address your problem of feature1 and feature2 having no correlation with the target class alone, but the combination feature1 AND feature2 having high correlation with the target class.

Unfortunately, I don't see any open-source software implementations on the web. However, there are a number of papers out there which implement some sort of EP mining.


Not completely true. In validation set, we find the best hyperparameters, but not with the same parameters of the model. That is, for every value of the hyperparameters we run the training process and obtain the loss for that set of hyperparameters, and we select the hyperparameters values with the lowest validation loss.

Edit

Ridge regression example: Ridge regression depends on one parameter, $\lambda$, that penalizes your model weights. Ridge regression has a single hyperparameter. As ridge regression is a linear model, it has also some weights $\theta_0, \dots, \theta_n$ (parameters) to train. The way to proceed to choose $\lambda$ and $\theta_0, \dots, \theta_n$ is the following:

For each value of $\lambda$ in a grid, we train by gradient descent the ridge regression model (thus optimizing the training loss) and obtain a set of parameters $\theta_0^*, \dots, \theta_n^*$. We measure the loss of the trained ridge model in the validation set.
We choose the $\lambda$ that minimizmes the loss in the validation set and the parameters $\theta_0^*, \dots, \theta_n^*$. If we wish, and we don't have much data, we can retrain the ridge model with the optimal $\lambda$ in the concatenation of the training and validation sets, obtaining some parameters $\hat\theta_0, \dots, \hat\theta_n$.
We compute the test error with the parameters obtained in the last step and report that as the most realistic value of the error of the model.


I'll say it because someone else will: It depends on a lot.

Always good to start off with a regular cross-entropy loss for classification problems, especially because it comes out of the box in many libraries (PyTorch, Keras, TensorFlow).
If you're dealing with binary classification task, then the Binary Cross Entropy might be an interesting alternative to try as well.
Surely someone can recommend more obscure losses, but starting with those simple ones could potentially give a very good return for minimal effort!

Again going for simplicity, Adam optimiser or SGD with momentum is often a low hanging fruit you can try.

I'd say use Adam and cross-entropy, and work your way through optimizing your hyper-parameters as much as you can (validation set!) and only then you can move other parts.

Also for more specific answers you should specify how many features, how many datapoints, and how you dealt with your categoricals (label encode / One-Hot).


First of all, I'd disagree that Markov Chains are dealing with a "single type of variable". If you look at the formal definition of a Markov Chain, you'll see that variables $X$ are random variables. And random variables are defined over arbitrary (well, measurable) sets of possible outcomes. So your $X$ can not only be from $\{sun,rain\}$ set, but if could be from a Cartesian product of $\{sun,rain\}$ and $\{windy,cloudy,calm\}$ and temperature from $[-60,60]$ interval.
About the relation between Markov Chains and Bayes Nets, I'd say that there is a common framework that lets you understand relationship between those (and, in fact, many other probabilistic structures). In all cases we have a collection of random variables - for Markov Chains these are $X_0,X_1,X_2\dots$ And for Bayes Nets lets call them $A,B,C,D,E,F$.
In both cases we are interested in the probability distributions over the whole collection of variables, called joint probability distribution:
$$P(X_0,X_1,X_2\dots)\qquad\text{and} \qquad P(A,B,C,D,E,F)$$
The problem is that trying to represent this distribution (say on a computer) is impossible for any reasonably complex problem - the number of possible combinations of variables grows exponentially with the number of variables. So we are coming up with a way to factorize these joints into smaller, manageable multipliers.
For the Markov Chains, the factorization property is
$$P(X_0,X_1,X_2\dots) = P(X_1|X_0)P(X_2|X_1)P(X_3|X_2)\cdots$$
For the Bayes Nets, the factorization property could look like this (for a particular dependency graph that I just made up):
$$P(A,B,C,D,E,F) = P_a(A)P_b(B)P_c(C|A)P_d(D|A,B)P_e(E|C,D)P_f(F|E)$$
In my opinion, considering the joint distribution and then seeing what factorization structure is imposed on it by your framework is a good place to start studying it.


Keras' Evaluate function training model on test set

Keras doesn't update your model with testing data.
It might be that your labels have been provided wrong in the test data, check the model.predict_classes() to get the output of your classes and crosscheck them with your actual output manually by picking random subset.


If you have a list of words for every topic you can indeed try to directly measure the similarity of this list against a sentence, but it's likely that a sentence doesn't always contain one of the topic words so it might not work very well.
A more advanced method would be to obtain a semantic representation for every topic (or topic word) from an external corpus. Any large corpus can be used for that, it doesn't have to be related to your input data. The traditional way to do that is to extract a context vector for every target word by counting the co-occurrences of the target in the corpus: for every occurrence of the target word, take for instance a window of 5 words to its left and 5 words to its right. Then count how many times every context word appears in the window of the target across the whole corpus. This way the final context vector contains the distribution of the context words, i.e. a representation of the meaning of the target word. Comparing this context vector against a sentence is likely to produce a more accurate semantic similarity score.
There are many variants about the exact definition of the context vector: usually stop words are removed but one could also use TF-IDF and/or other kinds of normalization. The more modern version of this method is probably to use word embeddings, but I'm not knowledgeable enough about this.


Why use Variational Autoencoders VAE instead of Autoencoders AE in Anomaly Detection?

Variational autoencoders encourage the model to generalize features and reconstruct images as an aggregation of those features.  This is what the latent space encodes, a compressed feature vector.

Vanilla autoencoders memorize the input and map to the output without the generalization.  If you want to extrapolate from your dataset, variational is the way to go.


How to calculate mAP for detection task for the PASCAL VOC Challenge?

To answer your questions:

Yes your approach is right
Of A, B and C the right answer is B.

The explanation is the following: In order to calculate Mean Average Precision (mAP) in the context of Object Detection you must compute the Average Precision (AP) for each class, and then compute the mean across all classes. The key here is to compute the AP for each class, in general for computing Precision (P) and Recall (R) you must define what are: True Positives (TP), False Positives (FP), True Negative (TN) and False Negative (FN). In the setting of Object Detection of the Pascal VOC Challenge are the following:

TP: are the Bounding Boxes (BB) that the intersection over union (IoU) with the ground truth (GT) is above 0.5
FP: two cases (a) BB that the IoU with GT is below 0.5 (b) the BB that have IoU with a GT that has already been detected.
TN: there are not true negative, the image are expected to contain at least one object
FN: those ground truthes for which the method failed to produce a BB

Now each predicted BB have a confidence value for the given class. So the scoring method sort the predictions for decreasing order of confidence and compute the P = TP / (TP + FP) and R = TP / (TP + FN) for each possible rank k = 1 up to the number of predictions. So now you have a (P, R) for each rank those P and R are the "raw" Precision-Recall curve. To compute the interpolated P-R curve foreach value of R you select the maximum P that has a corresponding R' >= R.
There are two different ways to sample P-R curve points according to voc devkit doc.
For VOC Challenge before 2010, we select the maximum P obtained for any R' >= R, which R belongs to 0, 0.1, ..., 1 (eleven points). The AP is then the average precision at each of the Recall thresholds. For VOC Challenge 2010 and after, we still select the maximum P for any R' >= R, while R belongs to all unique recall values (include 0 and 1). The AP is then the area size under P-R curve. Notice that in the case that you don't have a value of P with Recall above some of the thresholds the Precision value is 0.
For instance consider the following output of a method given the class "Aeroplane":
BB  | confidence | GT
----------------------
BB1 |  0.9       | 1
----------------------
BB2 |  0.9       | 1
----------------------
BB3 |  0.7       | 0
----------------------
BB4 |  0.7       | 0
----------------------
BB5 |  0.7       | 1
----------------------
BB6 |  0.7       | 0
----------------------
BB7 |  0.7       | 0
----------------------
BB8 |  0.7       | 1
----------------------
BB9 |  0.7       | 1
----------------------

Besides it not detected bounding boxes in two images, so we have FN = 2. The previous table is the ordered rank by confidence value of the predictions of the method GT = 1 means is a TP and GT = 0 FP. So TP=5 (BB1, BB2, BB5, BB8 and BB9), FP=5. For the case of rank=3 the precision drops because BB1 was already detected, so even if the object is indeed present it counts as a FP. .
rank=1  precision=1.00 and recall=0.14
----------
rank=2  precision=1.00 and recall=0.29
----------
rank=3  precision=0.66 and recall=0.29
----------
rank=4  precision=0.50 and recall=0.29
----------
rank=5  precision=0.40 and recall=0.29
----------
rank=6  precision=0.50 and recall=0.43
----------
rank=7  precision=0.43 and recall=0.43
----------
rank=8  precision=0.38 and recall=0.43
----------
rank=9  precision=0.44 and recall=0.57
----------
rank=10 precision=0.50 and recall=0.71
----------

Given the previous results:
If we used the way before voc2010, the interpolated Precision values are 1, 1, 1, 0.5, 0.5, 0.5, 0.5, 0.5, 0, 0, 0. Then AP = 5.5 / 11 = 0.5 for the class of "Aeroplanes".
Else if we used the way since voc2010, the interpolated Precision values are 1, 1, 1, 0.5, 0.5, 0.5, 0 for seven unique recalls that are 0, 0.14, 0.29, 0.43, 0.57, 0.71, 1.Then AP = (0.14-0)*1 + (0.29-0.14)*1 + (0.43-0.29)*0.5 + (0.57-0.43)*0.5 + (0.71-0.57)*0.5 + (1-0.71)*0 = 0.5 for the class of "Aeroplanes".
Repeat for each class and then you have the (mAP).
More information can be found in the following links 1, 2. Also you should check the paper: The PASCAL Visual Object Classes Challenge: A Retrospective for a more detailed explanation.


Storing Large dataset for processing and analysis of data

It would depend on the use cases - read vs write vs analytics etc. Nonetheless, you may want to explore Hadoop if not done already.


How to check for overfitting with SVM and Iris Data?

You check for hints of overfitting by using a training set and a test set (or a training, validation and test set). As others have mentioned, you can either split the data into training and test sets, or use cross-fold validation to get a more accurate assessment of your classifier's performance.

Since your dataset is small, splitting your data into training and test sets isn't recommended. Use cross validation.

This can be done using either the cross_validate or cross_val_score function; the latter providing multiple metrics for evaluation. In addition to test scores the latter also provides fit times and score times.

Using your example;

from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn import svm
from sklearn.metrics import accuracy_score

iris = load_iris()
X = iris.data[:, :5]  # we only take the first two features.
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=51)
svm_model = svm.SVC(kernel='linear', C=1, gamma='auto')
svm_model.fit(X_train,y_train)
predictions = svm_model.predict(X_test)
accuracy_score(predictions, y_test)

raw accuracy: 0.96666666666666667

Using the cross_val_score function, and printing the mean score and 95% confidence interval of the score estimate:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(svm_model, iris.data, iris.target, cv=5)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Accuracy: 0.98 (+/- 0.03)

Of course the iris dataset is a toy example. On larger real-world datasets you are likely to see your test error be higher than your training error, with cross-validation providing a lower accuracy than the raw number.

So I wouldn't use the iris dataset to showcase overfitting. Choose a larger, messier dataset, and then you can start working towards reducing the bias and variance of the model (the "causes" of overfitting).

Then you can start exploring tell-tale signs of whether it's a bias problem or a variance problem. See here:

https://www.quora.com/How-many-training-samples-are-needed-to-get-a-reliable-model-in-ML/answer/Sean-McClure-3?srid=zGgv


How can one generate future forecasts from probabilistic events?

If you have historical data on percentage of return for few weeks after the sales, I believe you can do it. It will be a multiple-output regression problem e.g. you will have to formulate the target values (i.e. output matrix) in a way so that %-return in first week will be in column one, %-return in second week will be in columns two and so on. After this, plug in all your input variables and run a multivariate multiple output regression algorithm. Such an algorithm on scikit-learn can be found (sklearn.multioutput.MultiOutputRegressor: here)


LSTM Time series prediction for multiple multivariate series

I think one question you need to answer is how the traffic is correlated across cities.

If they are not correlated (which is likely) then the city can be a categorical input variable. The network can learn what is common across all cities and what is specific to different cities.

Here is a reference that may be helpful. In this example, the inputs are clearly related, so that may be a difference in your case.

You also may consider developing a model for one city, and then improving it by adding other cities.


Fast answear
Mean Shift LSH which is an upgrade in $O(n)$ of the famous Mean Shift algorithm in $O(n^2)$ well know for its image segmentation ability
Some explanations
If you desire a true unsupervised approach to segment images, use clustering algorithms. The fact is that there is a lot of algorithms with different time complexity and specificity. Take the most famous one, the $K$-Means, it is in $O(n)$ so pretty fast but you have to specify how many cluster you want which is not what you intend by exploring an unknown image without any information about how many shapes are presents in it. Moreover even if you suppose that you know how many shape are present, we may suppose that there shapes are random which is another point where the $K$-Means fail because it is design to find elliptic clusters and NOT random shape ones.
At the opposite we have the Mean Shift that is able to find automatically the number of cluster -- which is useful when you don't know what you're looking for -- with random shapes.
Of course you replace the $K$ parameter of $K$-Means by others Mean Shift parameters which can be tricky to fine tuned but it doesn't exist a tool that allow you to do magic if you're not exercising to do magic.
An advice to image segmentation clustering
Transform your color space from RGB to LUV which is better for euclidean distance.
$K$-Means vs Mean Shift LSH time complexity

Mean Shift : $O(\alpha.n)$
K-Means : $O(\beta.n)$
$\alpha \gt \beta$

Mean Shift LSH is slower but it fits better with your needs. It stay still linear and is also scalable with the mentioned implementation.
PS : My profile picture is an application of the Mean Shift LSH on myself if it can help to figure out how it works.


Can the 'bin size' in a histogram be thought of as a regularity constraint?

Yes, I think that the idea is to think of a histogram as a particular case of Kernel Density Estimation. In this case, the kernel function for a KDE to be a histogram is an indicator function. In this case, the bin size is equivalent to the bandwidth of the KDE.

The bandwidth of the KDE is indeed a regularity measure: KDEs with very high bandwidth have huge bias (they say that the distribution is uniform), while KDEs with very low bandwidth have huge variance (they adapt to the training data but do not generalize well). As the bandwidth of a KDE is equivalent to the bin size, we can say that the bin size is a regularity measure.


How to do k-fold validation with classifiers?

So I cannot calculate the MSE of validation folds, there is no MSE
  here.

The question that you firstly need to ask is "how do I evaluate my model".

Since the model is a classifier, you can use the accuracy (number of correct classifications over the overall attempts) or the F-score as measures of your model's performance. Based on the chosen measure, you will thereafter evaluate the results of kfold-validation.


Disclaimer: I am not 100% sure if my solution will be good for all cases but this solved my problem considerably.

For me changing to XGBoost based imputation worked wonderfully. In the earlier case I was doing multiple different type of imputation like mean, mode etc. From there I retained only those imputations on which I was very much confident. Imputations on which I didn't have much confidence I did let XGBoost do the job. After this change most of the overestimation cases vanished.

Another benefit I got from this change was: model fit parameters were direct reflective of the overestimation. So models in overestimation was still happening; all of those have had bad model fit Vs models with no overestimation always have good good fit.

I believe this is happening because with XGBoost also doing the imputation it uncovers and leverages deeply hidden patterns in data. With this knowledge it is able to do better imputation of the scenarios where my ways of ordinary imputation were not much useful.


The "best" package depends on your goals and data really.

A few tree / forest packages that I've come across:

randomForest - an implementation of the original algorithm
ranger - a flexible implementation designed to run on high-dimensional data (my personal favourite forest package)
party's ctree / cforest functions - conditional inference trees / forests
rpart

Some of these packages focus on forests rather than trees; you can coerce the forest algorithm into building a tree by flexing appropriate parameters.


I'm answering my question. Well, I've ditched XGBoost for LightGBM (the only Microsoft product I seem to enjoy by far) but since the interface is quite similar, the same should apply to XGBoost. Apparently I don't need to apply Sigmoid to predictions. I don't know why the examples suggest otherwise.

def logistic_obj(y_hat, dtrain):
    y = dtrain.get_label()
    p = y_hat # p = 1. / (1. + np.exp(-y_hat))
    grad = p - y
    hess = p * (1. - p)
    grad = 4 * p * y + p - 5 * y
    hess = (4 * y + 1) * (p * (1.0 - p))
    return grad, hess

def err_rate(y_hat, dtrain):
    y = dtrain.get_label()
    # y_hat = 1.0 / (1.0 + np.exp(-y_hat))
    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
    loss_fn = y*np.log(y_hat)
    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y), False


Twitter POS and NER: What is state-of-the-art?

SOTA is changing so rapidly in NLP that even Data Science professionists struggle to cope with it. I have two main sources that I constantly check to gain some insights on SOTA:

NLP Progress from Sebastian Ruder. It contains updates on NLP on a whole lot of subfields, NER and POST included.

Paper with code contains a section on NLP. That's a great website for ML in general.

I know these links do not tackle the problem of Twitter specifically, however I don't think that domain is qualitatively different from others. IMO, of course.

About your other question:

Are industrial-strength programs like Spacy and SparkNLP accurate for such texts? How about FlairNLP and Stanford's CoreNLP accuracy measures?

As I wrote above, it's mostly a matter of personal preference and/or contingent project needs. There's no right or wrong tool. Personally, I found Stanford tools to be the best, for either the quality of their predictions and the amount of models available from a single pipeline. But as I said it's very subjective.


Can we Create a model using (bi lstm crf) where we only want to predict 1 entity.?

[edited] Yes, the trick I would use is to train a regular model and predict the top 10 most likely predictions for each sentence. This will give you the different possible labelings ordered by probability. The idea is to select the first labeling which contains a single entity, i.e. the most likely option after eliminating the predictions with several entities.

  In CRF States of the neighbors affect the current prediction so predicting 1 entity per sentence seems difficult specially with CRF?

There's a confusion here: this is true but the "neighbors" are not other entities, they are the other words of the sentence. This means that if a word belongs to an entity then the next word is more likely to belong to an entity as well (i.e. to be part of the same entity). So using CRF doesn't increase the probability to find several distinct entities in a sentence. However if all the target entities are single words then CRFs might not be needed (I'm not sure whether there would any better alternative though).

  if i Cannot achieve this with CRF can I use Bert to train a model having 1 entity per model?

I don't know the answer to this one but I doubt it's needed.


First of all I would look how many missing values there are in the column. If there are too much (~20%, generally its difficult to say how much is too much), I would drop the column because imputing 20 % of your data (without prior expert knowledge) or even more probably does not give you meaningful information anymore.

Secondly I would look at correlations between missing values and other features. Maybe you are lucky and there is some correlation between missing values in column x and a categorical value in column y. Simply look at conditional distributions.

If you choose to impute, check the distribution of the categorical values for non-missing entries. If the distribution is heavily skewed, say there is 95% value 0 and only 5% value 1, you can use the median to impute. Again, the question is how informative this is in the end. Otherwise create an additional categorical value which simply represents a missing value.


How to interpret classification output - Predective model

Your model is decent. Predictions are better than random chance.
Macro average is a simple average of the performance measures (precision, recall etc) across all classes/labels.
Micro average is a weighted average of the same , where the weights are based on the number of samples per class/label (support column).


CV (Curriculum Vitae) Recommender System using Machine Learning, Python, Apache Solr(Back-end), AngularJS (Front-end)

I guess, there are many ways, to recommend CVs, but here is what I would do:

I would use TF-IDF or LSI, so an unsupervised learning approach and I would use a corpus.

I would apply word stemming, create a dictionary containing unique word stems of the CVs, create a corpus containing word stem id from dictionary and word stem count for each word stem in each CV, so the document frequencies (DF) for all word stems, then create TF-IDF (term frequency inverse document frequency) model or an LSI (latent semantic indexing) model from dictionary and corpus. Then you have TF-IDF or LSI vectors for all CVs.

For matching and recommending, you apply word stemming to text input you want to match CVs with, calculate the LSI or TF-IDF vector for this input and match it with the most similar CV by using cosine similarity calculation.

Here and here you can find short Python code examples for the approach I described using LSI or TF-IDF.


Try using numpy's log or log1p. The math module also offers logarithmic functionality.

You can use sklearn to transform your variables - see MinMaxScaler, StandardScaler, etc.

Other transformations like Box-Cox can be found in scipy.stats


You need normal data to train on. If you have abnormal instances also, those should be excluded from the training set. Having access to labeled abnormal/normal data is very useful for the validation and testset.
Anything that differs from the normal data (as learned by the autoencoder) is considered an anomaly.

If you have a lot of labeled abnormal and normal data, then you can consider using binary classification instead.


Where can I find a software library for pairwise matching (ideally, Python, R, Java)?

If you can transform those sentences into number vectors (e.g. into a bag of words or tf-idf representation), I guess you could use k-Means or hierarchical clustering functionality from Orange, a GUI and machine learning library written in Python.

It also has an add-on for text mining specifically, but I cannot attest to it as I haven't tried it yet.


What methods exist for distance calculation in clustering? when we should use each of them?

Well, there is a book called

  Deza, Michel Marie, and Elena Deza. Encyclopedia of distances. Springer Berlin Heidelberg, 2009. ISBN 978-3-642-00233-5

I guess that book answers your question better than I can...

Choose the distance function most appropriate for your data.

For example, on latitude and longitude, use a distance like Haversine. If you have enough CPU, you can use better approximations such as Vincenty's.

On histograms, use a distribution-baes distance. Earth-movers (EMD), divergences, histogram intersection, quadratic form distances, etc.

On binary data, for example Jaccard, Dice, or Hamming make a lot of sense.

On non-binary sparse data, such as text, various variants of tf-idf weights and cosine are popular.

Probably the best tool to experiment with different distance functions and clustering is ELKI. It has many many distances, and many clustering algorithms that can be used with all of these distances (e.g. OPTICS). For example Canberra distance worked very well for me. That is probably what I would choose as "default".


why is SVM cost function the norm of the hyperplane parameters, and not the mean square error?

Given a linearly separable data set, the intuition behind the SVM-algorithm is to find a hyperplane that seperates both classes and maximizes the margin (which is twice the distance of the hyperplane to the closest data point).

So from all hyperplanes that seperate both classes, we want to find the one with the biggest margin (to achieve the best generalization).

Let's imagine we have two parallel seperating hyperplanes defined by the vector $\theta$ and with no data points in between, their distance is then given by: $$m= \frac{2}{\|\theta\|}$$
To maximize this distance, the two hyperplanes must pass through data points of the opposing classes.
In this case we can define another hyperplane in the middle (with same distance to both hyperplanes and the same vector $\theta$) that now has the margin $m$.

As you can see from the formula: the smaller $\|\theta\|$ the greater the margin $m$.

So maximizing the margin $m$ is equivalent to minimizing $\|\theta\|$. Thus, a separating hyperplane is optimal if it minimizes $\|\theta\|$.

This is just the rough intuition - if you want to go deeper into the maths behind it, I could recommend: http://www.svm-tutorial.com/2015/06/svm-understanding-math-part-3/


Choosing between CPU and GPU for training a neural network

Unlike some of the other answers, I would highly advice against always training on GPUs without any second thought. This is driven by the usage of deep learning methods on images and texts, where the data is very rich (e.g. a lot of pixels = a lot of variables) and the model similarly has many millions of parameters. For other domains, this might not be the case.

  What is meant by 'small'? For example, would a single-layer MLP with 100 hidden units be 'small'?

Yes, that is definitely very small by modern standards. Unless you have a GPU suited perfectly for training (e.g. NVIDIA 1080 or NVIDIA Titan), I wouldn't be surprised to find that your CPU was faster.

Note that the complexity of your neural network also depends on your number of input features, not just the number of units in your hidden layer. If your hidden layer has 100 units and each observation in your dataset has 4 input features, then your network is tiny (~400 parameters). If each observation instead has 1M input features as in some medical/biotech contexts, then your network is pretty big in terms of number of parameters. For the remainder of my answer I'm assuming you have quite few input features pr. observation.

One good example I've found of comparing CPU vs. GPU performance was when I trained a poker bot using reinforcement learning. For reinforcement learning you often don't want that many layers in your neural network and we found that we only needed a few layers with few parameters. Moreover, the number of input features was quite low. Initially I trained on a GPU (NVIDIA Titan), but it was taking a long time as reinforcement learning requires a lot of iterations. Luckily, I found that training on my CPU instead made my training go 10x as fast! This is just to say that CPU's can sometimes be better for training.

  Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?

It's important to note that while on a GPU you will always want to fill up the entire GPU memory by increasing your batch size, that is not the case on the CPU. On the CPU an increase in batch size will increase the time pr. batch. Therefore, if it's important for you to have a very large batch size (e.g. due to a very noisy signal), it can be beneficial to use a GPU. I haven't experienced this in practice though and normally small batch sizes are preferred.


Is the percepetron algorithm's convergence dependent on the linearity of the data?

Yes, the perceptron learning algorithm is a linear classifier. If your data is separable by a hyperplane, then the perceptron will always converge. It will never converge if the data is not linearly separable.

In practice, the perceptron learning algorithm can be used on data that is not linearly separable, but some extra parameter must be defined in order to determine under what conditions the algorithm should stop 'trying' to fit the data.

For example, you could set a maximum number of iterations (or epochs) for the algorithm to run, or you could set a threshold for the maximum number of allowed misclassifications.


Simple way to visualise word2vec vector space

You can easily visualize word2vec vectors using TensorBoard, which is very easy to understand.

I have made a video about using TensorBoard.


Try t-SNE (http://lvdmaaten.github.io/tsne/), it can help you group pictures that have the same properties via dimensionality reduction. You can see an example here http://cs.stanford.edu/people/karpathy/cnnembed/ where t-sne is applied on various imagenet samples.

This will group images with similar content close to each other (dogs near dogs, black dogs near black dogs...). However, if your black dog is yellowish because of the lighting, i suspect it will not be close to the same black dog under normal lighting condition. For that, maybe you can try black and white pictures? or add a black & white layer on top of your RGB input.

I did not experiment it myself, but you may also want to have a look at DeepBit http://www.iis.sinica.edu.tw/~kevinlin311.tw/cvpr16-deepbit.pdf "an unsupervised deep learning approach to learn compact binary descriptor for efficient visual object matching".


Can embeddings generated by word2vec be similar for words which never share any words in the same sentence?

As you send a bag of words into the CBOW as an input, and it works based on the n-grams, as in your case words A and B do not share any co-occurrences with each other, it means their not presented in any shared n-grams, and their vectors should not be near each other at all.


What distance should I use for edges weights in textrank algorithm

As lib author said using lev-dist was just en experiment and might not work (and it really doesn't), but for some reason people use broken lib w/o questions ¯\_(ツ)_/¯


It seems that there is a way of using sample weights which requires a little more work than just using a single argument, see this stackoverflow answer.


How to find vertical clusters in 1-D data

You might find some more useful methods if you look for event detection or outlier detection, rather than clustering. Given the shape of the spikiness in the data, you could try using the mean value of the time series as a threshold, then contiguous chunks of values that are greater than the global mean will be your abnormal states. Now this method might be a little bit fragile in the long run, so you might want to make it more sophisticated. You could use the mean value over a large rolling window, rather than the global mean, if you are more interested in local changes.

If you wanted to keep things in a clustering framing, then it might work better if you did the clustering in 2D, i.e. your points will be (time, value) tuples/vectors, rather than doing the clustering on the 1D data. Another thing that would probably work would be to use K-means clustering on the 1D data, but only have 2 clusters (normal and abnormal) you would then have to use time information to separate individual events/anomalies.


Should I remove outliers if accuracy and Cross-Validation Score drop after removing them?

As a rule of thumb, removing outliers without a good reason to remove outliers rarely does anyone any good.  Without a deep and vested understanding of what the possible ranges exist within each feature, then removing outliers becomes tricky.  Often times, I see students/new hires plot box-plots or check mean and standard deviation to determine an outlier and if it's outside the whiskers, they remove the data points.  However, there exist a myriad of distributions in the world that if you did that, you would be removing perfectly valid data points.

In your example, it looks like your dealing with social media data.  If I were to sample 1000 users from a social media database and then plotted a box-plot to find "outliers" for number of likes a post gets, I can imagine that there could be a few so-called outliers.  For example, I expect my Facebook post to get a handful of likes on any given day, but when my daughter was born, my post related to that got into the hundreds.  That's an individual outlier.  Also, within my sample of 1000 users, let say I managed to get user Justin Bieber and simply like at his average number of likes.  I would say that he's an outlier because he probably gets into the thousands.

What outliers really mean is that you need to investigate the data more and integrate more features to help explain them.  For example, integrating sentimental and contextual understanding of my post would explain why on my daughter's birthday, I received hundreds of likes for that particular post.  Similar, incorporating Justin Bieber verified status, large following may help explain why a user like him receives a large number of likes.

From there you can move on to either building separate models for different demographics (average folks like me vs people like Justin Bieber) or try to incorporate more features.

TL;DR.  Don't remove outliers just because they are abnormal.  Investigate them.


How to force a NN to ouput the same output given a reverse input?

The most general and robust way is to double the training set using the "mirror image" of each sample. Also it is sound approach from a statistical learning perspective.
There can be some approaches to symmetry where one hacks the NN architecture (eg by symmetrising the network) but these are usualy very fragile and do not generalise well, while the above approach is robust and lets the network adjust itself as it needs.
A similar but cheaper approach is to sort the samples before feeding them to the network. Eg for your example:
If you have this input [1, 1, 1, 0, 1, 0] then the sorted related mirror input (in lexicographic order) is the [0, 1, 0, 1, 1, 1] (reverse). So you train the NN only the sorted lexicographic inputs and before testing each sample you convert it to the sorted lexicographic related input (by reversing it) if it is not already in that form.
So effectively you normalise the samples (by reversing them if necessary) both before training and before testing. This is similar approach to doubling the train set, only it is cheaper (and in practice exactly accurate, while doubling the training set is only theoreticaly accurate, in practice it may have discrepancies).


What you are trying to do is text normalization, which is a part of NLP pipeline.

To successfully preprocess your data, please try this below:

Install ekphrasis

pip install ekphrasis

Apply seg & spell correction:

from ekphrasis.classes.spellcorrect import SpellCorrector
from ekphrasis.classes.segmenter import Segmenter
sp = SpellCorrector(corpus="English")
seg_eng = Segmenter(corpus="English")

words_to_correct = ['Carpanter','Carepnter','Carpentor','Labourer','Labor','Labour','Housewife','House Wife','housewife.']
for word in words_to_correct:
    segmented = seg_eng.segment(word)
    corrected = sp.correct(segmented)
    print(word + " -> " + corrected)

Outputs as:

Carpanter -> carpenter
Carepnter -> carpenter
Carpentor -> carpenter
Labourer -> labourer
Labor -> labor
Labour -> labour
Housewife -> housewife
House Wife -> housewife
housewife. -> housewife

Note: Check this amazing work and paper if you are interested in!


I've seen some suggestions elsewhere, but they are old and do not apply very well to newer TF versions. What worked for me was this:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

When that variable is defined and equal to -1, TF uses the CPU even when a CUDA GPU is available.


Problem with important feature having a lot of missing value

What is the best way to deal with this kind of missing value problem can only be answered empirically? It will vary depending on your dataset and algorithm of choice. But here is a few things you can try.

Impute the missing value

Impute missing value of the mean
Impute missing value with special values. For example, If the variable takes only positive values, then you can encode the missing value as 0.

Try to predict the missing value

use the other variables to predict the missing value.
(However, if you can actually make a good prediction of the missing value out of the rest, then this might suggest you can drop this variable altogether)

  Another one is to include both the binary variable along with the
  original variable in the model with missing values replaced by some
  imputed values.

There is nothing inherently wrong with this method. You should certainly try. One possible downside of this method when you have 99% of value missing, this original variable is going to highly correlated with the derived is_missing variable. This can be problematic depends on the particular classification algorithm you are using. For example:

It is widely known that multicollinearity is a huge problem for any variant of linear regression
Support Vector Machine suffers a similar problem [Ref]
Naive Bayes assumes independence among features. This is even stronger.
The is_missing variable is a categorical variable, this makes it tricky to define a distance metrics in K-Nearest-Neighbour algorithm


Is there a safe and simple way to estimate a standard deviation for a next subset?

Basically, the two methods you are proposing are the same.

The first one is computationally more consuming, but they are the same.

In the first method you are calculating $\sigma$ generating possible values of random variable with already the same $\sigma$ you have had historically. This is the same as calculating $\sigma$ with all the historical data you have.

In the second method you are doing a stimation with limited data, this is the correct way unless you have sufficient amount of data to estimate a GARCH model.

A GARCH model is a statistical model for time series data that describes the variance of the current error term or innovation as a function of the actual sizes of the previous time periods' error terms.

Meaning:
$\sigma_t^2 = w+\alpha_1\epsilon_{t-1}^2+...+\alpha_q\epsilon_{t-q}^2+\beta_1\sigma_{t-1}^2+...+\beta_p\epsilon_{t-p}^2$

This model requires a sufficient amount of data and knowledge on time series analysis, of the two options you posted, I would use the second with as much data as possible.


Binary classification of similar images with small region of interest

I would use a CLAHE preprocessing, SIFT Image features and mask the wrong detected keypoints out as geometric constrains. then i would count sift keypoints in a image without and keypoints to get a thredhold boundary and classify by amount of robust keypoints or logistic regression


Should dimensionality reduction be done before k-means clustering if there are many features?

For the first idea about PCA, you can not simply just use 2 components. You need to take a look at the explained variance by your principal components and based on that you should select the required number of components. If, for example, you found that the first two components explain a significant amount of variance (e.g., more than 95%), then, you can use them to perform k-means clustering. In that case, it is expected (but not for sure and for all cases) that you get the same results as when you perform k-means using all features.

My suggestion is to use all of your features if you need to consider a large number of components. My reason is that your dataset is too small and it is not computationally demanding to perform k-means clustering using your dataset.

As a side note, also you can try all of your options because your dataset is small and then you will find out what's going on in your data.


Deep Learning with Spectrograms for sound recognition

RNNs were not producing good enough results and are also hard to train so I went with CNNs.

Because a specific animal sound is only a few seconds long we can divide the spectrogram into chunks. I used a length of 3 seconds. We then perform classification on each chunk and average the outputs to create a single prediction per audio file. This works really well and is also simple to implement.

A more in-depth explanation can be found here:
http://ceur-ws.org/Vol-1609/16090547.pdf


Should I remove features that occur very rarely to build a model?

Short answer:

yes, if they occur so rarely, they can only lead to overfit, so it's better to ignore them as features.

Longer answer:

Usually one puts all those unique occurrence in a single feature/token, and that's the way I suggest you to proceed. So if you have two features that appears only once, you can join them to create a feature that appears twice.

But anyway, 3000 short text are too few to create a NLP model that can work well.
To mitigate your lack of data, you can use pre-trained word embeddings like the ones available here. Doing so you can also keep most of the single occurring words/features, because in those word embeddings they already have a defined semantic meaning.


Which of the scikit learn classification algorithms accept Sparse matrices?

From this kaggle discussion, the classification algorithms from scikit-learn that support sparse matrices are at least:

linear_model.LogisticRegression()
svm.SVR()
svm.NuSVR()
naive_bayes.MultinomialNB()
naive_bayes.BernoulliNB()
linear_model.PassiveAggressiveClassifier()
linear_model.Perceptron()
linear_model.Ridge()
linear_model.Lasso()
linear_model.ElasticNet()
linear_model.SGDClassifier()

Also, from this quora question, you can check in the implementation of the algorithm if they import scipy.sparse.csr_matrix .


Check out the indicator function $1_S(\cdot)$. In your case it would be fined as

$$1_S: \{1, \ldots, d\} \rightarrow \{0, 1\}, j \mapsto \begin{cases}
1 & j \in S \\
0 & \, \text{else}
\end{cases}.$$

Multiplying this function with the respective values should give you what you are looking for.

Edit: If I understand your comment correctly, the vector you are looking for is

$$ \alpha_s = \sum\limits_{i = 1}^{d} \alpha_i e_i 1_{S^C}(i). $$

Here $e_i$ denotes the i-th unit vector of $\mathbb{R}^d$, $S^C$ is the complement $\{1, \ldots, d\} \setminus S$ in $\{1, \ldots, d\}$ and $1_{S^C}$ is the indicator function that is $1$ for $i \notin S$ and $0$ for $i \in S$.


If I understand your question correctly, you need to recommend one product out of 2000. You want to pick the one with the highest revenue expectation. This can be decomposed in the probability of making the sale times the revenue given that is a sale. The revenue for a sale is known, the probability is what is needed. I assume you have historic data of which customers (with the given features) bought which of these products. Now you can train regression models to estimate the probability of a sale for a specific product. Logistic Regression and Neural Networks with a sigmoid activation functions are well suited for unbiased probability estimates, by optimizing the log-loss cost function.

With this amount of data I propose to start on a subset of your data and using Logistic Regression. Since this is a generalized linear model you will likely need to do some feature engineering to increase performance. Since you mentioned you use Python, scikit-learn has a lot of functions for Logistic Regression optimization and feature engineering so take a look at that.

Once you have trained 2000 models, for a new recommendation you would run the features through the models, get all the probabilities, multiply them with revenue corresponding to this product and pick the highest one.


How to train a Model which can check whether image is from existing classes or not

There is an example of classification in this repository:
DogVsCat


Where are the 60 million params of AlexNet?


Random Forest: how efficient is it for features in a range of values (not binary thresholding)?

Is it correct that each feature is only used once in one decision tree?

No, if the variable is a continuous value like age, it can use threshold splits at each level. So, same feature can be used multiple times in any given branch of the decision tree.
And since the random forest is nothing but collection of decision tree it follows the same principle.
Reference


Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

As far as I understood your question you can work with NLTK.

Have a look on the following functions:

Tokenize
Stemmers
pos_tags (if you wanna categorize by Verb, Noun etc. beforehand)

And to come to your question of similarity or relative word/phrase usage you have FreqDistfunction in NLTK. Using this Output you could set each occurance relative to the length of the list that is returned by tokenize


Is it possible to do Normalization before Xgboost?

Xgboost is an ensemble algorithm based on decision trees, so doesn't need normalization.
You can check this on Xgboost official github: Is Normalization necessary? and this post What are the implications of scaling the features to xgboost?

I'm new in this algorithm but I'm pretty sure of what I've written


How to view entire 5788 rows and 7 columns.?

pandas has a max rows setting - https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html

Though perhaps looking at a 5,000+ row csv in an editor, or a spreadsheet or some IDEs have a csv editor would be more useful.


Ways to deal with longitude/latitude feature

Lat long coordinates have a problem that they are 2 features that represent a three dimensional space. This means that the long coordinate goes all around, which means the two most extreme values are actually very close together. I've dealt with this problem a few times and what I do in this case is map them to x, y and z coordinates. This means close points in these 3 dimensions are also close in reality. Depending on the use case you can disregard the changes in height and map them to a perfect sphere. These features can then be standardized properly.

To clarify (summarised from the comments):

x = cos(lat) * cos(lon)
y = cos(lat) * sin(lon),
z = sin(lat)


When Should I Use ggplot2 Instead of Tableau?

There is one big economic difference between the two: ggplot2 is an open source package for an open source programming language. On the contrary Tableau is a proprietary software. That might be a dependency that you might not want to risk, e.g. you do everything in Tableau but then the license gets more expensive and your organization does not want to spend money for it anymore. Academics would often stress this reason, another reason of low adoption there is that Tableau has always been the BI tool and R the "research" tool.

Apart from this, they are simply different tools (in the same domain of data viz) and as such will be suited for different problems. This can be subject to personal preference, but I would say that there is a few basic considerations:

Speed: how fast can you get what you need?

Use the tool that gets you results. Tableau might often be faster.

Reproducibility & automation:

If you know you might want to plot this chart many times in future in a consistent manner or embed your plots in some sort of notebook like R Markdown or dashboard, then R is most likely the better choice. You write the code once and produce thousands of plots within seconds if you need to.

Type of project:

If you are starting with messy data, or you are doing some modeling work, it seems as a good strategy for me to just stick to R, I see it as more efficient to have everything in one place.
If you need to get awesome visualizations from clean data, Tableau will excel
Based on the type of visualizations and output you want, one or the other might be better, ggplot2 is extremely flexible, Tableau has a lot of amazing functionality which is ready to go.


Since there are no predefined topics, the task is unsupervised: the goal is to group tweets which are semantically similar together (as opposed to classification, which requires training a model to predict among specific classes).
The standard unsupervised approach is topic modelling. In the traditional LDA approach, a topic model groups the documents into clusters and also provides the probability of a word given a topic, so a list of "top words" by topic can be extracted from the model. LDA requires the number of topics as input parameter but Hierarchical Dirichlet Processes can be used to avoid this issue (it's less common however).


What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

Yes learning online from different tutors can lead to different strategies. But this stats that the flow depends upon the type of data. Not everytime you have to create features or deal with outliers.

In case of images you reduce noise by applying different methods & filters, in case of text you remove stopwords, punctuation etc. So it depends upon data & its amount. General way to proceede for me is as follow

Understand definition & get domain knowledge
Gather data
Pre-process it (Involves removing noise, applying transformation, pruning data etc)
Do feature selection if necessary
Do data visualisation
Do different model experiments with sample data.
Fix 2 3 models for case
Train with real data
Use different matrices to evaluate (depends upon the data & definition)
Make model accessible using API/Direct to client

This is how in industry i have seen everyone around me works & i follow the same. Has helped me but this is way general, here you have much to change based upon definitions & tasks.


Prepare JSON data from sentiment analysis to perform Logistic Regression

Reputation barrier hence mentioning this as an answer, instead of a comment. This article should lead you the right way, as author has used Twitch API and Python to create his dataset. You shall get a feel of entire data flow.

EDIT: This is a nice tutorial on understanding (basically read/write) JSON files using Python.


How to interpret Variance Inflation Factor (VIF) results?

The Variance Inflation Factor (VIF) looks at how well a single $x_i$ is determined by all the other $x_i$ (jointly) in your model.
How does the VIF work?

For each $x_i$ in your model, you run a (auxiliary) linear regression:
$$ x_{1,i} = \beta_1 + \beta_2 x_{2,i} + ... + \beta_n x_{n,i} + u .$$
You retrieve the $R^2$ for each of these models and calculate the $VIF$:
$$ VIF_1 = 1 / (1-R^2_1). $$

Example in R:
Calculate VIF:
library(car)
library(ISLR)
reg = lm(mpg~disp+wt+qsec+hp, data=mtcars)
vif(reg)

Result:
    disp       wt     qsec       hp
7.985439 6.916942 3.133119 5.166758

Do this manually (for disp)
rsq = summary(lm(disp~wt+qsec+hp, data=mtcars))$r.squared
1/(1-rsq)

Result:
7.985439

What about the $VIF=10$ rule of thumb?
$VIF = 10$ is equal to having an $R^2=0.9$ in the auxiliary regression in step 1 above (because $1/(1-0.9)=10$). This means that your other $x_i$ (in the model) explain the $x_i$ under consideration to a large extent (90% if you want to say so). This of course is just a rule of thumb.
In essence, the $VIF$ boils down to the question: "How well is one of my $x_i$ explained by all other $x$ jointly".
In your example tax has the highes $VIF$ (tax=7.279426). This means that the auxiliary regression (step 1) for tax has an $R^2=0.862627$. This means that tax is well explained by all the other $x$ so that there may be a problem with multicollinearity.


How to count categorization instances in a NN?

But your intuition about regression is right. Basically, you have multi target regression problem. Each category is a separate target. If your get float output as prediction, you can round it. Float output also makes sense, because it shows you how confident is your model about prediction and the expectation of the number of bought items.
Number of bought items is structured as ordinals, so it makes sense to predict it as multi target regression. Moreover, you may not know the upper limit of the number of bought items, so it makes no sense to predict number of items as categories.
Look at this for reference:

Neural Network for Multiple Output Regression
Resources for learning about multiple-target techniques?


Not Access to Confusion Matrix in SVM.SVC.score Scikit-learn Python

Instead of using the score method on your trained model, you should use the predict method.

You can then pass the results into the confusion matrix function from sklearn:

from sklearn.metrics import confusion_matrix
y_pred = svmObject.predict(X)
cm = confusion_matrix(y_true, y_pred, sample_weight=sample_weight,
                      labels=labels, normalize=normalize)

There is also a nice function called plot_confusion_matrix:

from sklearn.metrics import plot_confusion_matrix

plot_confusion_matrix(svmObject, testXtr, testY)
plt.show()


Yes, the built-in models pull from S3 and write the computed model back to S3.


Adding high p-value and low R square features in linear regression model to improve result

In general, adding more features will increase the quality of model fit.
If your goal is best fitting modeling, add as many features as possible (regardless of p-value).
Sometimes people care about parsimonious models, they are will to lower the overall model fit because they also value a simpler model. Then they apply a threshold to features using p-values.


Multilabel image classification: is it necessary to have traning data for each combination of labels?

Both cases will work. The point that you have to consider is that you shouldn't use Softmax activation in the last layer because your classes have intersection and they are not mutually exclusive. You should employ Sigmoid activation function and there should be two of them which each of them can be one or zero as the output vector. Consider that in this case you should not compare each activation to the other. Each one shows the probability of existence of the corresponding object, cat or dog.

For your case I suggest you to provide images which don't have either and in some cases have both. The former is more important. Because if you don't provide negative labels, not existing, the net always try to make something out of nothing. Means the net always try to flag similar patters as cats or dogs.


what is difference between the DDQN and DQN?

In particular, DQN is just Q-learning, which uses neural networks as a policy and use "hacks" like experience replay, target networks and reward clipping.

In original paper authors use convolutional network, which takes your image pixels and then fit it into a set of convolutional layers. However there are a couple of statistical problems:

DQN approximate a set of values that are very interrelated (DDQN
solves it)
DQN tend to be overoptimistic. It will over-appreciate being in this state although this only happened due to the statistical error (Double DQN solves it)

$$Q(s,a) = V(s) + A(s,a)$$

By decoupling the estimation, intuitively our DDQN can learn which states are (or are not) valuable without having to learn the effect of each action at each state (since it’s also calculating V(s)).

We’re able to calculate V(s). This is particularly useful for states where their actions do not affect the environment in a relevant way. In this case, it’s unnecessary to calculate the value of each action. For instance, moving right or left only matters if there is a risk of collision

As @emilyfy said self.target_model.set_weights(self.model.get_weights()) - the updating of the target model.


Differences of get_dummies and labelbinarizer?

A couple of things comes to mind.

get_dummies can transform a dataframe with many columns, whereas LabelBinarizer will only do one column.

get_dummies outputs a dataframe (if the input is a dataframe) with a nicely formatted columns, whereas LabelBinarizer outputs a numpy array, so if you want to attach labels to them, you'd need to get them from the fitted instance of the LabelBinarizer.

Inverse transform is more intuitive with LabelBinarizer, it has a method named inverse_transform, whereas with get_dummies you would need to do something like dummies.idxmax(axis=1)

So overall, get_dummies seems to be a better choice


Any research on segmentation of non-text contents out of (mostly) text-documents?

Although I didn't implement it so far, I am pretty sure natural language text vs code snippets is easy:

For each block, you make compare the distribution of characters to ground-truth natural language text vs. code. See my paper The WiLI benchmark dataset for written language identification, page 4 "Single-Character Frequency Analysis".


Model for predicting duration based on categorical data

So, from what I understand of the question, you are asking how to model the duration of a job, given the input (which includes City and ClientType).
In this case you can use something like a feedforward neural network to model this problem. You might find, that using this methods, the level of prediction error will be lower than that produced by your model, which could act as a baseline to see if these models do work better for your problem.
When representing each of the categorical variables, we use something called one-hot encoding. Then we concatenate these categorical variables into one n-dimensional input vector, which represents all the features for one example.


Difference between OrdinalEncoder and LabelEncoder

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while LabelEncoder is for converting target variable.

That's why OrdinalEncoder can fit data that has the shape of (n_samples, n_features) while LabelEncoder can only fit data that has the shape of (n_samples,) (though in the past one used LabelEncoder within the loop to handle what has been becoming the job of OrdinalEncoder now)


Why Extra-trees should only be used within ensemble methods?

In a random forest tree, a random subset of features is available for consideration at each split. Extra-trees takes this a step further by using a random threshold at each split. The idea is that a forest (an ensemble of trees) with a large number of trees that have learned something different from each other (due to the random nature of the features and thresholds at each split) will result in a better prediction than any single tree.

If one were to use a random threshold for each split (what extra-trees do) in a single tree, one would likely end up with a suboptimal tree that predicts poorly.


Neural Network backpropagation taking forever

To make this question answerable, I have to assume that you have implemented back propagation correctly. That is, in terms of the underlying maths, the steps are in the right order, there is not any non-necessary repetition, and the values are calculated using correct formulae.

You can check whether the gradient values are correct by implementing gradient checking for weights, which in general you do by taking a random test case (any network architecture, random weights, inputs and target values), run your back propagation code to get your gradients, then for each weight in turn run the network forward, measure error, then adjust the weight by a very small delta, measure error again and use the difference in error to get an estimate for the gradient. If the estimate and your back propagation agree - approximately - for all weights, then your back propagation code is likely correct. If you are going to implement just one unit test for a NN implementation, this is the one to do.

To address the question directly: "Is there a more efficient way to approach backpropagation?", the answer is complicated.

The basic answer is "no". You absolutely have to do all the calculations involved in back propagation, there are no shortcuts.

However, most neural network frameworks will perform much better than your implementation, where you give an example of a network that has layers [39,39,39,1] training on 10,000 examples and taking 28 minutes to process a single epoch.

As you don't link your implementation, it is not possible to figure out precisely what the difference is. However, ignoring the possibility of bugs, the following things will probably apply:

Avoid calculating results using loops in an interpreted language. If you are looping through neuron or weight indices to multiple some values together and store back in an array in a high-level language, then this can be very inefficient. Most frameworks take advantage of fast loops and parallelisation that is possible if you frame the neural network processes (both forward and back propagation) in terms of matrix and vector manipulations. This is sometimes called vectorizaton, and most frameworks split this up so that the vector and matrix operations are black box routines implemented with strong attention to optimisation in a low-level language (like C), whilst passing the results around and deciding what to do with them can be done in higher-level language like Python or Matlab.
Mini-batches. You have to process forward and back propagation for each example seen by the network. But you don't have to apply weight updates for each example. It turns out that applying an update only once every so many examples is not only more efficient in terms of calculations, but also has benefits for convergence. Frameworks that support large amounts of parallel processing via GPU can also improve performance greatly from working in batches.

It is unlikely you will find the time to explore optimisation to the extent that the better resourced public frameworks already have. However, you can get quite far by taking advantage of vectorisation options built into your language - for instance using matrix operations in Matlab, or Numpy features in Python. This might be worthwhile so you can continue to work on and understand low-level detail in neural network models. However, at some point, if you want to work on practical problems instead of your own NN library, you will find yourself naturally switching to a third-party library because the optimisation problems and many useful additions have been solved for you already.


Segmentation of time series into one-second non-overlapping windows

This answer has an implementation for audio spectrograms, but it is exactly the same for any multivariate time-series.
Here is the most relevant part of the code. To not have any overlap, set window_hop = window_size.
window_size = 64
window_hop = 30
start_frame = window_size
end_frame = window_hop * math.floor(float(frames.shape[1]) / window_hop)

for frame_idx in range(start_frame, end_frame, window_hop):
    window = frames[:, frame_idx-window_size:frame_idx]


ValueError: Numpy arrays that you are passing to your model is not the size the model expected

Your network has two Inputs, so you need to pass in two arrays.  From the rest of the code, perhaps you want to pass the same data through both strands, in which case you can just duplicate the input:

history=model.fit([trainDataVecs, trainDataVecs], ...


How to create a confusion matrix for one node of a decision tree?

CM is about True Vs Predicted.
Since only one node is in the discussion which means we will have only one column of "Predicted" but all the possible rows of "True"
Let's assume 100 samples in the Node. The Node must be classified as "A"
Below should be the CM - Rows are "True" and the Column is the "Predicted"
\begin{array} {|r|r|}
\hline
 &A &B &C &D &E &F \\
\hline
A &62 &- &- &- &- &- \\
\hline
B &21 &- &- &- &- &- \\
\hline
C &13 &- &- &- &- &- \\
\hline
D &00 &- &- &- &- &- \\
\hline
E &02 &- &- &- &- &- \\
\hline
F &02 &- &- &- &- &- \\
\hline
\end{array}


You can find below a worked out example (on a univariate version for the temperatures attribute) which, using statsmodels ARIMA, does not even finish the excution after hours trying for it.
An option you have regarding the performance issues is to actually use another efficient library like Keras with Tensorflow modelling an LSTM useful for time series data, which is capable of dealing with so many samples without performance issues.
This dataset contains 420551 samples, which is very costly with the ARIMA statsmodels approach via walk-forward validation...
You can extract the temperature values for carrying out the univariate time series forecast.
See below an example (based on this tutorial) where leaving hyperparametrization aside for now, you can train a nearly half a millon time series samples without performance issues (you can try on google colab for faster checks for free).
From the example in the tutorial, you should pay special attention at:

the way the data is splitted:

def univariate_data(dataset, start_index, end_index, history_size, target_size):
data = []
labels = []

start_index = start_index + history_size
if end_index is None:
  end_index = len(dataset) - target_size

for i in range(start_index, end_index):
  indices = range(i-history_size, i)
  #### Reshape data from (history_size,) to (history_size, 1)
  data.append(np.reshape(dataset[indices], (history_size, 1)))
  labels.append(dataset[i+target_size])

return np.array(data), np.array(labels)

where, using 20 samples as recent history to predict the next value you can use:
univariate_past_history = 20
univariate_future_target = 0

x_train_uni, y_train_uni = univariate_data(uni_data, 0, TRAIN_SPLIT,
                                       univariate_past_history,
                                       univariate_future_target)
x_val_uni, y_val_uni = univariate_data(uni_data, TRAIN_SPLIT, None,
                                   univariate_past_history,
                                   univariate_future_target)

having something like:

the way the model is defined:
simple_lstm_model = tf.keras.models.Sequential([
tf.keras.layers.LSTM(8, input_shape=x_train_uni.shape[-2:]),
tf.keras.layers.Dense(1)
])

which comparing to a baseline model by predicting simply the mean of the history values, you have:

baseline model predicting based on simply the mean of past values:

VS

predicting wih the trained LSTM model:


Error when trying to run RandomForestClassifer with Pipieline and GridSearch

In addition to adding a '_' I had to replace RF by randomforestclassifier in all grid params - Please see link below.
https://stackoverflow.com/a/62166539/9592484


Tensorflow Adjusting Cost Function for Imbalanced Data

This seems like a good solution for the loss function. I've had success with a similar approach recently, but I think you'd want to reorder where you multiply in the class_weight.

Thinking about it logically, the class_weight will be a constant w.r.t. the output, so it will be carried along and applied to the gradient in the same way it's being applied to the cost function. There is one problem though.

The way you have it, the class_weight would affect the prediction value. But you want it to affect the scale of the gradient. If I'm not wrong I think you'd want to reverse the order of operations:

# Take the cost like normal
error = tf.nn.softmax_cross_entropy_with_logits(pred, y)

# Scale the cost by the class weights
scaled_error = tf.mul(error, class_weight)

# Reduce
cost = tf.reduce_mean(scaled_error)

I'd be very interested to know how this performs in comparison to simply oversampling the underrepresented class, which is more typical. So if you gain some insight there post about it! :)

Interestingly I successfully used a very similar technique in a different problem domain just recently (which brought me to this post):

Multi-task learning, finding a loss function that "ignores" certain samples


Evaluating correlations between dichotomous/binary variables? What test?

You want to be using a chisq test instead of looking at correlation. Correlation isn't well defined for categorical variables, since it relies on variance and covariance, which doesn't really make intuitive sense in the categorical (binary) case


When is feature transformation required?

Even if there are models that are robust w.r.t. feature transformations (like Random Forest), in general it's a good practice to transform the features in order to have better performance in a ML model.

For three reasons:

1) Numerical stability: computers cannot represent every number, because the electronic which make them exist deals with binaries (zeros and ones). So they use a representation based on Floating Point arithmetic. In practice, this means that the numerical behavior in the range [0.0, 1.0] is not the same of the range [1'000'000.0, 1'000'001.0]. So having two features that have very different scales can lead to numerical instability, and finally to a model unable to learn anything.

2) Control of the gradient: imagine that you have a feature that spans in a range [-1, 1], and another one that spans in a range [-1'000'000, 1'000'000]: the weights associated to the first feature are much more sensitive to small variations, and so their gradient will become much more variable in the direction described by that feature. This can lead to other instabilities: some values of learning rate (LR) can be too small for one feature (and so the convergence will be slow) but too big for the second feature (and so you jump over the optimal values). And so, at the end of the training process you will have a sub-optimal model.

3) control of the variance of the data: if you have skewed features, and you don't transform them, you risk that the model will simply ignore the elements in the tail of the distributions. And in some cases, the tails are much more informative than the bulk of the distributions.


Statistical programs, such as R, typicall use Least Squares estimation. It's a deterministic algorithm that makes a linear model find its optimal tuple of parameters. Because of this, you don't have to worry about the choice of a loss function.

In case you wanted to train your linear regression with a gradient descent algorithm, instead, you'd have to specify a loss function to run it. Classical loss functions for regression are: Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE).


How to compare a machine learning model and a rule based model

For the rule base model I would take the exact same approach as for ML model, split the data into train and test set since at the end you want to check whether or not your rule base and ml models generalize well
It is also recommended to used the same sets to train and evaluate both models in such a way that it is a fair comparison.


How would I apply anomaly detection to time series data in LSTM?

I've been in that situation before,

there's this article on medium where the guy uses keras,tf for predicting credit card fraud detection using autoencoders which have Dense layers, but you can try the same with LSTM, can't say for sure whether it will work, but if in case it doesn't work, please try Conv1d because nowadays convolutional networks are more promising than LSTMs and GRUs -> source

Even if autoencoder doesn't help, that guy has visualizations for seeing anomalies, you must at least try them.

Hope this helps.


You could use the tools from Network Science (1) to model (A,B) as edges in a network. Since you have timing information on the transactions you can reconstruct paths on that network.

Using statistical network analysis you can find for example which points A from your dataset (or nodes in the network) are of importance, as in how many routes lead to that point. For this you'd study a centrality measure (degree centrality, closeness centrality, betweenness centrality, etc.) (2)

Or you can find which points are close together in terms of transactions between them, for this you'd look for community detection or clustering. (2)

You can also look at the timing data, to calculate the intra- and inter- transaction time distributions.

(1): Barabási, A. L. (2016). Network science. Cambridge university press.

(2): Salter‐Townshend, M., White, A., Gollini, I., & Murphy, T. B. (2012). Review of statistical network analysis: models, algorithms, and software. Statistical Analysis and Data Mining: The ASA Data Science Journal, 5(4), 243-264.


Using a K-NN Classification Approach for Time Series Data?

It seems pretty clear from looking at the data when an event starts and ends(basically whenever there is a sequence of positive values). So, instead of starting with some complicated models, I'd suggest calculating a few simple features (like length of the event, total amount of water, amount/seconds, time to previous event, time of day in seconds from start of recording) for every event and then try some clustering algorithm on that new data. k-NN might even produce something meaningful. But a statistical summary of the features can probably already give you a better idea of how to further approach this.

EDIT1

import pandas as pd
import numpy as np

rng = pd.date_range('2017-01-01 14:00:00', '2017-01-01 14:01:00', freq='S')
water = [0,0,0.2,0.3,0.4,0,0,0.3,0.2,0.5]*6+[0]
df = pd.DataFrame({'time_stamp':rng,'water_amount':water,'event_number':np.zeros(len(water))})

j = 1
for k in range(len(df)):
    if df.ix[k,'water_amount']== 0:
        df.ix[k,'event_number'] = -1
    else:
        if df.ix[k-1,'water_amount'] > 0:
            df.ix[k,'event_number'] = df.loc[k-1,'event_number']
        else:
            df.ix[k,'event_number'] = j
            j = j+1

df.groupby('event_number').agg({'time_stamp':'first',
                                'water_amount':'sum'}) #feature matrix

EDIT2

rng = pd.date_range('2017-01-01 14:00:00', '2017-01-01 14:01:00', freq='S')
water = [0,0,0.2,0.3,0.4,0,0,0.3,0.2,0.5]*6+[0]
df = pd.DataFrame({'time_stamp':rng,'water_amount':water})

starts = (df['water_amount']>0)&(df['water_amount'].shift(1)==0) #find all starts of events
n_events = sum(starts) #total number of events
df.loc[starts,'event_number'] = range(1,n_events+1) #numerate starts from 1 to n
df['event_number'] = df['event_number'].fillna(method='pad').fillna(-1) #forward fill all the values
df.loc[df['water_amount']==0,'event_number']=-1 #set all event numbers to -1 where the water amount is 0

df.groupby('event_number').agg({'time_stamp':'first',
                                    'water_amount':'sum'}) #feature matrix


How to visualize convolutional network focus on autoencoder?

You can visualise the activation maps of the various layers, to show how a deep CNN decomposes your input into more and more abstract building blocks. There is a good discussion on Cross Validated SE here. This explains the main idea and intuition as to how CNNs work and normally gets layman motivated to listen and learn more  :-)

There is another great idea names saliency maps - [Simonyan et al.]. As well as reading that paper, I suggest watching Lecture 12 of CS231n (slides here). The general idea of 'how to do this' is, I believe, can really be applied to most deep neural networks.

Here is a sample from the linked paper:

The steps are something like this:

take a trained network architecture
choose the layer that you are interested in (be it the final layer or an intermediate layer
set all weights in layers prior to your chosen layer to zero
perform backpropagation rom your selected layer back to your input

Now you will have a set of weights that match the dimensions of your input, but the values will be those of the weights at your selected layer, simply extrapolated backwards onto your input.

You can then use these weights with an input image, perhaps simply add these value to one or all of the colour channels, and you will have an image which has certain regions highlighted. The highlighted regions signify where the corresponding neurons of your selected layer are activated - i.e. what they focus on to make their contribution towards the final prediction.

Yet another approach is to systematically occlude (cover up) parts of your images, and track how the predictions fluctuate. The intuition is that, when you cover up a certain part of an image, and the prediction becomes terrible, you know your network really looks for that part, which you just covered up. The first time this was really published (as far as I can find) was in the 2013 paper from Zeiler and Fergus.

In the image below (from that linked paper), you can see how the grey square was shifted over and image to create a final heatmap, which highlights the critical portions of the image to getting the correct image classification.


Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

Amirhossein's blog post explains the intuition for positional encoding very well.
My takeaway from the blog is that: Consider just a pair of sinusoids (sine and cosine). Suppose we are within 1 full cycle (e.g. 0 to 2pi), the resulting encoding is basically guaranteed to be unique. I.e. there is a 1-to-1 mapping from real numbers (1, 1.5, 2, 2.34,etc.), x, to the pair of (sin(x), cos(x)). Thus, encoded to this 2-tuple vector.
Therefore, the purpose of the 10000 is probably just to make sure the full cycle is extremely large.
If you plot sin(1/10000*x), you will observe that to complete 1 full cycle, x >50k. This is more than sufficient to encode the words which would probably have <1k words.
Note that according to the original equations, the 1st pair of sinusoids is just sin(pos) and cos(pos). In this case, the positional encoding is arguably not unique. i.e. There may be two positions (note: they are integers) that has the same encoding, though this may only occur if we have a very long sentence such that the positions have exactly the same encoding. Though one may argue that since pi is irrational, it is unlikely any integer positions will have same encoding..