Why do popular search engines not follow the usual AND, OR logic for queries?

Nice question!

An exact answer should be given by looking in the search engine source code but here is a possible explanation.

I run the queries at Google

burglar 33,800,000
burglar AND burglar  29,200,000
burglar OR burglar 26,500,000

The results indeed do not respect the expected Boolean relation burglar AND burglar <= burglar OR burglar = burglar

However, that is since the search engine doesn't process the "and" and "or" as binary operator but just as search tokens.
Looking for them we get

And 25,270,000,000
Or  16,320,000,000

A term alone appears most times. "and" is more common than "or" so a term with "and" is more common than the term with "or"

Note that

burglar burglar 29,000,000

Apparently looking for documents in which the term appears twice.

By the way, Google's Search operators documentation claim that "OR" should indeed act as a binary operator.
You found a case in which they fail to do so.

Note that this behaviour is very specific to search engine.
In Bing you get the following results:

burglar 4,400,000
burglar AND burglar  1,610,000
burglar OR burglar 1,610,000
And 10,400,000,000
Or  3,750,000,000
burglar burglar 1,610,000

The number of results is similar for  "burglar AND burglar", "burglar OR burglar" and "burglar burglar" though we see that "And" is more popular than "OR". It seems that Bing treatments is the removal of "And" and "OR", possibly as stop words.

Bing documentation suggest the operators "&&" for "and" and "||" for "or".
 - burglar || burglar  4,400,000 = burglar
 - burglar && burglar  1,610,000 = burglar burglar

These results fit the claim that when a term appears twice in the search query it should appear at least twice in the document too.


Is there a way to calculate value from a radar (spider) chart?

You basically want to display the importance of each feature in the model as it relates to your final score. Rather than starting with the visualization, I would figure out the data first and then see which visualization fits. Here, they use a simple bar chart:

http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html


What kind of machine learning algorithm can I use?

A couple ideas:

If you have a large data set of tweets, you could try using latent semantic indexing to find out which terms are semantically related based on their usage and co-occurrence. Then after transformation, you can apply some document similarity metric, e.g. cosine similarity, to find tweets most relevant to your queries, e.g. "California measles".
Use knowledge or lexical databases like DBPedia and WordNet to calculate semantic relatedness between your query and the text of your tweets, or to identify tweets with related concepts.


Does adding of many FC layers during re-training increase the model size ? Are there any ways to optimize the size of model?

You can change the type of your weight, you will loose precision but reduce the size of the weights. You can also do some regularisation during training to learn sparse weight matrix

You can read these two articles :

Deep compression optimisation techniques for inference
Model quantisation for production level nn inference


computational time and precision trade off

A good reference can be found there http://scikit-learn.org/stable/modules/


There are multiple options, depending on your problem and the algorithms you want to use. The most promising (or closest to your original plan) is to use a generator to prepare batches of training data. This is only useful for models that allow for partial fits, like neural networks. Your generator can just stratify examples by for example generating a batch that includes exactly one of each target. One epoch would be when you served all the samples from the biggest class.

Downsampling is not a bad idea but it depends on the difficulty of your task, because you do end up throwing away information. You could look at some curves depending on the amount of samples for your model, if it looks relatively capped this wouldn't be a big issue.

A lot of models allow for weighting classes in your loss function. If we have 10,000 of class A and 1,000 of class B, we could weight class B 10x, which means mistakes that way count much harder and it will focus relatively more on samples from class B. You could try this but I could see this going wrong with extreme imbalances.

You can even combine these methods, downsample your biggest classes, upsample your smaller classes and use weights to balance them perfectly.

EDIT: Example of the batch options:

We have 4x A, 2x B and 1x C, so our set is:

A1 A2 A3 A4 B1 B2 C1

Regular upsampling would go to:

A1 A2 A3 A4 B1 B2 B1 B2 C1 C1 C1 C1

But this will not fit in our memory in a big data setting. What we do instead is only store our original data in memory (could even be on disk) and keep track where we are for each class (so they are seperated on target).

A: A1 A2 A3 A4
B: B1 B2
C: C1

Our first batch takes one of each class:

A1 B1 C1

Now our C class is empty, which means we reinitialize it, shuffle them (in this case it's only one example).

A: A2 A3 A4
B: B2
C: C1

Next batch:

A2 B2 C1

B and C are empty, reinitialize them and shuffle:

A: A3 A4
B: B2 B1
C: C1

Next batch is:

A3 B2 C1

And our last one of the epoch would be A4 B1 C1

As you can see, we have the same distribution as the full memory option, but we never keep more in memory than our original ones, and the model always gets balanced, stratified batches.


The natural choice would be the total squared error across the N predicted values, averaged across all examples. This is the simple extension of mean squared error from the univariate case. If you're using multivariate linear regression, this is in fact what you want to optimize in order to get the maximum likelihood estimate of the parameters as well.


What is the use of user data collection besides serving ads?

A couple of days ago developers from one product company asked me how they can understand why new users were leaving their website. My first question to them was what these users' profiles looked like and how they were different from those who stayed.

Advertising is only top of an iceberg. User profiles (either filled by users themselves or computed from users' behaviour) hold information about:

user categories, i.e. what kind of people tend to use your website/product
paying client portraits, i.e. who is more likely to use your paid services
UX component performance, e.g. how long it takes people to find the button they need
action performance comparison, e.g. what was more efficient - lower price for a weekend or propose gifts with each buy, etc.

So it's more about improving product and making better user experience rather than selling this data to advertisers.


Problems in building model predicting probablity of student's admission

Make sure you model the $Y$ data in the correct way.

Some reasons I can think on why do you have such strange results:

You are modelling binary Yes/No answers for all the students vs all the faculties, make sure you only take into account students which presented to the given faculty (some of them passed and some didn't), not all the students.
Sometimes more exams add noise to your answer, because the faculties look for specialized individuals who outperformed in what they are looking for, and that means you will find that the other results are "noise" for the answer.


Do numerical inaccuracies play any role in training neural networks?

Are there publications which mention numerical problems in neural network optimization?

Of course, there has been a lot of research on vanishing gradients, which is entirely a numerical problem. There is also a fair amount of research of training with low precision operations, but the result is surprising: reduced floating point precision doesn't seem to affect neural network training. This means that precision loss is pretty unlikely to be the cause of this phenomenon.

Still, the environment can affect the computation (as suggested in the comments):

Most obviously, random-number generator. Use a seed in your script and try to make a reproducible result at least on a single machine. After that you can compute the summary of activations and gradients (e.g. via tf.summary in tensorflow) and compare the tensors across the machines. Clearly, basic operations such as matrix multiplication or piece-wise exponent should give very close if not identical result, no matter what hardware is used. You should be able to see if the tensors diverge immediately (which means there is another source of randomness) or gradually.
python interpreter, cuda, cudnn driver and key libraries versions (numpy, tensorflow, etc). You can go as far the same linux kernel and libc version, but I think you should expect reproducibility even without it. cudnn version is important, because the convolution is likely to be natively optimized. tensorflow is also very important, because Google rewrites the core all the time.
Environment variables, e.g. PATH, LD_LIBRARY_PATH, etc., linux configuration parameters, e.g. limits.conf, permissions.

Extra precautions:

Explicitly specify the type of each variable, don't rely on defaults.
Double check that the training / test data is identical and is read in the same order.
Does your computation use any pre-trained models? Any networking involved? Check that as well.

I would suspect hardware differences the last: it's an extraordinary case if the high-level computation without explicit concurrency leads to different results (except floating point precision differencies) depending on the number of cores or cache size.


It is not necessary to scale your numerical values when using linear regression. However, people still do it since it can speed up training if the algorithm uses gradient descent and it might make your coefficient and intercept terms more interpretable.

You can read more about it here: https://stats.stackexchange.com/questions/185624/feature-scaling-normalization-in-multiple-regression-analysis-with-normal-equa

On another note, there are a lot of binary classifiers that will give you a probability instead of a binary output. For instance, in sklearn many binary classifiers have a predict_proba method that does just this. I can recommend random forest.


If you know that you want to become a data scientist, you can pretty much rule out pure mathematics.  Note, I'm not saying that pure mathematicians cannot become data scientists, but it's not the most natural transition.  Between the other two branches, stats is probably the most natural path.  Both will having you think about applying math to answer real world problems, but stats is very specifically geared towards focusing on larger scale data analysis.

EDIT:
@rocinante mentions the need for software skills, and suggests CS over econ as a minor.  I would say this really depends; if you are in a bigger data science team, you'll probably work alongside dedicated programers, who will be able to implement your analytics more efficiently than you would be expected to as an analyst.  If you know that you want to apply D.S. techniques to either finance or business, domain specific knowledge is helpful.  Additionally, if you know that it is your desire to end up in data science, you'll be able to look for ways to use programing along the way, and keep those skills sharp.


How can be proved that the softmax output forms a probability distribution and the sigmoid output does not?

Softmax maps $ f:ℝ^n\rightarrow (0,1)^n$ such that $\sum f(\vec x) =1$. Therefore, we can interpret the output of softmax as probabilities.
With sigmoidal activation, there are no such constraints for summation, so even though $ 0<S(\vec x)<1$, it is not guaranteed that $\sum S(\vec x)=1$. The sigmoidal function does not normalize the outputs, so in your example where class 0 has output $0.7$, class 1 could have any value in $(0,1)$, which might not be $0.3$.

Here's an example:
$\vec x=[-5,\pi,\frac{1}{3},0] $
$ f(\vec x)\approxeq [2.6379\times10^{-4},0.9059,0.05464]$
$ S(\vec x)\approxeq [6.693\times10^{-3},0.9586,0.5826,0.5] $
Because $0<f(\vec x)<1$ and $\sum f(\vec x)=1$, the softmax output vector can be interpreted as probabilities. On the other hand, $ \sum S(\vec x) > 1$, so you cannot interpret the sigmoidal output as a probability distribution, even though $ 0<S(\vec x)<1$
(I chose the above $\vec x$ arbitrarily to demonstrate that the inputs need not be negative, non-negative, rational, etc., hence $\vec x\in ℝ^n$)


How to improve accuracy of ethnicity prediction using name?

Feature engineering

The question you want to ask yourself is "When you look at a name how do you as a human deduce the community?" I would claim that this is possible by looking at the way the letters are grouped together. For example "aoui" and "wi" is very common in India and Arabic, whereas "Zh" would be very common in Chinese. So how can we extract these types of features?

n-grams

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

This would capture the distribution of letter combinations which are present for each community.


Sort a data frame column based on another sorted column value in R

You can simply arrange by two columns:

library(dplyr)

df %>%
   arrange(desc(var2),var1)

Edit:

To clarify why this works, in your example simply arranging the df by var2 in descending order will already put all 0 and NA values at the bottom (since NA is "worth" less than 0). However since these  rows all have the same value their order will be "random" because we do not specify a tie method. By supplying a second column for arrange, we sort all ties by a second criteria, var1.

This will actually also re-arrange rows with a duplicate var2 value as well but that should be a bonus.


How can we conclude that an optimization algorithm is better than another one

Whenever possible, you should use the average performance when comparing different methods, and preferably even mention the standard deviation across different runs (see this question for an example why it's important sometimes). It's perfectly fine to also provide the best performance, ideally you can even present a boxplot comparison of the different methods.

What is really unacceptable is to compare a best performance for one method against a mean performance for the other method (it should go without saying but I remember a paper where the authors were happily doing just that in order to make their method look better).


Define your own distance function.

I suggest you simply use

dist(x,y)=haversine(x[0],y[0])+haversine(x[1],y[1])


The suggestion in the comment is appropriate.
Still, if you want to try NN, you may try these suggestions -
 - None of the models seems best as per general guidlines
 - Keep ReLu as all the hidden layer, linear for last layer(Regression)
 - Standardization/Normalization must be done before training
 - Add Batch normalization layer
Can also try
 Since your output seems like Classes [0,1,2]. Try a Classification model for power and a Regression model for output angle
 For classification - last activation - Softmax, Loss - categorical_cross_entropy, Label - one-hot encoded


One Neural network with multiple outputs or multiple neural networks with a single output?

Given the information you provided, the most honest answer is: You have to test it by yourself, there is no general answer for it.
Still, it has been shown empirically in research that a neural network may benefit from having multiple outputs.
So let's say we have a neural network that has multiple outputs. Further, let us group them into specific tasks:
For example:

The output neurons of group 1 tell if the image containts a dog or a cat.
The output neurons of group 2 tell the size of the animal (width and height)
The output neurons of group 3 tell the color of the animal's hair (in some encoding)

and so on...
A common example would be Faster-RCNN vs Mask RCNN.
Assume that $g$ denotes the number of different groups of output neurons.
Now if you take a feed-forward neural network, you will have common layers that eventually branch to the different output groups. Let us call $\pi$ the function that maps an input image to this particular last common layer $L$ and let $\phi_{j}$ be the function that takes the information from layer $L$ to output the result of group $j$.
Thus, given an input image  $\mathbf{I}$, the neural network maps it to $\begin{pmatrix} \phi_{1}(\pi(\mathbf{I})) \\ \vdots \\ \phi_{g}(\pi(\mathbf{I})) \end{pmatrix}$.
The output of the last common layer $\pi(\mathbf{I})=:\mathbf{f}$ can be understood as an image descriptor $\mathbf{f}$ of the input image $\mathbf{I}$.
In particular, all predicted outputs rely on the information contained in $\mathbf{f}$.
$\textbf{Therefore}$: Merging multiple outputs into a single neural network can be understood as a regularization technique. The image descriptor $\mathbf{f}$ must contain not only the information if the images shows a dog or a cat, but also all the other information. It must therefore be a more comprehensive (or "more realistic") description of the input, which makes it more difficult for the network to overfit. The network cannot solve a specific task using a non-plausible explanation, as the corresponding image descriptor would lead to bad results  on the other tasks.
As a consequence adding additional (auxiliary) tasks to the neural network can improve the accuracy on the initial task, even if you are not interested in predicting these additional tasks.
So essentially, if there is a common description of your data, that can be used to solve your required tasks, the system may benefit by using one model with multiple outputs.
You may have a look into the literature, e.g. collaborative learning, multi-task learning, and auxiliary tasks.
I hope this answers your question.


Is there such thing as linear and non-linear data?

Naming linear data or non-linear data is a bit misleading and wrong I would say. Instead, there is linear relation and non-linear relation between variables would be better and correct naming. It can easily be explained by example from real life.

Non-linear relation: Any kind of relationship that is not linear can be put forward as an example (e.g. quadratic relation: y = intercept + x^2, square root relation: y = intercept + 6 * sqrt(x) etc.) Let's imagine a world where we are not aware of the free fall equation and we use ML to learn and predict it :). In that case (specific and simplified case), the algorithm would need to learn an equation in the following structures = v₀t + (1/2)gt², where t is our independent variable, s is our target or dependent variable and v₀, g would be our coefficients to be learned. In other words, you would want to predict free fall (distance or height) using time. In that fictional world, Newton’s Law of Universal Gravitation (F = G * (M1 * M2) / r^2), Area of a circle (A = π * r^2) would also be a quadratic problem for ML. Linear functions do not meet the requirements of such cases unless you use some kind of transformation, let's say take the square root of r^2 from Newton's law or the square root of t^2 from free fall then use them in a linear function. The most known example of non-linear real-life relation is a prediction of height using age. Height changes continuously, however, in different rates: until 13 gradually increases, between 13-18 significantly increases, between 18-25 slightly changes, and after 25 it does not change. Thus, it is impossible to fit it into a linear equation, because it is not dependent on a single coefficient or i.o.w. it does not fit into the formula height = intercept + b * age because coefficient (b) is not constant over time (age). Moreover, Graph, Tree, and other similar structures are also fitted into a non-linear relationship. In cases where the result is highly dependent on ifs, linear algorithms are useless because you cannot fit the relationship into y = a + b * x. Let's say, you want to predict who will win a chess game basing on the moves they make, you would use a tree-like (e.g.: alpha-beta pruning) algorithm to predict it.
Linear relation: It is simply the relations where those relations can be explained by the y = a + b * x formula or even y = a + b1* x1 + b2*x2 + .... The simplest example is predicting the cost that is spent in the bar (or any kind of entertainment venue). cost = intercept (let's imagine you pay money for entering to the bar)  + (the price of a drink) * (the number of drinks bought) + (the price of appetizer) * (the number of appetizers bought) would be our formula.


When to use t-distribution instead of normal distribution?

Student t-distribution by definition is a distribution of mean estimates from samples taken from the normally distributed population.

T-distribution has thicker tails and it gets thinner with increase of degrees of freedom, which in turn depends on sample distribution. So, at some point it closely resembles normal distribution and can be substituted by it.

As far as I remember (although, not 100% sure) this sample size of 30 threshold is valid for the t-test with α=0.05 (widely accepted Type I error level). Although, of you have much smaller α (e.g. 0.0001) you need to go much further to the tails of the distributions where the difference between t-distribution and normal distribution will be much more evident and thus you are better off using t rather than normal for larger sample size.

Another issue is standard deviation (or rather here we are talking about standard error). Normal distribution (and thus z-test) requires knowledge of population standard deviation. If you do not know it, you need to estimate it from the sample and it is obviously will be only an estimation of the population standard deviation. Student t-distribution handles estimated standard deviation better because using it with normal distribution (which should have only population standard deviation) will create extra errors (you will incorrectly estimate your type I (and Ii) errors).

So, the answer is, if you do not know population mean (which is almost always the case in the real world), use t-distribution. If you know population mean, be careful with 30 sample threshold. Depends on your application it can be significantly higher.


Although not directly answering your question, you may be interested in the field of automated question answering. To answer natural language text questions they must first be understood, which overlaps with your problem.

A good resource is the course by Jurafsky and Manning. Particularly the sections on semantics and question answering may help with what you are looking for. There are accompanying lecture videos available on youtube here.


Why such a big difference in number between training error and validation error?

I think this is because your training set is much larger than your validation set. What this means is that for training; losses are being accumulated for comparatively larger number of examples than validation. Hence your training error is larger than your validation error.

I don't think its a problem with your code. If you see closely the trend line has almost same slope for both training loss and validation loss; this suggests that you are okay as far as code is concerned.


Why and how BERT can learn different attentions for each head?

After re-read the code, I am wrong. Each head is different part of the tensor's last dim.

Each head is to split the last dim of the tensor.

For example, head number is 12 and last dim of the tensor should be 12*N.

Each value of the tensor's the last dim represent a feature for the tensor.

But it seems the training could let different heads be different representations. This is amazing.

Think about CNN. CNN also has a lot of kernels. And the info of kernels may have some redundancy. As the init values of kernels are different, the kernels also learn some different info.

Then watch the figure in the link again, BERT's attentions also have some redundancy.


Matplotlib WIndow Not responding when loop is initialized

The while True: loop will be causing the python interpreter to think there's something more to do. It will remain unresponsive until it 'finishes' that loop, which it will never do. Remove the Loop or replace pass with break


Get a portion of a long field in Pandas?

Fairly simply, for 'A' only:

max_chars = 300
for index, row in df.iterrows() :
    print(row['A'][:max_chars], row['B'])


Whether AUC can be calculated as average of sensitivity and specificity or not?

The article you mentioned is referring to the AUC of Receiver Operating Characteristic Curve (ROC-Curve), that is different from the AUC of the Precision-Recall Curve (PR-Curve). See this post to know about the differences between the PR-Curve and the ROC-Curve. If what you want is that same value for the PR-Curve you should look at the F1-Score, see this wikipedia article for an explanation of the relation between the AUC of the ROC-Curve, the Youden Index and the F1-Score. Also a good indicator of the AUC of the PR-Curve is the Average Precision (AP) as explained here.


Training a floor detection model: use full room images or only the cropped floor?

It isn't clear what you are trying to do (floor detection or just classification).

If your task is to classify different types of floor, you could use just the segmented parts. But if you want to detect the floor and classify it, you have to give your model some negative examples as well (it needs to learn how not-floor looks like as well).

However, in general it would be better practice to train the model on full images. Even if it is just a floor classification task, the context of the image can also help (kitchen increases the probability of tiles, bedroom increases the probability of a wooden floor, etc.).


One option is spectral clustering. Spectral clustering can find the "connectedness" in data.


Multidimensional scaling producing different results for different seeds

I think the answer to your problem is to understand that euclidian distance matrices are too much to uncover the information. Basically there is a lot of underlying constraints to verify (n*(n-1)/2 constraints for 2n variables).

One way to solve that kind of problem in human acceptable time is to consider 'physical' approaches. Basically you put tiny springs and other forces between your data points and solve a physical system problem. Thats why you usually see graphs 'giggle' when plotting them. As you can guess, these kind of approaches depend heavily on the initialisation of points in space. That's why there is a 'random' component in usual technics.


How to combine data having similar distribution?

Grouping values together is commonly called quantization or binning. Binning increases the signal-to-ratio in data.

Generally, time-series data is binned by time (also called rolling-up). For example, rolling-up all values over a range of days into a single month value.


How to obtain with a recurrent neural network the Xor function using keras?

I think following this link helps you, I have gone trough those tutorials previously.
Also take a look at this tutorial .


How to match a user with another user based on their taste?

You can perform clustering of your customers based on a distance function.
Definition might look like this:

First, calculate euclidean distances between the first item of the first customer's basket and all of the items in the second customer's basket.
Then find out, what is the closest item from second customer's basket (minimum euclidean distance).
Perform the same operation for each item in first customer's basket.
Calculate mean of the minimum distances.
Do the same for the second customer.
Take maximum of means from the first and the second customer.


(Python Basic) more elegant way of creating a dictionary

You can try:

my_dic = dict()

my_dic['Model'] = ['Apple', 'Banana', 'Pineapple', 'Melon', 'Orange', 'Grape']
y_list = [y_Apple, y_Banana, y_Pineapple, y_Melon, y_Orange, y_Grape]
keys = zip(['AAA', 'BBB', 'CCC', 'DDD', 'EEE'], ['method1', 'method2', 'method3', 'method4', 'method5'])
func  = lambda F, a, b: eval(F)(a,b)

for name, method in keys:
    my_dic[name] = [ func(method, y, y2) for y2 in y_list]


How can I group by elements in a column in pandas?

If I understand what you need, I think it is this:
b = a.pivot_table(values='TOTAL_BALANCE_EUR', index=['NSFR_GROUP', 'BALANCE_GROUP'], columns='GAP', aggfunc='sum')
b

It's easier for others to help you if you make the data available to others. Just make a tiny dataframe with 10 rows for instance. Also, you can make the code a bit easier to read by enclosing it in three backticks: ``` or using the code sample button when you write your post.


You can possibly use a combination of Named Entity Recognition and Syntactical Analysis -
while the word Edwin is certainly propping up, imagine a situation where the name is Edward Philip Martel. NER detects each word as a separate entities (hence 3 different entities) - thus, you will anyways have to string them together based on some logic. Further, in the case of multiple names being present, it can get harder to disambiguate (e.g. John & Ramsey dined at Winterfell).

This is where the analysis of the sentence syntax would also help (assuming that the end user enters a relatively coherent and proper sentence - if slang and short forms of text are used, even the Stanford NLP  can help upto a certain extent only).

One way of leveraging on syntax analysis / parsing and NER is in the following examples -

 1. User: Edwin is my name.
 2. User: I am Edwin.
 3. User: My name is Edwin.

In each of the cases (as is generically the case as well), the Entity Name (Proper Noun / Noun) is associated in close proximity to a Verb. Hence, if you first parse the sentence to determine verbs and then apply NER to surrounding (+/- 1 or 2) words, you may have a relatively decent way to resolve the problem.
This solution would depend primarily on the syntax rules you create to identify NERs as well as the window around the verbs.


Is there a quick way to check for multicollinearity between categorical variables in R?

In my work I usually use Normalized Mutual Information (NMI) to get an understanding of how "correlated" two categorical variables are.

Normalized Mutual Information is an information-theoretic measure that tells you how much information is shared by two variables. If NMI is close to 1, the two variables are very "correlated", while if NMI is close to 0 the two variables are "uncorrelated".

I wrote this function that computes the NMI between the first two variables in a data.table. It is quite fast thanks to data.table. Feel free to use it!

compute_mutualinfo <- function(df){
  require(data.table)
  require(entropy)
  var1 = names(df)[1]
  var2 = names(df)[2]
  tmp_tab <- df[,.N,by=c(var1,var2)]
  names(tmp_tab)[1:2] <- c('V1','V2')
  cross_tab <- tapply(tmp_tab$N,list(addNA(as.factor(tmp_tab$V2),ifany = T), addNA(as.factor(tmp_tab$V1),ifany = T)), sum)
  cross_tab[is.na(cross_tab)] <- 0
  mi.plugin(cross_tab)/sqrt(entropy(df[,.N,by=var1,with=T][,N])*entropy(df[,.N,by=var2,with=T][,N]))
}


Here's some starter code:

from keras.applications.vgg19 import VGG19
from keras.preprocessing import image
from keras.applications.vgg19 import preprocess_input
from keras.models import Model
import numpy as np

# define the CNN network
# Here we are using 19 layer CNN -VGG19 and initialising it
# with pretrained imagenet weights
base_model = VGG19(weights='imagenet')

# Extract features from an arbitrary intermediate layer
# like the block4 pooling layer in VGG19
model = Model(inputs=base_model.input, outputs=base_model.get_layer('block4_pool').output)

# load an image and preprocess it
img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

# get the features
block4_pool_features = model.predict(x)

You can then use these features for passing to a SVM classfier.
Here are some additional links for reference:

Understanding CNNS

Keras pretrained networks

Coding a convolutional neural network in keras


Evaluation is based on the task, not the type of model used for it. In the tutorial that you link the task would be simple document similarity. Afaik a more common variant is the information retrieval setting, where the goal is to rank documents according to their similarity against a query document.

The tutorial mentions that they use a test set for which a link to a paper is provided. This paper explains that human annotators rated the similarity level between pairs of documents. That would be the standard way to obtain annotated data, but of course it takes a lot of manpower. It doesn't seem that the ground truth information is provided with the test set used in the tutorial though, I assume that this is why the author only proposes a manual inspection of the results.

In general there are probably many benchmark datasets (with ground truth annotations) publicly available for similar tasks, but they are not always easy to find and I'm not very knowledgeable about these. I think a good place to start would be the SemEval series of competitions, every year they release various datasets related to this kind of task.


I would find the unit variance of the all the weeks and then divide by that. Scikit can do this for you using scale.


I would say this highly depends on how you have your code set up.  What type of model are you creating?  Are you not setting batch-size as a global parameter?  Can you not just use that?  Also, assuming y_true is a numpy matrix, can you just use:
y_true.shape[dimension_representing_batch_size] as a means of getting the size?


Is it possible to have a validation error less than train error for a while followed by the reverse behaviour?

The behavior you notice in the graphs will always be the case regardless of the machine learning algorithm. Machine learning will require you to fit some parameters such that the model can capture the mappings from your input space to the desired outputs.

At the start of the training these parameters will all be assigned randomly, thus we should assume random performance, thus 50% for a 0/1 classification. As the model parameters are tuned using the training data, the model will tend to decrease both the training and evaluation loss. However, after some training there will be a point where the evaluation loss starts increasing. This is the point where the parameters are no longer generalizing for the data but is instead overfitting and specifically identifying points in the training data.

This is NOT what we want. This is overfitting. We want to stop the training at the minimum loss of the evaluation set.

Another way to think of this is if I ask a child to look through some multiplication examples,

3*2 = 6
4*3 = 12
...
...
10*1= 10

If the kid learns some general rules as to how x*y=z then he will be able to answer any new multiplication question I ask him. However, if he just memorizes the outcomes of the few examples that I showed them then they will get any new instance wrong. This is overfitting.

As another example, imagine fitting a 2nd order relationship between 2 variables with a 5th order polynomial. This will match the points too specifically and will not generalize to new data.


How to predict an approximate weekly/monthly number, when the Unique Daily Visitors for that week/month are already known

To calculate the (estimated) value of the monthly/weekly unique visitors based on the per day count you may use the HyperLogLog algorihm.

This will give you the historical values and you can use any time series approach to predict future counts


dividing Mean by standard Deviation meaning

You have created what can be called a normalised mean. Normalised in the sense that it corresponds to a related random variable which now has $\sigma = 1$ (since you divide by standard deviation, look it up in any statistics textbook).
The normalised mean can be a better predictor resulting from its normalisation which makes it robust to the range of fluctuations.


Regarding the being new to decision trees and wanting to get off the ground, I wrote a tutorial on decision trees that will help.

Regarding methods to avoid overfitting:  The game for any model is to limit its complexity to what is reasonable given the data you have.  Complexity in decision trees is manifested as adding new decision boundaries, so any limit in complexity is a limit in the decision boundaries it can draw.  Two common ways to do this is to place constraints on when a new decision can be created (a minimum of data in a leaf, significant increase in information, etc) or more simply to limit the max depth of the tree.


If you need something interpretable but accuracy is less important, use the methods on the left side of the question (linear regression, logistic regression). If accuracy is more important, use the methods on the right side of the question (neural nets). SVM is a somewhere in the middle.


Choosing best methods for estimating the unknown parameters in a linear regression model

After the Data Munging, this is the most difficult task on a prediction model. However, in order to answer it, we need more details. What do you mean by the "best model"? Do you want accuracy and long training time? Do you need something really fast with lower accuracy? Something between the two of them? What are your features? Have you just taken them or have you created new features from them?

In any case, I suggest that you should spend some time and read this perfect tutorial from Microsoft about Machine Learning. A part of the tutorial to understand what I mean:

Regression

Ordinal regression: Data in rank ordered categories
Poisson regression: Predicting event counts
Fast forest quantile regression: Predict a distribution
Linear regression: Fast training, linear model
Bayesian linear regression: Linear model, small data sets
Neural Network regression: Accuracy, long training time
Decision forest regression: Accuracy, fast training
Boosted decision tree regression: Accuracy, fast training

When I have a similar question and I don't know which one to choose, I usually end up to 3-4 different algorithms based on the cheat sheet of Microsoft or the one from the scikit-learn, try them all and choose one or two of them with the best results.


Decision tree and random forest over fitting

Here are some questions that might help:

Do you have a balanced dataset? How does the distribution look like?
Is the distribution of the training and test set similar?
Did you try using cross validation?
What is your accuracy metric?
Are the features correlated among themselves?
I would also suggest using xgboost and compare its results.


.h5 file size is same before and after training?

This is not surpising.
h5 is the save file of the model's weights. The number of weights does not change before and after training (they are modified, though), therefore, your file should have the same size.

To know if your model after training is better, you have to measure it : you have to make predictions on a test set, and measure it, with usual metrics like accuracy or F score and so on. look at this to see usual metrics used in machine learning.


What fits in a Data Description Report/ Data Exploration Report?

Generally speaking each data set may have different structure and may relate to different business aspects.
Because of that I think its hard generalizing the description/exploration steps.

Yet, these are my two cents:

Data description:
  * Data sources - how was data created, generated, collected.
  * Data shape - number of rows, columns.
  * Data types(per each column) - numeric, strings, other..
  * Missing values - how much, where, why?
  * Time frames - if relevant.
  * Entities(geographical, markets, population segments, devices) - what identifies a row in the data(if applicable).

Data exploration:
* Counts per entity(grouping keys).
* Categorical variables breakdown.
* Values ranges distributions.
* Simple correlations.
* Story telling(visualizations that depict key aspects in the data<>business realm).
* Missing data deep dive.


Deriving Feature Map formula for Inhomogeneous Polynomial Kernel

Let $x, y \in \mathbb R^N, \mathbf \alpha \equiv (\alpha_0, \dots, \alpha_N) \equiv (\alpha_0, \mathbf \alpha'_0), \; \mathbf z^\alpha \equiv z_0^{\alpha_0} \dots z_N^{\alpha_N}$ and ${d \choose \alpha}$ be the multinomial coefficient. Then

$$\left(\left<\mathbf x, \mathbf y \right> + c\right)^d = \sum_{|\mathbf \alpha| = d} {d \choose \alpha} c^{\alpha_0} (\mathbf x \mathbf y)^{\mathbf \alpha'_0}$$

If you redistribute the coefficient ${d \choose \alpha} c^{\alpha_0}$ between $\Phi(\mathbf x)$ and $\Phi(\mathbf y)$, it follows that

$$\Phi(\mathbf z) = \left(\sqrt{ {d \choose \alpha} c^{\alpha_0}} \mathbf z^{\alpha'_0} \right)_{\forall |\mathbf \alpha| = d}$$

References

A Tutorial on Support Vector Machines for Pattern Recognition (cf. 6.1. The VC Dimension for Polynomial Kernels)
Data Mining and Analysis: Fundamental Concepts and Algorithms, Ch.5, pp 149-151.


What is a dimension?

To put it simply, if you have a tabular data set with m rows and n columns, then the dimensionality of your data is n:

  What is a manifold?

The simplest example is our planet Earth. For us it looks flat, but it really is a sphere. So it's sort of a 2d manifold embedded in the 3d space.

  What is the difference?

To answer this question, consider another example of a manifold:

This is so-called "swiss roll". The data points are in 3d, but they all lie on 2d manifold, so the dimensionality of the manifold is 2, while the dimensionality of the input space is 3.

There are many techniques to "unwrap" these manifolds. One of them is called Locally Linear Embedding, and this is how it would do that:

Here's a scikit-learn snippet for doing that:

from sklearn.manifold import LocallyLinearEmbedding

lle = LocallyLinearEmbedding(n_neighbors=k, n_components=2)
X_lle = lle.fit_transform(data)
plt.scatter(X_lle[:, 0], X_lle[:, 1], c=color)
plt.show()


On coursera what exactly does Andrew Ng say in videos Lectures 60 & 61 of machine learning?

No, he actually says the opposite:

  One final note: I should say that in the machine learning as of this practice today, there are many people that will do that early thing that I talked about, and said that, you know...

Then he says (the "early thing" he talked about):

  selecting your model as a test set and then using the same test set to report the error
  ...
  unfortunately many people do that

In this lesson he explains about separating the data set:

training set to train the model;
cross validation set to find the right parameters;
test set to find the final generalization error (of the function with the best parameter values found during using the cross validation set).

So Andrew Ng is complaining that many people us the same data set to find the right parameters, and then report the error of that data set as final generalization error.


How regularization parameter in SVM affects hyperplane parameters

This is really an interesting question.
In fact, many people like Svm, many use it, many understand the idea of svm " maximize the distance between the two margins "subjects to the condition of classifying all the samples correctly!! Quite easy but can get a little bit more complicated when we are dealing with none separable data and overfitting problems.

In your question, you mentioned that this parameter is helpful to perform regularization, but why we need regularization? The answer is to avoid overfitting.

But what is overfitting? Overfitting is a problem of overfitting the data. Overfitting can be more serious if you have noisy data.

In Svm, you can think about the distance between the margins as an indicator of how much you are fitting the data. If the distance is too small, then you are trying to fit " seperate between" the samples that are very close to each other. Usually, if two samples are very close to each other and they have different labels, one of them is incorrect, or it is a noisy sample.

Now, from your question, you have two parts, the first part control the distance between the margins, the second part controls the error in miss classification. Changinging " increasing" lambda will give more attention to the error rather than the distance.

The more you reduce lambda the maximum the distance between the margins you can get.

Why you do not see this clearly in your plot?
Because you plot the hyper plane between the two margins, if you can plot the margins, you will see how changing lambda is affecting the distance between the margins.


How can I use variable length inputs to train a regression model?

A quick approach can be using a library like tsfresh, which extracts features from your time series, e.g. max value, number of peaks, median value, etc.

Normally, a thoughtful solution to this kind of problem involves domain knowledge, that is, insights from an expert that can tell you which aspects of your time series are important, e.g. frequent variations, very high peaks, plateaus, local patterns, etc. This knowledge is then used to derive ad hoc feature extractors.


There is, in fact, a very reasonable list of publicly-available datasets, supported by different enterprises/sources.

Some of them are below:

Public Datasets on Amazon WebServices;
Frequent Itemset Mining Implementation Repository;
UCI Machine Learning Repository;
KDnuggets -- a big list of lots of public repositories.

Now, two considerations on your question. First one, regarding policies of database sharing. From personal experience, there are some databases that can't be made publicly available, either for involving privacy restraints (as for some social network information) or for concerning government information (like health system databases).

Another point concerns the usage/application of the dataset. Although some bases can be reprocessed to suit the needs of the application, it would be great to have some nice organization of the datasets by purpose. The taxonomy should involve social graph analysis, itemset mining, classification, and lots of other research areas there may be.


use the same gradient to maximize one part of the model and minimize another part of the same model

The trick you are looking for is called the Gradient Reversal Layer. It is a layer that does nothing (i.e., identity) in the forward pass, but it reverts the sign of the gradient, so everything behind the layer optimizes the opposite of the loss function.
There are several PyTorch implementations:

https://github.com/janfreyberg/pytorch-revgrad
https://github.com/jvanvugt/pytorch-domain-adaptation/blob/master/utils.py

Initially, it was introduced for unsupervised domain dataptaion. Now it has quite a lot of applications, such as removing sensitive information from CV representation or removing language identity from multilingual contextual embeddings.


Cosine similarity vs The Levenshtein distance

As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between sequences of characters.

Nevertheless they both can be used in non-traditional settings and are indeed comparable:

the vectors compared with cosine can for instance contain frequencies of characters or characters n-grams, hence making it a string similarity measure
one can replace the sequence of characters with a sequence of strings or a sequence of n-grams, thus making Levenshtein a more general distance measure.

The main conceptual difference between Cosine and Levenshtein is that the former assumes a "bag-of-words" vector representation, i.e. compares unordered sets, whereas the latter takes into account the order of the elements in the sequences.

In the context of comparing sequences of words many combinations are possible. In case that's what you're looking for you might be interested in this paper: https://www.aclweb.org/anthology/C08-1075/ (full disclosure: I'm one of the authors).


Math Behind GOSS (Gradient-Based One Side Sampling)?

Wang et al., (2019) have provided a nice and clear explanation. Please, check out their paper to find the answer you are looking for:
Part II. BAYESIAN OPTIMIZED LIGHTGBM
Section: A. The Principle of the LightGBM
Wang, R., Liu, Y., Ye, X., Tang, Q., Gou, J., Huang, M., & Wen, Y. (2019). Power System Transient Stability Assessment Based on Bayesian Optimized LightGBM. 2019 IEEE 3Rd Conference On Energy Internet And Energy System Integration (EI2). doi: 10.1109/ei247390.2019.9062027


Why can't I choose my hyper-parameter in the training set?

I wouldn't say you can not tune the hyper-parameters in the trainig dataset, but the purpose of doing so is different than in the validation set.

In general, what it is intended with a ML algorithm is how to optimally classify or perform regression given some training data. Once the model is trained, it will be used with new data to obtain predictions, thus:

We need some training data. With this dataset, the model will look for the optimal weights and biases that minimizes the selected loss function.
We need some upper bound to our performance metric: for example, we can decide that our model has to perform like a human, which has showed an average metric of 95% for some specific task. The upper bound can be the best score in a certan benchmark, etc.
We start training the model on the training data and we evaluate its metric, lets take the accuracy. If the accuracy is too low, we can tune the hyper-paramters until the accuracy increases in the training data (no evaluation dataset is used here). The difference between our model's accuracy and the upper bound is taken as the bias of the model (this bias has not to be confused with the biases of the neurons). So, the accuracy on the training data gives us the bias of our model.
Once we decided our bias is reasonable (optimally 0) arises the question of how the model will perform when it is fed with data that has not be used for training (the real application). The difference between the accuracy on this unseen data and the accuracy on the training data is the variance of the model. This unseen data is the validation set and give us an idea of how the model generalizes to new data. If the variance is high, then the model poorly generalizes to new data. Then, we perform hyper-paramater tunning and evaluating the accuracy on the validation data until the variance is low enough, trying to not worsening the bias (variance-bias trade-off).

Example: desired accuracy = 95%. Training accuracy = 93%. Validation accuracy = 82% -> Bias = 2%, Variance = 11%

Summarizing:

Tunning in the training data = decreasing bias
Tunning in the validation data = decreasing variance

What's more unclear is the role of the test dataset. I usually use it when I decided that the model will be no longer changed and I need the final metric that describes the model. The test set can also be used to have an idea about how the model performs with data which have not be intended to work with. In general, the test set gives the power of the model for the inference task.


ValueError: y should be a 1d array, got an array of shape (285, 30) instead

The problem is that you are using the everydaydata to build the traintarget dataset, but you should use the labels in everytarget. That is why is complaining abut the shape, because the labels should be one dimensional.
Try replacing this:
traintarget = np.concatenate((everydata[:140],everydata[212:357]),axis=0)

with this:
traintarget = np.concatenate((everytarget[:140],everytarget[212:357]),axis=0)


Training a regression algorithm with a variable number of features

I think in the specific case that you have given as an example, you can use approach 1 which you have mentioned above i.e.

either replace the missing features with a zero value

or otherwise you can also

implement your algorithm in a sparse manner so that it doesn't need to always have all the features for an instance.

Using a sparse format will not affect the runtime (or any other aspect) of your algorithm. Whereas, if you use the zeroing features approach, then your algorithm will have to do the unnecessary multiply by zero operations and this will increase your runtime ( How much? that depends on how sparse is your actual data).

Hope that helps.


Relationship between Sigmoid and Gaussing Distribution

Yes sure, the core idea is that the sigmoid function approximate very nicely the cumulative density function of a Gaussian distribution. Therefore, if the input of a neuron is a continuous Gaussian distribution, the probability of the output being inferior to a certain value will follow (almost) the shape of the sigmoid function.

A detailed explanation can be found p16 of the following article:

https://www.researchgate.net/publication/220233479_Catastrophic_Forgetting_and_the_Pseudorehearsal_Solution_in_Hopfield-type_Networks


This looks like a case of the model outputting the probability of being in category 1. It then is up to you to decide on the cutoff.

You give an example of an output of $(0.43, 0.56, 0.1, 0.8)$. If your cutoff is $0.5$, you’d get classifications of $(0, 1, 0, 1)$. If you set your cutoff at $0.2$, which you’re allowed to do, you get classifications of $(1,1,1,0)$. There should be a way to do this in whatever software you’re using. If all else fails, you can loop through your vector of probability values and produce a new vector with some conditional statements about if the value exceeds the threshold and if the value does not. I’ll give some pseudo-code.

for value...
  if value>threshold
    new value =1
  else
    new value =0

This leads to ROC curves and area under the curve, where the model has its accuracy assessed at all cutoffs. Perhaps even better is to use a strictly proper scoring rule. You can read more about this on the Cross Validated Stack, stats.stackexchange.com. Look out for posts on this topic by Frank Harrell. Shamelessly, I will mention a post of mine where I somewhat challenge this idea: https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email.

Loosely speaking, a strictly proper scoring rule penalizes a model for lacking confidence in its prediction. Outputting a probability of being in class 1 of $0.56$ is not as good of a prediction as $0.99$. Think of the first as, “Sure, I guess it’s category 1,” while the second is, “Oh heck yes it’s category 1!”


The idea is really simple, just look at some online resources like https://en.m.wikipedia.org/wiki/Sparse_matrix

The implementation is also really simple. For pandas this page might help you https://pandas.pydata.org/pandas-docs/stable/sparse.html


Can we use feature selection and dimensionality reduction together?

Every time you compress the feature space you are losing some information. The original feature engineering stage you outlined sounds like a meaningful compression & might make sense in the context of your problem. The second compression on the other hand might only serve to lose some information.
I would only perform the second compression if the classifier you are training is struggling resource-wise to train your model after the initial compression. In this case you might want to sacrifice some information to train the model quicker. But this step should only be investigated after you try training your model with only the original compression & are not satisfied with the speed of training.
Good luck!


In which epoch should i stop the training to avoid overfitting

As long as your validation accuracy increases, you should keep training. I would stop when the test accuracy starts decreasing (this is known as early stopping). The general advise is always to keep the model that performs the best in your validation set.
Although it is right that your model overfits a little since epoch 280, it is not necessarily a bad thing provided that your validation accuracy is high. In general, most machine learning models will have higher training accuracy compared to validation accuracy, but this doesn't have to be bad.
In a general case, you expect your accuracy to behave in the following way.

In your case, you're before the early stopping epoch, so even if your training set accuracy is higher than your test set accuracy, it is not necessarily an issue.


Can someone explain what exactly breaks down for non-episodic tasks for Monte Carlo methods in Reinforcement Learning?

It is this first part of the sentence that you quote:

"To ensure that well-defined returns are available..."

In more detail, the return distribution in an episodic MDP can be defined as
$$G_t = \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}$$
where $R_x$ is reward distribution at time step $x$, $\gamma$ is discount factor, and $T$ is the last time step of the episode.
From this you can see that you will need to collect all of $r_{t+1}$ to $r_T$ for an episode in order to calculate a single concrete return value $g_t$. You can only collect $r_T$ at the end of an episode.
In non-episodic tasks, there is no terminal time step $T$, and the return definition becomes:
$$G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$
This cannot be resolved directly by collecting data for infinite time steps. Or in other words you can never sample an actual return value as it is defined. To get around this you could do one or more of the following:

You could use an artifical time horizon $h$, and define $T = t + h$. This will result in a biased sample. However, you can make the bias arbitrarily small relative to total return by making $h$ large. If you are using a discount factor, that may allow for smaller value of $h$.

You could use temporal difference (TD) learning to bootstrap return estimates without needing a full sample of return. This will result in a biased sample, but the bias due to bootstrapping should reduce as more samples are collected.

You could use TD with eligibilty traces and a high trace decay parameter $\lambda = 1$, which is a way of getting very similar learning behaviour to Monte Carlo, but still allowing learning to occur on each step in a continuing problem.

These options technically stop the approach being true Monte Carlo. The last two are described in the book under TD methods.


How to encode a column containing both string and numbers

Encoding is a way to transform categories to numerical variables, there are a lot of techniques.

The best technique depends on what is the information you want to encode and what is the model that you are going to use. Some models benefit more of one techniques than others.

You should ask yourself the following questions to try to find the best solution:
 - Why do you have numerical and categorical values in a column?
 - Does it make sense to have them?
 - Which model am I using?
 - What is the best way to feed them to this model?

From the bit that I get I would reccomend that either you do target encoding with everything or that you split that feature into two. And then you do target encoding in the categorical and even in the numerical. Just check which works better.


How can I tokenize a text file with BERT or something similar?

With Huginface's Transformers, it should be doing with not much boilerplate.

A minimum example using PyTorch:

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

Depending on how data you want to process, you might want to speed it up a little bit. In that case, you should do it batches which would require doing padding and passing the padding mask to the model.


Support Vector Regression trained with data sets

Why not concatenating the 10 datasets you have and simply creating one big dataset?
Alternatively you could train 10 seperate SVMs and let them vote (for examples: output = mean of the 10 SVMs).


Unsupervised k-NN

Unlike k-means, the unsupervised k-nn does not associate a label to instances. All it can do is tell you what instances in your training data is k-nearest to the point you are polling for.

For example:

import numpy as np
from sklearn.neighbors import NearestNeighbors
samples = [[0, 0, 2], [1, 0, 0], [0, 0, 1]]

neigh = NearestNeighbors(2, 0.4)
neigh.fit(samples)

neigh.kneighbors([[0, 0, 1.3]], 2, return_distance=False)

  array([[2, 0]]...)

You can see that this returned the index of the k-nearest points, and not the label.

k-means

This algorithm is completely different. The k here denotes the number of assumed classes that exist in your dataset. For example if you have unlabeled pictures of red and green apples, you know that $k=2$. The algorithm will then move the centroids (the average of the cluster distributions) to a stable solution.

Here is an example:

Let's first make some artificial Gaussian distributed data.

import numpy as np
import matplotlib.pyplot as plt

params = [[[ 0,1],  [ 0,1]],
          [[ 5,1],  [ 5,1]],
          [[-2,5],  [ 2,5]],
          [[ 2,1],  [ 2,1]],
          [[-5,1],  [-5,1]]]

n = 300
dims = len(params[0])

data = []
y = []
for ix, i in enumerate(params):
    inst = np.random.randn(n, dims)
    for dim in range(dims):
        inst[:,dim] = params[ix][dim][0]+params[ix][dim][1]*inst[:,dim]
        label = ix + np.zeros(n)

    if len(data) == 0: data = inst
    else: data = np.append( data, inst, axis= 0)
    if len(y) == 0: y = label
    else: y = np.append(y, label)

num_clusters = len(params)

print(y.shape)
print(data.shape)

  (1500,) (1500, 2)

plt.scatter(data[:,0], data[:,1])
plt.show()

The k-means algorithm from scratch

class Kmeans(object):

    def __init__(self, k=1):
        self.k = k

    def train(self, data, verbose=1):

        shape = data.shape

        ranges = np.zeros((shape[1], 2))
        centroids = np.zeros((shape[1], 2))

        for dim in range(shape[1]):
            ranges[dim, 0] = np.min(data[:,dim])
            ranges[dim, 1] = np.max(data[:,dim])

        if verbose == 1:
            print('Ranges: ')
            print(ranges)

        centroids = np.zeros((self.k, shape[1]))
        for i in range(self.k):
            for dim in range(shape[1]):
                centroids[i, dim] = np.random.uniform(ranges[dim, 0], ranges[dim, 1], 1)

        if verbose == 1:
            print('Centroids: ')
            print(centroids)

            plt.scatter(data[:,0], data[:,1])
            plt.scatter(centroids[:,0], centroids[:,1], c = 'r')
            plt.show()

        count = 0
        while count < 100:
            count += 1
            if verbose == 1:
                print('-----------------------------------------------')
                print('Iteration: ', count)

            distances = np.zeros((shape[0],self.k))
            for ix, i in enumerate(data):
                for ic, c in enumerate(centroids):
                    distances[ix, ic] = np.sqrt(np.sum((i-c)**2))

            labels = np.argmin(distances, axis = 1)

            new_centroids = np.zeros((self.k, shape[1]))
            for centroid in range(self.k):
                temp = data[labels == centroid]
                if len(temp) == 0:
                    return 0
                for dim in range(shape[1]):
                    new_centroids[centroid, dim] = np.mean(temp[:,dim])

            if verbose == 1:
                plt.scatter(data[:,0], data[:,1], c = labels)
                plt.scatter(new_centroids[:,0], new_centroids[:,1], c = 'r')
                plt.show()

            if np.linalg.norm(new_centroids - centroids) < np.finfo(float).eps:
                print("DONE!")
                break

            centroids = new_centroids
        self.centroids = centroids
        self.labels = labels
        if verbose == 1:
            print(labels)
            print(centroids)
        return 1

    def getAverageDistance(self, data):

        dists = np.zeros((len(self.centroids),))
        for ix, centroid in enumerate(self.centroids):
            temp = data[self.labels == ix]
            dist = 0
            for i in temp:
                dist += np.linalg.norm(i - centroid)
            dists[ix] = dist/len(temp)
        return dists

    def getLabels(self):
        return self.labels

And the results

kmeans = Kmeans(5)
kmeans.train(data)

  Ranges:  [[-15.42553872  14.88894099]  [-13.33192554  16.15415347]]
  Centroids:  [[-11.39200726 -10.71208054]  [  3.73634888  -8.9230959 ]
  [  6.17589734 -10.66376228]  [  0.78973744  -0.44245535]  [
  9.29524253   9.59127574]]

Initialize, the red points are the random centroids

Iteration 1

and after a few iterations


You could use Topic Modeling as described in this paper:
http://faculty.chicagobooth.edu/workshops/orgs-markets/pdf/KaplanSwordWin2014.pdf

They performed Topic Modeling on abstracts of patents (limited to 150 words).  They identified papers as "novel" if they were the first to introduce a topic, and measured degree of novelty by how many papers in the following year used the same topic. (Read the paper for details).

I suggest that you follow their lead and only process paper abstracts.  Processing the body of each paper might reveal some novelty that the abstract does not, but you also run the risk of having much more noise in your topic model (i.e. extraneous topics, extraneous words).

While you say that all 500 papers are on the same "topic", it's probably safer to say that they are all on the same "theme" or in the same "sub-category" of Bio-medicine. Topic modeling permits decomposition of the "theme" into "topics".

The good news is that there are plenty of good packages/libraries for Topic Modeling.  You still have to do preprocessing, but you don't have to code the algorithms yourself.  See this page for many resources:
http://www.cs.princeton.edu/~blei/topicmodeling.html


Unseen classes - 'Other' class or classifying based on the existing classes

You could definitely add more samples and tag it as other with your classifier. However, realistically this might not be an option. For this exact purpose, you could use confidence values with the output of your classifier to predict the "other" class. You could bring in confidence thresholding to support this, i.e if your classifier is not confident > 0.8 (as an example) with its prediction, then you tag it "other" class.

What is the best value to threshold with, you can identify by running a number of experiments with different threshold values and plot a graph for correctness i.e classification error rate graph.


Is there any clear tutorial for how to use AutoEncoders with text as input

Concerning Encoding, this link helped me a lot. If you try the code in part 'One Hot Encode with scikit-learn', you will get your encoded vectors. You just have to feed it a list of all your tokens. So I have extracted the fields of the register into a list of markups.

As an output of the one hot encoding part, you will get a n dimensional array, you  feed it to the script in part 'Let's build the simplest possible autoencoder' of the AutoEncoder link

In the Input vector you need to put how many classes you have (instead of the value 784) and in our case, you specify the number of columns of the nd array which represent the unique tokens.

Then to compare the predicted output, you have to use decoding in order to compare visually the text that was regenerated.


Classification models with multi-class allowed for each record

The task is called multilabel modelling. Many libraries in sklearn support this -http://scikit-learn.org/stable/modules/multiclass.html.

These algorithms generally use one vs rest strategy, building separate binary classifiers for each of the label (zero for all others) and assigning all labels which clear a probability threshold.


Collection Of Variable Length Sequences and Descriptions: A Search Problem

While an RNN using one-hot encoded moves is possible, I would suggest that your model needs to understand chess (or similar complex games) at a deeper level to be able to associate comments to positions.

I would encode the position itself (eg a layered representation like in Alpha-Zero paper), and pass those through a conv-RNN to model the temporal relationship between positions corresponding to annotations. (Maybe you can find a pretrained model for evaluation as a baseline).

I read a fair amount of chess commentary, and one thing that is common is that there are cases of single move as well as comments about several moves at a time (think opening lines, or forced continuations), which are fairly discrete (my 'move 10 sacrifice' and my 'move 15 blunder' are not related by a hidden state). This leads me to think that instead of an RNN model, you are really trying to first Partition then Embed moves in a comment space - which then leads to a model that is first an RNN which may learn likely partitions of moves to be commented on, then another model, perhaps conditional autoencoder, is trained jointly on (position | comment) subsequences (you should still have plenty of features to work with at this point, so no worry about a weak signal).

The 'search' functionality here points not at a pure deep learning model, since you want to find specific instances of positions, rather than a typical position with that comment (ie sampling a latent space). This leads me to think that you create an explicit data-structure to map points in your final embedding space to their positions. So if you wanted to search 'Nice Sacrifice', model would find closest points in 'comment space' and return those games using something like cosine distance, or this last step is clustering of the (position | comment) space.

Either way, interesting problem, try the simpler thing first.


Temporal difference learning with a neural network

Should I scale it before apply learning?

Definitely, you need to either:

Scale your network output so that it lies in the range (-2, 2), or,
Scale your training data so that it lies in the range (-1, 1)

The latter is marginally preferable because it will only need to be computed once, rather than on every pass through your network.

  Can we say anything about it from theoretical point of view?

This is a very standard design, and is simply a consequence of the fact that most commonly chosen activation functions have bounds of [-1, 1], [0, 1], [0, inf) or (-inf, inf). In particular for finite bounded activation functions (i.e. those with outputs bounded by [-1, 1] or [0, 1]), you will need to make sure that the output space of the network is capable of representing the desired target space. For example, for binary classification problems, you will often see people choosing labels of either 0/1 or -1/1 depending on whether the final output is activated by a tanh or sigmoid function.


Algorithm for generating rules for classifying documents

I think a simplest way is like this: convert the document to a vector like "bag of words" or "n-grams", then apply algorithm which can be used to derive rules Decision tree or LEM2.


Feature extraction from sequence of images with Siamese Neural Network

Which is better single frame similarly check or sequence check?

Sequence Check. Because in the Single frame Similarity check (Consider SNN), You need to input pairwise images. So that you need so many pairwise images for your SNN for your Batch Generation.
For the Batch Generation idea is to make usuable batches for training the network. We need to create parallel inputs for the A and B images where the output is the distance. Here we make the naive assumption that if images are in the same group the similarity is 1 otherwise it is 0. If we randomly selected all of the images we would likely end up with most images in different groups.
So no problem if you chose randomly or not.
For the Sequence Check, I picked this paper : Deepfake Video Detection Using Recurrent Neural Networks

Here propose a two-stage analysis composed of a CNN to extract features at the frame level followed by a temporally-aware RNN network to capture temporal inconsistencies between frames. And in this paper, they actually detect deepfake from the videos. They used 600 videos to evaluate the proposed method and get 94% more accuracy.


You made three mistakes:

You omitted the offset terms before the nonlinear transformations (variables b_1 and b_out). This increases the representative power of the neural network.
You omitted the softmax transformation at the top layer. This makes the output a probability distributions, so you can calculate the cross-entropy, which is the usual cost function for classification.
You used the binary form of the cross-entropy when you should have used the multi-class form.

When I run this I get accuracies over 90%:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets('/tmp/MNIST_data', one_hot=True)

x = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])

W_h1 = tf.Variable(tf.random_normal([784, 512]))
b_1 = tf.Variable(tf.random_normal([512]))
h1 = tf.nn.sigmoid(tf.matmul(x, W_h1) + b_1)

W_out = tf.Variable(tf.random_normal([512, 10]))
b_out = tf.Variable(tf.random_normal([10]))
y_ = tf.nn.softmax(tf.matmul(h1, W_out) + b_out)

# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)
cross_entropy = tf.reduce_sum(- y * tf.log(y_), 1)
loss = tf.reduce_mean(cross_entropy)
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# train
with tf.Session() as s:
    s.run(tf.initialize_all_variables())

    for i in range(10000):
        batch_x, batch_y = mnist.train.next_batch(100)
        s.run(train_step, feed_dict={x: batch_x, y: batch_y})

        if i % 1000 == 0:
            train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})
            print('step {0}, training accuracy {1}'.format(i, train_accuracy))


Can we automatically chose k value in k-means algorithm?

This is a known problem of automatic clustering, how to choose / adapt the number of clusters so it represents the "real" clusters.
Hierarchical clustering is more helpful in this regard. For algorithms like K-means this is not so easy and research has tried various approaches to determine the optimum number of clusters (eg employing Information-Theoretic criteria like Akaike information criterion - AIC).
An overview in given in wikipedia article and references therein.

Determining the number of clusters in a data set, a quantity often
labelled k as in the k-means algorithm, is a frequent problem in data
clustering, and is a distinct issue from the process of actually
solving the clustering problem.
[..]The correct choice of k is often ambiguous, with interpretations
depending on the shape and scale of the distribution of points in a
data set and the desired clustering resolution of the user. In
addition, increasing k without penalty will always reduce the amount
of error in the resulting clustering, to the extreme case of zero
error if each data point is considered its own cluster (i.e., when k
equals the number of data points, n). Intuitively then, the optimal
choice of k will strike a balance between maximum compression of the
data using a single cluster, and maximum accuracy by assigning each
data point to its own cluster. If an appropriate value of k is not
apparent from prior knowledge of the properties of the data set, it
must be chosen somehow. There are several categories of methods for
making this decision.


It is impossible to answer this question because you have not defined what precisely you want your chatbot to do (and not to do!). Once you have defined that, you have your metric.

So the answer to your second question is: no, there is no standard validation metric.


Instead of one-hot encoding a categorical variable, could I profile the data and use the percentile value from it's cumulative density distribution?

(Edited after @D.W. suggestion).

To the best of my knowledge, there is nothing wrong with what you have in mind; thus it is certainly valid.. As you said, you have to try out all possible ways you can think and see which one works better. The most important point adopting a particular encoding is to be aware of you have certain amount of information getting lost and it varies one case to another depending on the problem at hand. For example in the case of binary coding based on high/low frequency sublevels, there will large of information (details) lost which could help the algorithm to do classification. I liked your idea of percentile coding based on cumulative density distribution. Maybe you want to look at Quantile-based discretization which is available in pandas.qcut.

The rest is my earlier answer as it was (below). I intended to suggest trying other techniques as well on top of what you have had in mind; but apparently the message was not clear. Please note that I do not seek to get my answer marked as final answer, as I aware that still it does not fully answer your question; it is simply to brainstorm and exchange ideas in length. ;-)

Perhaps you have already digged out enough about ways to convert categorical variables into continuous data. In case you did not and missed checking this answer, check it out. As discussed, they are many ways to convert cat-to-num, and your problem is one of the hardest yet the most common across many domains. You have a high cardinality in your categorical variable, and as far as I understood imbalance distributions of those sublevels and you are not sure whether messing up the order of those sublevels matters or not. You may need to try Ordinal Encoding (if order really matters) or weight of evidence (WoE) transformation (see this blog post for instance) that I have heard but not tried or even going beyond mixing them in a meaningful way to represent your categorical data properly.

What I have learned, despite all efforts in the field, that this problem is still an open challenge in data science and machine learning. Thus there is no best solution, or well-established method as far as I checked. Please do let me know if you come across one.


Difference between various linear regression implementations

In the first, you need a linear model and the cost function is RMSE. with this model and cost function, w is generated.

on the other hand, for the second equation, you use a non-linear NN equation as estimation and the cost function may RMSE, log or ... .( NN without activation-function is linear)
NN without activation function is a linear model with bias. if you use RMSE it is like the first example.

any linear regression with the same cost function will generate, the same value for coefficient after convergence (In some condition the equation has many solutions, In this case, the cost function may converge to a new value)

Note: many solutions for an equation meaning, for example, if one point is the solution of the equations, then any hyperplane that crosses that point is the solution.


the code seems ok plotting the distribution. let me know the exact error.


Does a matrix factorization recommendation engine use user/item related features?

If I wanted to incorporate a user feature like age, would it be possible with matrix factorization?

Yes, it's possible. It's commonly called a hybrid recommendation system.


What Kind of Network should I be considering?

You want to use a Recurrent Neural Network.  More specifically, look into the LSTM variant and understand it's use cases over a "regular" RNN, and go from there.  RNN's are good for capturing sequential data patterns and this is essentially what you want to do.


You can define a custom distance metric using the metric parameter from sklearns KNeighborsClassifier where you can specify your callable function that gets called when calculating the distances from the algorithm. There you can weigh var_3 inversely proportional.


It's a loss function applied to a regression with l2 penalty on the parameters.
The first square brackets can be interpreted in the following way:

$ - \frac{1}{n} $ has the minus because it wants to minimize.
$\sum_{i=1}^{n}$ means for each data point.
$\sum_{j=0}^{k-1} $ means for each class.
$y_i == j$ means that the fraction after this term is calculated only for true class. This check makes sense because y true is an OHE vector like [0, 0, 1], therefore you want to evaluate only the predicted probability associated with the true class. Take into account softmax function: if you increase the probability of a single output in output in the softmax you are implicitly reducing the probabilities of the other outputs.
the log division is the softmax function.
$ + \frac{\lambda}{2} \dots $ is a l2 regularization term on the model parameters.


Training algorithm for finding relationships within a data set

Given your example, if you want to group phrases based on terms, a simple approach would be to use grep to find rows that contain specific letter sequences (e.g. "art", "ball"). For example, say your table is a csv file (myfile.csv), where each line is a word or phrase (e.g. "Football", "Jazz music"). We can use a UNIX terminal to get all lines that contain "ball":

grep -n ball myfile.csv | cut -d: -f1 > myresults.txt

This would yield a file (myresults.txt) that containts the lines numbers of phrases that contain "ball". You can also apply grep to a data frame version of your table using R or Python, so you can loop over different letter sequences and save the found relationships to a separate column of the data frame or a separate data frame or vector.


Why do I need pre-trained weights in transfer learning?

When you are calling a pre-trained model (resnet50 in your case) for performing transfer learning, only the model's architecture is actually called.
This architecture is practically useless to perform any sort of computational operation on the data and predicting any output unless trained. For that, you will need the weights (variables understood by the model on some previous dataset).

Weights are more like mathematical coefficients which take the input and performs the computation on the input to generate a required output. They are more like the coefficient in a linear algebra problem. This is the actual reason why it is required to call the weights for the model.

Even if you are not calling any weights in you model, you can make your model trainable and train the same to generate your own weights that fit your data properly rather than fine-tuning your model. Read more about the same here


Thank you @thanatoz for answering that in the comments.  In fact, this did solve my problem.  I also realized from other people's VQVAE code, there is also another possible solution in which the retrain_graph parameter is not needed.  My example code could be written ...
x = self.linear1(x)
min, max = int(x.min()), int(x.max())
bins = torch.linspace(min, max+1, 16)
x_buckets = torch.bucketize(x.detach(), bins) # forced to detach here
x = x + (x - x_buckets).detach() # Reintroduce gradients here. <------------
x = self.linear2(x)

The answer is in the fifth line, by subtracting and then adding back again.  Gradients can then be reintroduced.


Neural Network Architecture for Identifying Image Copies

U need to read about triplet loss function. Triplet loss function gets result embeddings from a network, that process 3 images by a network (2 similar and 1 non-similar) for one step:

After that loss computing as:

For more details read the paper from triplet loss authors.

Also may help PSNR, but this is not Deep Learning.


How to find the residuals of a classification tree in xgboost

There are 2 methods to "continue a model training":

Training from another xgboost model

If you have a pre-trained xgboost model, you can continue training using the xgb_model parameter in xgb.train.

  xgb_model  a previously built model to continue the training from.
  Could be either an object of class xgb.Booster, or its raw data, or
  the name of a file with a previously saved model.

Training residual

Using xgboost to training a residual or prediction of a model, you can following the demo provided in xgboost:
https://github.com/dmlc/xgboost/blob/master/R-package/demo/boost_from_prediction.R

Simply speaking, you can use the "base_margin" element when preparing data.


What about imshow? Check these different interpolation methods which will allow you to define the "discreteness".
Example with quadric interpolation:
import matplotlib.pyplot as plt
import numpy as np

harvest = np.array([[0.8, 2.4, 2.5, 3.9, 0.0, 4.0, 0.0],
                    [2.4, 0.0, 4.0, 1.0, 2.7, 0.0, 0.0],
                    [1.1, 2.4, 0.8, 4.3, 1.9, 4.4, 0.0],
                    [0.6, 0.0, 0.3, 0.0, 3.1, 0.0, 0.0],
                    [0.7, 1.7, 0.6, 2.6, 2.2, 6.2, 0.0],
                    [1.3, 1.2, 0.0, 0.0, 0.0, 3.2, 5.1],
                    [0.1, 2.0, 0.0, 1.4, 0.0, 1.9, 6.3]])

fig, ax = plt.subplots()
im = ax.imshow(harvest,interpolation="quadric")

Output image:


What's the point with neural networks if you can only predict linear test data?

Neural networks can generalize and successfully predict outside their training data. This ability is hindered by overfitting, where the network memorizes the training data and does not perform well on unseen data.
As with any other problem faced with neural networks, it is key for the network to have an inductive bias that is appropriate for the problem data. From this SO answer:

Every machine learning algorithm with any ability to generalize beyond the training data that it sees has some type of inductive bias, which are the assumptions made by the model to learn the target function and to generalize beyond training data.

For instance, convolutional networks perform well in image data due to their spatial locality inductive bias.
The main problem in your examples is that the function you are modeling is periodic while the activation functions you are using lack a periodic inductive bias. This problem is studied in the article Neural Networks Fail to Learn Periodic Functions and How to Fix It presented at NeurIPS 2020. From the abstract:

[...] we prove and demonstrate experimentally that the standard
activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a “periodic” inductive bias. As a fix of this problem, we propose a new activation, namely, $x + sin^2 (x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations.


How to forecast product/item sales next one week using Xgboost regressor

Yes, you cannot pass in a datetime object as a feature to XGboost Regressor, because it isn't time-series forecasting tool such as ARIMA.

Instead, you create various features from the timeseries object such as year, month, day of month, day of week, number of days until public holiday etc. All of these are of type int. Drop the timeseries column.

When you're passing in the test data, you convert the datetime object into the above features.


I would suggest parametrizing with a logarithm of volatility so you don't have to care about positivity, run the estimation and then invert back to original scale. Alternatively, you can consider constrained optimization routine. Without knowing more about the problem (at least the language you're using), that's about it.


What is the mean of inconsistent in machine learning and why 1NN is well known to be inconsistent

Concistency of any algorithm in machine learning or statistics rather means that assuming you train on an infinite amount of data that your algorithm will converge to the true value of your estimate. Meaning that if you feed infinite datasets into the algorithm your error will converge to 0

Now regarding 1NN:
1NN has an important property: When feeding infinite training examples the error rate of the 1NN converges to a limit of twice the Bayes error. Now if you take the definition above and match the 1NN's property you'll see the reason why it's inconsistent.

See his paper for a proof of the property: http://cseweb.ucsd.edu/~elkan/151/nearestn.pdf


How to build a classifier with a rejection class

What is the best way to do that? Should I just add a new class label that includes pictures of anything but food?

Adding a new class which can be called none is a thing that is usual for such tasks.

  Is there a best way to chose which pictures I should use?

Yes, you should pick the images of the none type from the real distribution that your classifier is going to face while test time. It means all the classes should have a same distribution while training and testing. It should be satisfied even for the none class. For instance, suppose that your none type class can be a human hand or a table or something like that. You should not put an elephant image for none type because you are not going to have an elephant in your kitchen.


I'll go through your questions one by one:

  Is the dataset imbalanced one?

The distinction between a balanced and an imbalanced dataset is a matter of degree. If you think that the difference in number of observations for each class is too big, I suggest you to train your model using Mini-Batch Gradient Descent, and build any mini batch in a way that observations have the same frequency. This would attribute greater weights to the less frequent observations.

  How do I distribute this dataset properly into training set,
  validation and test set?

Make sure that all the classes are evenly distributed among the sets.


Activation function vs Squashing function

Activation functions like sigmoid function, hyperbolic tangent function, etc. are also called squashing function because they squash the input into a small range like in sigmoid function output is in range of [-1,1]. But you cannot call ReLU as a squashing function because for a positive input value it returns the output as same.


How can I feed BERT to neural machine translation?

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model.
I got a memory error when I want to use more data like 100k for my task. and I came up that my tokenizer is a kind of problem here because it takes a lot of memory to make my tokenizer for such a huge volume of data so pre-trained models like BERT are the solution to this problem to feed more data like 200k or more to your model without worrying about memory error too much.

Also, in my task, I was worry about words that do not exist in my training phase but exist in a test phase so bert solved this problem for me too, because it was trained on a large corpus.
let's dive in and find out how I used BERT to fix my problem here.

Here I am going to make English to English translation system.

Loading pre-trained BERT for English ( if your source and the target language differs from each other, you have to load them separately you can look at tfhub.dev for them )

max_seq_length =  50 # i need to test the bert so I will keep this small for now
input_word_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,name="input_word_ids")
input_mask = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,name="input_mask")
segment_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,name="segment_ids")
#this is the path to pre-trained bert model
bert_layer = hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1",trainable=True)

then I call my tokenizer

FullTokenizer = bert.bert_tokenization.FullTokenizer
vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()
do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
tokenizer = FullTokenizer(vocab_file, do_lower_case)

here is an examples to use the tokenizer:

tokenizer.convert_tokens_to_ids(['is'])
tokenizer.convert_ids_to_tokens([2003])
tokenizer.convert_ids_to_tokens([30521])

the output is: ([2003], ['is'], ['##～'])

then I used my train data to get my sequence

s = "This is a nice sentence."
stokens = tokenizer.tokenize(s)
stokens = ["[CLS]"] + stokens + ["[SEP]"]
input_ids = get_ids(stokens, tokenizer, max_seq_length)

the output for me was:

input Ids: [101, 2023, 2003, 1037, 3835, 6251, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

this method is also needed for getting token ids and you should add it to your codes

def get_ids(tokens, tokenizer, max_seq_length):
    """Token ids from Tokenizer vocab"""
    token_ids = tokenizer.convert_tokens_to_ids(tokens)
    input_ids = token_ids + [0] * (max_seq_length-len(token_ids))
    return input_ids

for your model vocab size you can use these values,

len(tokenizer.vocab)

which in my case is: 30522

and after these steps, you can get your sequence for input and outputs and then feed them to your model.

I hope it helps. and please share your opinion if I am wrong in this case.

thanks


No, there is nothing really better than recalibrating the model and see if it improve performance. Most techniques will be approximation of what you are doing with your main model. If you happen to find a simpler model that perfectly explain which variable plays a role in your main model and how, you should probably just use the simpler model.

Basic a priori techniques (looking at correlation with output, with other variables) will fail in the context of complex machine learning models.

One practical way to go nowadays is to look at feature importance. Those techniques depends on the model you use and will provide an importance value for each feature of each instance. If a feature has a low importance for all instances you should try removing it. (you can do that by batches of k least important features).


I let it run for more epochs, as suggested, and i noticed that after 23 epochs, rnn did get 0% error.

I then tried changing activation function of GRUCell from the default tanh to relu and it greatly speed up training. It took 1-4 epochs to get 0% error.


Can distortion be derived from inertia rather than recalculating it from scratch in case of kmeans?

The reason is that
distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                        'euclidean'), axis=1)) / X.shape[0])

does not square the distance as stated in the definition

Distortion is the average of the euclidean squared distance from the centroid of the respective clusters

If you replace
    mapping1[k] = sum(np.min(cdist(X, kmeanModel.cluster_centers_,
                                   'euclidean'), axis=1)) / X.shape[0]

with
    mapping_squared[k] = sum(np.square(np.min(cdist(X, kmeanModel.cluster_centers_,
                                   'euclidean'), axis=1))) / X.shape[0]

you will get the desired result:
IN: for key, val in mapping_squared.items():
    print(f'{key} : {val}')

OUT:
1 : 12.802768166089965
2 : 4.025210084033612
3 : 0.954621848739496
4 : 0.7467787114845938
5 : 0.5647058823529413
6 : 0.42156862745098034
7 : 0.29901960784313725
8 : 0.24019607843137258
9 : 0.16666666666666669

That is, the values in mapping_squared can be multiplied by the number of samples to match inertia:
IN: np.array(distortions_square) * X.shape[0]

OUT:
array([217.64705882,  68.42857143,  16.22857143,  12.6952381 ,
         9.66666667,   7.51666667,   5.66666667,   4.08333333,
         3.        ])

On a side note: Distortion and SSE are usually used interchangeably. See, for example, the paper Scaling Clustering Algorithms to Large Databases:

Distortion is the sum of the L2 distances squared between
the data items and the mean of their assigned cluster

This question touches the subject too.


How to architect ConvNet to ignore top half of image

You could be right that ignoring top part of image would benefit the CNN. However, there is very little point in trying to architect this - if your premise that the CNN will ignore irrelevant details in the top half is correct, then that will occur anyway and there is no standard NN architecture that will help that other than disconnecting the top half of network, which is going to be logically exactly the same as programmatically slicing the image, with the disadvantage of storing and calculating with twice as many parameters.

You should either programatically cut the image in half or do nothing to the image and rely on the CNN's inherent ability to give low weights to irrelevant details. If you do the latter, you may be able to get around the learning of incorrect details in the top half by augmenting your data - e.g. add some noise to images*, especially in the irrelevant top half. Perhaps horizontally flipping a few images (and reverse relevant targets for the control class) might be another useful augmentation. Some augmentations could also be useful if you just take the lower half of the image.

* Noise should be something close to variations that could be seen when in use. E.g. slurring pixels left or right might be reasonable. Inserting "static" probably is not.


Count and summarise ID's each day while creating a new column that shows the accumulated ID's

This is a base solution for your problem:

aggregate(paste(ID , Date) ~ ID + Date, data = df, FUN = length)

there are many more solutions like any of the ones below, using dplyr:

library(dplyr)
df %>%  group_by(ID, Date) %>% summarise(PurchaseCount = n())
df %>% group_by(ID, Date) %>% tally(name="PurchaseCount")
df %>% group_by(ID, Date) %>% count(name="PurchaseCount")
df %>% group_by(ID, Date) %>% add_tally(name="PurchaseCount")
df %>% group_by(ID, Date) %>% add_count(name="PurchaseCount")

or by using data.table package:

library(data.table)
setDT(df)[, PurchaseCount:=.N, by = list(ID, Date)]

or using sqldf package:

library(sqldf)
sqldf("SELECT ID, Date, COUNT(*) as PurchaseCount
       FROM df
       GROUP BY Date, ID")

or plyr:

plyr::count(df, c('ID','Date'))

I personally prefer data.table as it is directly writes to the dataframe and often is time efficient. aggregate is also favorable when you wanna avoid loading new libraries. dplyr generally make your code more legible as it uses pipingpersonal opinion.


Plotting relationship between 2 data points where one data point is a boolean