Ways to reconstruct shuffled pixels of a video file?

A general solution to this does not exist, even if we add some assumptions about the distribution of e.g. colours and shapes in the images or temporal coupling such as consecutive frames being similar.

Problem

Let $F_1,\dots,F_i$ be the $n$ original frames, each with $m$ pixels. Let $P$ be the permutation that is applied to the pixels of each frame before we get them. You can think of $P$ as the enemy's code-book.

Now, as input we are receiving $P(F_1),\dots,P(F_n)$. The goal is to find the inverse permutation $Q$ to restore the images. Thus $QP=I$ is the identity map and for example $Q(P(F_1))=F_1$. Note that we do not know any of the correct frames $F_i$.

Let $Q_1,...,Q_{m!}$ be the $m!$ possible permutation functions of the $m$ pixels.

The goal is to select the unique $j\in\{1,\dots,m!\}$ so that $Q_jP=I$.

No General Solution

Under our statistical model this means selecting the $Q_j$ which maximises the likelihood that $Q_j(P(F_i))$ is drawn from the same distribution as the reference statistics for images and the temporal statistics between consecutive frames $Q_j(P(F_{i})$ and $Q_j(P(F_{i+1})$ which is our prior knowledge.

There is a canonical counter-example where the enemy gives you a scrambled movie with two frames where all the pixels are the same colour, so $n=2$, $F_1=F_2$ and $Q_j(F_1)=Q_j(F_2)=F1=F2$ for every $j$. Thus, for all $j$, the in-frame and inter-frame statistics are equiprobable for each $j$ and give us no information to select the maximum likelihood permutation $Q_j$ (except in the degenerate case where $m!=1$).

Thus, we cannot guarantee uniqueness and the problem is unsolvable without further assumptions.

Further Assumptions

It is interesting to see if we can solve the problem by adding more constraints.

If we restrict the enemy to only sending us "real" movies and assuming there are enough different pixels and frames to so that a unique $Q_j$ with maximum likelihood exists, we would still have to calculate statistics for $O(m! \times n)$ permuted frames to find the maximum.

This is brute force code-breaking.

In order to benefit from neural networks, and back-propagation in particular, we would need a differentiable loss function with respect to the input (which is an encoding of $j$ or our permutation $Q_j$). The question then, would be to see if such a function can be found.

Otherwise the problem is more similar to cryptanalysis in the special case where we know that the enemy's code book is a permutation of the clear-text (or clear-image).


data science / machine learning resources?

Start with the Coursera's Machine Learning course. It does a really good job in introducing the student to the domain of Machine Learning and helps you lay a solid foundation in the concepts.

In case, you feel that the math is a bit dumbed down in that course, you can take this course, taught by the same professor and is more math-intensive than the former.

Now, you would have a clear intuition about the basic concepts of Machine Learning. Take this course, which can be said as a follow-up or a supplement to the course of Andrew Ng.

This resource from IAPR has in-depth notes on a lot of ML concepts like cross-validation, regularization, etc.

You can also have a look at these amazing list of resources compiled into a blog on Quora.

Now, for diving into advanced concepts of neural networks and deep learning, you can make use of this free book.

Finally, the free e-book: Elements of Statistical Learning is a wonderful book for beginners in ML or Statistical Learning.


How to animate growth of a social network?

Fancy animations are cool
I was very impressed when I saw this animation of the discourse git repository. They used Gourse which is specifically for git. But it may give ideas about how to represent the dynamics of growth.
You can create animations with matplotlib
This stackoverflow answer seems to point at a python/networkx/matplotlib solution.
But D3.js provides interaction
If you're looking for a web-based solution then d3.js is excellent. See this, this and this for example.
See also this stackoverflow question, the accepted answer points to D3.js again.
Conclusion
I would be drawn towards the python/networkx options for network analysis (possibly to add attributes to your raw data file for example). Then, for visualisation and dissemination D3.js is perfect. You might be surprised how easy it can be to write d3.js once you get into it. I believe it even works within an ipython notebook!


Best approach to create a text classification model with two inputs?

All current pre-trained Transformers (BERT, RoBERTa, ...) can take two sentences as an input, simply by concatenating the sentences and separating them by a special token. You can then finetune the model on the paraphrase dataset.
The Microsoft Research Parallel Corpus is part of the GLUE benchmark for sentence representation evaluation. It has a public leaderboard, so you can have a look and get an idea, how the best models look like.


What are the latest and most succesful classification algorithms at the moment?

Deep Learning is your friend. Neural Nets with the growth of publicly available data, and also privately, have beaten every other architecture with the sufficient hardware for it to run properly on.Here's a depiction of why NN's are so popular


So there are some things you could consider.

First of all, normally datasets like this (or for example mnist) will be one hot encoded. So you have a vector with the length of all classes, each position stands for one class and is 1 if it is the class and 0 otherwise.
As you are predicting values between zero and one, the activation of your output should do the same. So the first thing i would change is the activation function of the last layer to sigmoid. It is linear in your case.

Other things to consider are data preprocessing like normalizing pixel values between 0 and 1. Also no data preprocessing at all can lead to unstable training it is unlikely to cause no learning at all.


Beautifulsoup iterating through scraped data

I could iterate and get the text from all the h5 tags with this code:
    main_block = soup.find(class_='Node_full startups content_class_folder')
    block = [h5.get_text() for h5 in main_block.find_all('h5', class_='Company_line-title')]

I'm wondering if there is another way to extract the text from the h5 tags.


Is my bayes classification right or meaningful?

Do you think this result is right?

Depends what you mean by "right"... the results seem reasonable, I don't see any obvious sign of mistake in the process.

  Can you explain what would you understand by looking at this result please?

I observe that you don't have any data for classes 1 and 5, so technically it's a 3-classes problem.

First with 3 classes the random baseline accuracy would be 0.33, 0.45 is better so your model does better than this (that's the bare minimum).
However according to the confusion matrix class 3 has 92 instances out of a total of 172, which means that a basic majority class learner always predicting class 3 would get 52% accuracy (if my calculation is correct). So 45% is not very good.


Which supervised learning algorithms are available for matching?

You can try to frame this problem as a recommender systems situation. Where you have your users (prospective students) and items (alumni) and want to recommend to the users one item.

It's not a perfect fit as you want just one item for each user and you don't have previous match data for each user. However you could investigate this idea a bit further. I'm applying these techniques to the recruitment problem, I'm matching users with job offers and I'm having some success.

Try to read a bit about recommender systems, to start I recommend chapter 9 of mining massive data sets, it's really introductory, but gives a good overview of the most common techniques.


Public dataset for news articles with their associated categories for multilabel data classification

I found one on Kaggle by searching "news categories". I believe this dataset should work for you. Its a JSON file which contains a link to the article, an associated category (e.g. "crime"), headline, authors, date, and a short description.
You may also want to check out the Open Data Stackexchange:
https://opendata.stackexchange.com/
Enjoy!
https://www.kaggle.com/rmisra/news-category-dataset


This is not currently implemented. So while you can add any number as a concept value, you're probably best served by adding 1s and 0s depending on whether a not misspecification occurred.

The reason it is not implemented is probably that ADWIN essentially performs a statistical test (or a heuristic approximation) whether the mean of two (large enough) windows is significantly different. And comparison of multivariate mean vectors is very tricky, in particular if one cannot make any assumptions about normality etc.

Otherwise, your workaround of using 4 drift detectors seems like the best solution!


Processing csv file with more than 700K rows of data

I think the main time-consumer will be the fact that you are iterating over 700,000 rows, one-by-one.

You could perhaps do a few of your checks/comparisons (used in the line with bit-wise stuff: &, | etc) to begin with, for all rows. So for that row, it looks like this now:

data.loc[
    (data['Category'] != category) &
    (pd.Series(list1).isin(data["List1"]).any()) &
    (pd.Series(list2).isin(data["List2"]).any()) |
    (data["List2"][0] == 'NULL') |
    (list2[0] == 'NULL'))]

Before the loop, you could pre-compute each of those rows, adding the results for each comparison as a new columns:

data["check1"] = data['Category'] != category
data["check2"] = ...
data["check3"] = ...
data["check4"] = ...
data["check5"] = data.List2 == "NULL"

Now use those columns' values in your line with all comparisons.

To gain a better understanding of what it taking up all the time, you could profile the code, e.g. using the CProfile module. You could just run over the first 100 rows of your data for that purpose - the profiling tool will give summary statistics.

If you really want to get stuck into it, check out tha Pandas performance improvement methods, which get into using Numba and Cython. There can be huge gains here, but you need to maybe invest a little more effort. Numba would be the easiest to begin with; unless you are experienced with C-languages.


You can accomplish what you want by doing:

  df.groupby('Users')['content'].unique().apply(pd.Series)

First, what you want is to groupby your values by users, to get all content values. Since you want unique ones, you can apply unique() built-in pandas function.

Then, since you will have a columns with a list of values, you can apply pd.Series to each row, to expand it into columns, by default, they are named from 0 to n, where n is the maximum length of a list, in this case, two.


Backpropagation with a different sized training set?

Your cost function implies that you want the network to predict the magnitude of the output vector based on the magnitude of the input vector. Is that what you want to do? If not, e.g. if you want to accurately predict the output vector components, you need to believe that these are predictable based only on the magnitude of the inputs. If you don’t think this is possible, then you need to find new data. If you do, then a normal error function should be fine (e.g. MSE) and you can use normal backdrop as implemented in any NN library. If you’re getting poor performance, it’s probably because the norm of the input vector does not carry enough information to make a good prediction.


Your specific case

After [seq1 0-1s] (1st sec of long sequence seq1) at index 0 of batch b, there is [seq1 1-2s] (2nd sec of the same sequence seq1) at index 0 of batch b+1, this is  exactly what is required when we set stateful=True.

Note that the samples inside each batch must be the same length, if this is done correctly, difference in sequence-length between (not inside) batches causes no problem. That is, when all samples from batch b are processed, then next batch b+1 will be processed, and so on and so forth.

A general example

As a general example, for stateful=True and batch_size=2, a data set like

seq1: s11, s12, s13, s14
seq2: s21, s22, s23
seq3: s31, s32, s33, s34, s35
seq4: s41, s42, s43, s44, s45, s46

where sij denotes j-th time step, must be structured like

    batch 1         batch 2         batch 3         batch 4

0   s21, s22        s23, <pad>      s31, s32, s33   s34, s35, <pad>   ...
1   s11, s12        s13, s14        s41, s42, s43   s44, s45, s46

or like (with overlap)

    batch 1         batch 2         batch 3

0   s21, s22        s22, s23        s23, <pad>    ...
1   s11, s12        s12, s13        s13, s14

where, for example, long sequence s21, s22, s23 (3 time steps) is broken down to two sub-sequences s21, s22 and s23, <pad>. Also, as you see, it is possible to have batches with different sequence lengths (by using a custom batch generator).

Note that <pad> (padded values) should be masked to prevent RNN from considering them as actual values (more info in this post). We can also avoid using padded values by opting for batch_size=1 which might be too restrictive (more info in this post).

Here are two examples of a sequence with 5 time steps:

            s11   s12   s13   s14   s15

example 1   23,   25,   27,   24,    28     # 5 temperature readings for t to t+4

example 2   I,    like, LSTM, very,  much   # 5 128-dim word embeddings

Some resources

You may find this article on stateful vs stateless LSTM helpful. Some quotes from the article:

  The stateless LSTM with the same configuration may perform better on
  this problem than the stateful version.

and

  When a large batch size is used, a stateful LSTM can be simulated with
  a stateless LSTM.

This blog on Stateful LSTM in Keras by Philippe Remy
Some opinions on Keras github, like this.


"meaningful" is a vague word anyway, but yes you got the idea: in the context of a particular task meaningful data is the information which contributes to solving the task. Non-meaningful is the opposite, so information which doesn't help for the task. Sometimes non-meaningful data makes it harder to the ML algorithm to capture the relevant information, since it has to correctly identify what is meaningful and what is not (as opposed to the case where it's provided only with relevant information).

Example: if you want to predict how fast a car can go, engine type and manufacturer are meaningful data. Colour is not.


GridSearchCV returns the minimal bandwidth in the grid

Then alter your grid to have a lower lower bound.


Transformer: where is the output of the last FF sub-layer of the encoder used?

We can see this in the original Transformer diagram:

The output of the last encoder FF layer is added to the original input of the same layer, then layer normalization is applied and that is the output of the whole encoder, which is used as keys and values for the encoder-decoder attention blocks in the decoder.


Can neural networks have multi-dimensional output nodes?

The output layer does not have to be 1D (excl. batch size) but even if it is, it does not necessary mean you cannot transform it to a n dimensional space. Consider an autoencoder used to reconstruct an image:

In the simplest case we could flatten a image (e.g. 24 x 24 pixels) and learn a network to predict the 24 x 24 pixels (output a 1D image). These pixels can then be transformed back to an 2D image (https://www.tensorflow.org/tutorials/generative/autoencoder). So in other words, even if your network outputs a 1D shape, nothing prevent you from reconstructing it to a higher space.

We can achieve similar results as stated in the point above, by using an encoding network (convolutional + pooling layers) followed by a decoding layer (transposed convolutional + up sampling layers). In this case you can effectively generate a 2D image directly (https://www.tensorflow.org/tutorials/generative/cvae).

You can also look at image segmentation networks for inspiration of how higher dimensions outputs can be generated.


Look up a number inside a list within a pandas cell, and return corresponding string value from a second DF

I created my own data.

train.csv

id,attrib
1,1 2 3
2,3 4 5
3,2 3 5
4,1 1 1

labels.csv

attrib_id,attrib_name
1,a
2,b
3,c
4,d
5,e

Read the csv files and create df1 and df2

After that use

def get_name(x):
    result = []
    for t in x.split(' '):
        result.append(df2[df2['attrib_id']==int(t)]['attrib_name'].values[0])
    return result

df1['attrib'] = df1['attrib'].apply(lambda x: get_name(x))

This will result in df1 looking like

I guess you also doing the same thing when you referred @Danny. The only thing important here is to convert the string into integer


How can I extract the reason of the legal compensation from a court report?

Can you clarify how consistent the structure of these sentences will be?

If the court reports contain very consistent phrasing, you might be able to get away with using a rule-based approach (or perhaps even regular expressions).
But, if the sentences would require human-level language abilities to parse, you would probably need to invest a lot of energy into training a model to accomplish this, which would also require a large(ish) set of labelled training data.

That said, you may be able to apply parts of speech (POS) tagging to isolate the right class of word in each sentence. I'm sure other options also exist, but I know NLTK accomplishes this task.


Cosine similarity or logistic regression for spam filtering

Use logistic regression which allows the weights to be learnt. By using cosine similarity you are forcing the weights to be the same for all features (assuming that you normalize the features first). This is putting unnecessary restrictions on the model.


Loading and querying a Spark machine learning model outside of Spark

If PMML supports the model you're trying to express, it's probably the best option. Spark has some partial support for exporting as PMML. Look into JPMML + OpenScoring to score the PMML in an embedded instance.

There are loads of tools in this space that implement something like this, but they're typically entwined with their own formats or model management system.


Purpose of converting continuous data to categorical data

Your intuition is generally correct - in many cases, premature discretization of continuous variables is undesirable. Doing so throws away potentially meaningful data, and the result can be highly dependent on exactly how you bucket the continuous variables, which is usually done rather arbitrarily. Bucketing people by age decade, for example, implies that there is more similarity between a 50-year-old and a 59-year-old than there is between a 59-year-old and a 60-year-old. There can be some advantages in statistical power to doing this, but if your binning doesn't reflect natural cutpoints in the data, you may just be throwing away valuable information.

You can find a very similar question here:

https://stats.stackexchange.com/questions/68834/what-is-the-benefit-of-breaking-up-a-continuous-predictor-variable?noredirect=1&lq=1


Explanation on some steps of AdaBoost.R2

Q1. $\operatorname{sup}$ is the supremum, or least upper bound.  You may think of it as the maximum, but relaxed in case the set of values doesn't actually have a maximum.  (And the pipes are just absolute values.)  https://en.wikipedia.org/wiki/Infimum_and_supremum
Q2. No, they mean $w_i \cdot \beta^{1-L_i}$.  See also https://stats.stackexchange.com/a/299266/232706.


How many model parameters did Microsoft's 2015 ImageNet-winner network have?

The number of parameters is 60,344,232, according to:

https://resources.wolframcloud.com/NeuralNetRepository/resources/ResNet-152-Trained-on-ImageNet-Competition-Data

Although, from https://arxiv.org/pdf/1512.03385.pdf (page 6), it looks like an ensemble was used for ILSVRC 2015:

"We combine six models of different depth to form an ensemble (only with two 152-layer ones at the time of submitting). This leads to 3.57%
top-5 error on the test set (Table 5). This entry won the 1st place in ILSVRC 2015."


Since you are specifically asking about deep learning techniques, nothing strikes me out of the bat other than Autoencoders.

You can try using autoencoders for clustering, basically, you need to stick with clustering. Since you don't have the labels.

To answer your question whether they rely on embeddings that are out there, it depends on your data, if you have a domain-specific data or you have the data in a weird language then you should go for creating your own embeddings.

I found this article really helpful, though it has been done on images, you can try it on text, using Conv1D. Also yes, you can try out word embeddings like word2vec or fasttext. There is this good article where they use gensim for doing attaching the embeddings, in case your data is just plain ordinary English.

there's also this article from keras blog where the author uses pre-trained GloVe embeddings.

Hope this helps.


How to calculate LFW accuracy of face recognition model?

FaceNet should embed pictures in a high dimensional space in such a way that similar pictures are close (wrt to a given distance metric) to each other and different images are farther apart.

That means, to calculate the accuracy of your model you need to find your threshold distance d which delivers the highest accuracy. Given such a distance, you can calculate the accuracy like you would do for a binary classifier. That is, the accuracy is the "Number of correctly classified pairs/Number of pairs"

For example, consider the three images A1,A2 of person A and B1 of person B. And say your DNN embedds the images such that the pair-wise L2 distance (i.e., the euclidean distance) is 1,3, and 7. Then you can see that if use a distance threshold of 2 you get 100% accuracy, and otherwise only 66%.

+----------+--------------+-------------+------+------+
|   Pair   | Ground Truth | L2-distance | d=2  | d=4  |
+----------+--------------+-------------+------+------+
| A1 A2    |            1 |           1 |    1 |    1 |
| A1 B1    |            0 |           3 |    0 |    1 |
| A2 B1    |            0 |           7 |    0 |    0 |
+----------+--------------+-------------+------+------+
| Accuracy |              |             | 1.00 | 0.66 |
+----------+--------------+-------------+------+------+

If you are using Tensorflow, you can follow David Sandberg's instructions to caculate evaluation metrics (including Accuracy) of your model here.

IMPORTANT: To compare the performance of your approach to other approaches, you should follow one of the protocols of the LFW dataset. The protocols are described in detail here (see Section 4). Basically, they have a special view on the dataset, which asks you to report the mean accuracy + error rates on 10 folds of that dataset.


Is there an NLP corpus that contains common medical terms?

Welcome to the biomedical domain, one of the few domains in NLP where there are too many resources to choose from :)

Data resources:

Medline is a database corpus of 30 millions abstracts.
Each Medline abstract is annotated with Mesh descriptors, Mesh being a structured hierarchy of medical concepts.
PubMed Central (PMC) is a database of around 6 millions full biomedical articles (not only abstracts).
UMLS is a database of millions of medical terms grouped by concept, themselves grouped by semantic type (e.g. disease, gene, etc.)
PubTator is a resource which provides all the Medline and PMC documents fully annotated with a combination of Mesh and other ontologies.

Software tools:

MetaMap is the venerable annotator system which annotates any medical text with UMLS labels.
cTakes is another annotator system which is more specialized with clinical texts.
SciSpacy is a Spacy variant specialized for biomedical text. It can also annotate medical terms with UMLS labels.

I think that's all the main ones that I know of... so far.
From your description it looks to me like you probably just need cTakes or SciSpacy. In case you're going to start working with Medline or PMC, be aware that these datasets are massive (a few hundreds GBs).


Can features passed into Conv1d layers be randomized?

Typically in a Conv1D layer for a time series, the features can be different measurements taken or recorded at the same time period. So they can be related to each other. For example, if you are trying to predict time=4 below, the question is whether there is a relationship between your features, meas1 and meas2? If so, you want to keep the features together to give you a better result.
| time | meas1 | meas2 |
|------|-------|-------|
| 1    | 4     | 6     |
| 2    | 3     | 7     |
| 3    | 2     | 8     |
| 4    | ?     | ?     |

You can look at cross-correlation in your dataset to answer this. Since this is time series data, you can use Pearson's correlation for this.


I've been building web scrapers for over 5 years, and I have to say that a web scraper is rarely "relatively straightforward" simply because of how idiosyncratic each website is. It usually takes a minimum of 10 hours per site to code your scraper if you know what you're doing.

Whenever you have to interact with the page that you're scraping, I recommend Selenium. It's open source and works with most major languages, including Python, Java, and Scala. Moving a mouse around is possible, but I think in your case it might be easier to directly call the javascript that is triggered by the mouse movements. Your web scraper would iterate over all of the hoverable html elements and call the on-hover javascript on each element. However, the devil is in the details, and you're going to need to post a lot of questions on Stackoverflow and do a lot of Googling before you get to a final solution.

I've heard that Google has an API for their maps capability. That might be a lot easier.


Time Series Classification for loan data

It sounds like you are interested in multivariate time series forecasting. Below are some references to an example that sounds similar -- using multiple input variables to forecast air quality.

Kaggle competition- see Discussion tab for some nice examples and, well, discussion.
Corresponding tutorial


What is dilated pooling and how it works mathematically?

From Tensorflow Github:

Dilated max-pooling is simply regular max-pooling but the pixels/voxels you use in each "application" of the max-pooling operation are exactly the same pixels/voxels you would select with dilated convolution.
Dilated convolution/pooling are useful for connectomics and 3D shape datasets (3D deep learning).

So, similarly as in convolution every nth pixel is selected for the operation.
Edit
Selecting points like this:

making pooling for them like this:

Picture sources:
https://www.quora.com/Is-a-pooling-layer-necessary-in-CNN-Can-it-be-replaced-by-convolution
https://www.quora.com/What-is-max-pooling-in-convolutional-neural-networks
Edit2:
2 x 2 max pooling with 3 x 3 dilation:
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0 (5) 0  0 [6] 0  0 (3) 0  0 [6] 0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0 (3) 0  0 (4) 0  0 (2) 0  0 (4) 0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0 (1  0  0 (2) 0  0 (2) 0  0 (1) 0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0 (3) 0  0 [4] 0  0 [6] 0  0 (3) 0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0  0  0  0  0

becomes
6  6
4  6


Does gensim use Negative sampling in Word2vec?

Gensim's word2vec can use negative sampling. When calling class gensim.models.word2vec.Word2Vec, there is a parameter negative. The default amount is negative=5.

  negative (int, optional) – If > 0, negative sampling will be used, the int for negative specifies how many “noise words” should be drawn (usually between 5-20). If set to 0, no negative sampling is used.

The documentation is here.


How to train a neural network for high recall?

There is a very simple way of increasing the recall of a network without requiring retraining:

A network's output is the probability it gives that a sample belongs to a class, let's say class $0$ or class $1$. The output of the net would look like this $(0.53, 0.47)$. This means that the net gives a $53\%$ probability of the sample being class $0$ and a $47\%$ probability of it being class $1$. Normally we check which class has the highest probability and consider that to be the network's prediction. However, you could set a threshold over which the sample would be considered to be class $1$.

This way you would get more samples predicted to class $1$, which would increase the class' recall (at the cost of its precision though).


One-Hot Vector representation vs Label Encoding for Categorical Variables

The main difference I can think of is that using one-hot encoding will mean that all your strings will be at the same (hamming) distance from each other, while using a scalar value means that distances between the resulting features will be meaningless (it may encode "red" as 1, "blue" as 2 and "green" as 3, but there is no reason why red is more similar to blue than to green).


Text Dataset Help: Need to figure out how to get the publication date of a text

I would hazard a guess that most books uploaded on Gutenberg also have a Wikipedia page. Especially the well-known books. There are a few Wikipedia related Python packages (i.e. Wikipedia) that can make it very easy to search for the title of the book, navigate to the book's page, and extract the publication date. An optimal example might be Alice's Adventures in Wonderland. It'd be trivial to extract the date from that page, which appears neatly in the info box, and convert it to a machine readable format in Python.

Or, you could attempt to scrape the info straight from Gutenberg. Perhaps consider Selenium, although make sure you limit your bot's rate so you don't adversely effect the site.

Lastly, trying to predict the publication date might be similar to predicting the author; both may be derived from writing style. There's a lot of work on this topic, a Google search or an arxiv search will help. Here's one of the first articles I opened.


Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis

Yes, you can. Instead of using lda.fit(...)or lda.fit_transfrom(...) you basically just need to call lda.partial_fit(...). Here you take a mini-batch to update your model.
See this link for more information.


The figure looks exactly how it should.

plt.scatter(X[:, 0], X[:, 0], c=labels, s=50, cmap='rainbow');

You are plotting the a value again itself, in two dimensions it will give you a line of the form y = x
Check what you want to plot, if you have only one feature it doesn't mean that the K-Means wouldn't work, you will have a clustering, but the plot its correct.


Advice on imputing temperature data with StatsModels MICE

MICE does generate several datasets, but it does not then combine these datasets.  Rather, it fits your model on each of those datasets and combines those models.  If you really need an imputed dataset, you could just choose one or combine them in whatever way makes sense for your problem (or you might be better off with another method):

https://stats.stackexchange.com/questions/188288/how-to-know-which-imputation-is-best-for-impute-my-dataset-from-multiple-imputat
https://stackoverflow.com/questions/28814384/how-to-extract-aggregated-imputed-data-from-r-package-mice
https://www.theanalysisfactor.com/multiple-imputation-in-a-nutshell/

Now, for the statsmodels implementation, imp.data only keeps track of the latest imputed set [1]; you can loop through updates rather than using fit to get all of the datasets as in an example in [2].


What is the difference between GPT blocks and BERT blocks

BERT is a Transformer encoder, while GPT is a Transformer decoder:

You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the masking in the multi-head attention block.
There is, however, an extra difference in how BERT and GPT are trained:

BERT is a Transformer encoder, which means that, for each position in the input, the output at the same position is the same token (or the [MASK] token for masked tokens), that is the inputs and output positions of each token are the same.

GPT is a Transformer decoder, which means that it is meant for autoregressive inference. This means that the tokens in the input are shifted one position to the right with respect to the output, that is, if the output is [the, dog, is, brown, </s>], the input is [<s>, the, dog, is, brown, </s>].

Apart from that, at inference time BERT generates all its output at once, while GPT is autoregressive, so you need to iteratively generate one token at a time.


SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?

I am working on Random Forest Classifier and this classifier has probability attribute in prediction i.e if you get the summary of predictions = model.transform(testData) as print(predictions) in PySpark you will get the probability of each labels, You can check the below code and output of the code:

from pyspark.sql import DataFrame
from pyspark import SparkContext, SQLContext
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.feature import StringIndexer, VectorIndexer
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a Random Forest model.
rf = RandomForestClassifier(labelCol="label", featuresCol="features", numTrees=12,  maxDepth=10)

# Chain RF in a Pipeline
pipeline = Pipeline(stages=[rf])

# Train model.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

Now your work start from here. try printing predictions and values of predictions

print(predictions)

Output:

DataFrame[label: double, features: vector, indexed: double, rawPrediction: vector, probability: vector, prediction: double]

So, in the DataFrame you have the probability which is the probability of each indexedLabel, Further I have checked it as:

print predictions.show(3)

Output:

+-----+--------------------+-------+--------------------+--------------------+----------+
|label|            features|indexed|       rawPrediction|         probability|prediction|
+-----+--------------------+-------+--------------------+--------------------+----------+
|  5.0|(2000,[141,260,50...|    0.0|[34.8672584923246...|[0.69734516984649...|       0.0|
|  5.0|(2000,[109,126,18...|    0.0|[34.6231572522266...|[0.69246314504453...|       0.0|
|  5.0|(2000,[185,306,34...|    0.0|[34.5016453103805...|[0.69003290620761...|       0.0|
+-----+--------------------+-------+--------------------+--------------------+----------+
only showing top 3 rows

Only for probability column:

print predictions.select('probability').take(2)

Output:

[Row(probability=DenseVector([0.6973, 0.1889, 0.0532, 0.0448, 0.0157])), Row(probability=DenseVector([0.6925, 0.1825, 0.0579, 0.0497, 0.0174]))]

In my case I have 5 indexedLabels and so the probability vector length is 5, Hope this will help you to get the probability of each labels in your problem.

P.S: You will probably get the probability in Decision Tree, Logistic regression. Just try to get the summary of model.transform(testData).


Predicting results of tennis matches based on historical data

I think your current under-performance is a data problem. The dataset in your google sheet seems like it contains far too few features that you could conceivably use to predict whether someone would win a tennis match.
You should consider some creative feature-engineering options. Stuff like (but not limited to):

how many wins or loses in a row has this tennis player had
what is the weather during the game
how many years of professional tennis have they played
how many tournaments have they won (somehow encode their previous accolades)

Additionally - you might want to re-structure your data. The schema of having one player on the left and the other on the right of the same row will always make the machine learn that there's a distinction between right and left, when in fact this is just a bias you are introducing via featurizing.
You could consider a system whereby you double-up your data and put each match where player 1 is on the left and then the right. This could conceivably force your estimator to consider the match from both sides.
As for the estimator consider looking into some ensemble methods like gradient boosting classifier, or random forest classifier instead of logistic regression.


How to plot cost versus number of iterations in scikit learn?

Note The Code Is Self Explanatory(It's Hard-coded)..

  Here's the function which we are considering

def f(a,b):
    return a**2 + b**2

fig = plt.figure(figsize=(10, 6))
ax = fig.gca(projection='3d')
plt.hold(True)
a = np.arange(-2, 2, 0.25)
b = np.arange(-2, 2, 0.25)
a, b = np.meshgrid(a, b)
c = f(a,b)
surf = ax.plot_surface(a, b, c, rstride=1, cstride=1, alpha=0.3,
                       linewidth=0, antialiased=False,cmap='rainbow')
ax.set_zlim(-0.01, 8.01)

  Here's a 3D view of Gradient Descent Reaching The Optimum(in case interested, it will not always work, Have a look at the end plot..)

def gradient_descent(theta0, iters, alpha):
    history = [theta0] # to store all thetas
    theta = theta0     # initial values for thetas
    # main loop by iterations:
    for i in range(iters):
        # gradient is [2x, 2y]:
        gradient = [2.0*x for x in theta]
        # update parameters:
        theta = [a - alpha*b for a,b in zip(theta, gradient)]
        history.append(theta)
    return history

history = gradient_descent(theta0 = [-1.8, 1.6], iters = 30, alpha = 0.03)

fig = plt.figure(figsize=(10, 6))
ax = fig.gca(projection='3d')
plt.hold(True)
a = np.arange(-2, 2, 0.25)
b = np.arange(-2, 2, 0.25)
a, b = np.meshgrid(a, b)
c = f(a,b)
surf = ax.plot_surface(a, b, c, rstride=1, cstride=1, alpha=0.3,
                       linewidth=0, antialiased=False)
ax.set_zlim(-0.01, 8.01)

a = np.array([x[0] for x in history])
b = np.array([x[1] for x in history])
c = f(a,b)
ax.scatter(a, b, c, color="r");

plt.show()

And Here's the output we will get

  When Gradient Descent will fail (unfortunately)..

Unfortunately, if the function has many extrema, then the Gradient Descent could find the local minimum instead of global minimum. One trick is to overcome this disadvantage is to run SGD several times with different initial guessed values for $x$.


Annotating the vocabulary using Word2vec model

WordNet is certainly an interesting resource to explore for this task. It might not cover all your vocabulary but I can't think of any other way to capture fine-grained semantic relationship between words.


You could cluster your data with something like k-means and use the assigned cluster as a new feature.

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

data = load_iris()

X = pd.DataFrame(data['data'], columns=data['feature_names'])
y = data['target']

X_train, X_test, y_train, y_test = train_test_split(X.copy(), y, test_size=0.2)

km = KMeans(n_clusters=3)
km.fit(X_train)
training_dist = km.transform(X_train)

X_train['dist1'] = training_dist[:,0]
X_train['dist2'] = training_dist[:,1]
X_train['dist3'] = training_dist[:,2]

clf = DecisionTreeClassifier()
clf.fit(X_train, y_train)

# new data
test_dist = km.transform(X_test)

X_test['dist1'] = test_dist[:,0]
X_test['dist2'] = test_dist[:,1]
X_test['dist3'] = test_dist[:,2]

clf.predict(X_test)


How much is the Class Imbalance Problem rates?

There is no general rule but you better to use such techniques (SMOTE, sampling etc) and try to obtain 50:50 if you can.
You can utilize my another answer (just ignore multiclass part).

My suggestion is you should create more samples by synthetic data generators like SMOTE. You have a model with very less observations (197). The model may not fit well thus data generation may help your for that case.

Hope it helps!


Under what circumstance is lemmatization not an advisble step when working with text data?

NLP tasks that would be harmed by lemmatization:

1) Tense classification

      sentence        |  tense
------------------------------------
He cooked a nice meal |  past
He cooks a nice meal  |  present

The sequence of characters at the end of verbs can help in this task.
The verbs cooked and cooks differ at the last characters ed
 and s repectively.

With lemmatization, this information is lost. Both verbs become cook, making both sentences seem (in this case) in the present tense.

2) Author identification

Given

a set of documents $\mathcal{P}$ written by author $a$,
a set of documents $\mathcal{Q}$ written by author $b$,
a set of documents $\mathcal{S}$ written by either author $a$ or $b$,

classify if a document $s\in\mathcal{S}$ is written by author $a$ or $b$.

One way to achieve this is by looking at the histogram of words present in $s$ and compare it to documents from $\mathcal{P}$ and $\mathcal{Q}$ and select the most similar one.

This works because different authors use certain words with different frequencies. However, by using lemmatization, you distort these frequencies impairing the performance of your model.


xgboostclassifier prediction error after saving the model and restoring it

It's better to use joblib to save the model:

import joblib

#This one saves the model
joblib.dump(model, 'trained_model_full')
#This one loads the model
model= joblib.load('trained_model_full')


Xgboost - How to use feature_importances_ with XGBRegressor()?

from xgboost import XGBClassifier
model = XGBClassifier.fit(X,y)

# importance_type = ['weight', 'gain', 'cover', 'total_gain', 'total_cover']
model.get_booster().get_score(importance_type='weight')

However, the method below also returns feature importance's and that have different values to any of the "importance_type" options in the method above. This was raised in this github issue, but there is no answer [as of Jan 2019].

model.feature_importances_


What is the ideal size to a binary CNN? Is my dataset long enough?

There is no mathematical function that is able to relate size of model and performance.

Architecture(filter size, activation, depth, weights initialization) of a neural network can be thought as hyperparameter and hence to improve your model you need to employ hyperparameter tuning methods.

The other way to improve your model is to experiment with state-of-the-art features with your architecture. I believe your dataset being small enough give you a lot of room to experiment(because of the supposedly fast training time).


Closed form of Weighted Ordinary Least Squares calculation of the trend line

We found the answer:

The formula is really close to the original OLS closed formulation:

$\beta = \frac{n\sum{(wx)(wy)}-\sum{xw}\sum{wy}}{n\sum{(wx)^2}-(\sum{wx})^2}$

There is something that should be warned:
When the sample is small, the method will simply don't work at all, resulting in absurd estimations of the $\beta$.

Make sure your sample is big enough!


why to use Scaler.fit only on x_train and not on x_test for normalizing value using MinMaxScaler?

If you would fit the scaler using x_test you would be using information from your test set and would be leaking data. This is information that you would not have if your model was in production and can therefore not use when fitting your model.


First of all, you can project your explanatory variable (continuous) in your first plane (PC1 + PC2). The direction of the arrow (projection) and how far goes from the axis origin will tell you how the points are distributed according to this representation of variables in your factorial plane.

On the other hand, the quick answer is to group your continuous variables into chunks (descretizise your variable into an ordinal one), then you'll have the same plot as the reference.

Furthermore, you could try to color your scatter plot using a color scale (from white to black, red to blue...), and you'll see if there's some kind of progression of your data in that factorial plane according to the continuous variable.

These three "strategies" actually are showing the same, although the second one is more sensitive to the cuts.

Summarizing:

Project the continuous variable in the factorial plane (what PCA usually does).
Group your continuous variable in groups and plot it (easier way to see although it is sensitive).
Color your scatter plot using an scale (using min and max value of your continuous variable).


Isolation Forest Prediction Mechanics: Does it compare value with every tree (and the original training subset)?

You can find the sklearn implementation of Isolation Forest in Python at https://github.com/scikit-learn/scikit-learn/blob/a24c8b46/sklearn/ensemble/iforest.py#L229

It calculates the mean path depth needed to classify new samples. They are scored relatively to a theoretical average path length. Original training data is not used, only the learned trees.


Alteryx user, I was looking for the Unique or Summarise tools...
My work around is to use the Pivot Table widget
Particularly useful when you want to merge data and overcome the cartesian block of Orange.


What is difference between intersection over union (IoU) and intersection over bounding box (IoBB)?

The Intersection over Bounding Box is the Intersection over Union (IoU) for object detection tasks, where you have a bounding box.

There are many tasks (e.g. image segmentation) where you have an IoU (the predicted segment vs the actual segment), but there are no bounding boxes.


Python how to add a condition to the groupby while calculating the median of a field?

This should work:

train["Age"].fillna(train.loc[train["PrSb"] == 0].groupby("Ticket")["Age"].transform("median"), inplace=True)

This adds an extra rowwise filter to the dataframe that is used in the groupby operation, which only selects rows where the PrSb colum is equal to 0.


How do I get Twitter Dataset for Visualization

(Fresh and complete) Twitter data is not easy to get nowadays, as Twitter is actively monetizing it (through GNIP). There is still a large Twitter graph collected by Kwak et al. for their "What is Twitter, a social network or a news media?" WWW'10 publicly available, but as to tweets, most nice datasets have disappeared from the Internet due to complaints from Twitter. There are still tiny datasets of tweets available online (e.g.), but they are small and stale. Thus, I see two "legal" ways to go about getting tweets now: 1) crawl Twitter using the new version of Twitter API (the obvious limitation of this approach is the limit on the number of requests you can send; this approach is unlikely to give you a representative picture of what is trending if you collect raw tweets, just because you won't have enough tweets); 2) buy data from Twitter (GNIP), which will cost an awful lot of money. Alternatively, you can 3) study Twitter API for other means of getting "summarized" information from it, not tweets directly. For example, they may provide a method to retrieve, say, 10 most popular hash tags at the moment, maybe, even filtered by geo-location -- you cannot do much data science with the result, but at least you can use it in your visualization application.


You're right, the encoding step itself can be a source of data leakage and normally it should be done inside the CV loop using only the current training set, as you describe.
The reason is indeed the one you mention in the comment: if there is a class label or a feature category which doesn't appear by chance in a particular training set during CV, the model is not supposed to know that this class/category even exists.
In general I would think that this issue can only decrease the performance on the test set, so it's probably not as serious as other kinds of data leakage. Still, it's definitely a cleaner experimental design to encode using only the training set.
A closely related issue in NLP is when the system is not designed to deal with out-of-vocabulary (OOV) words: if all the words in both the training and test set are encoded (same mistake), then it wrongly looks as if any text can be fully encoded, potentially causing to bad surprises later.
That being said, it's usually a good idea to discard rare features or label values, and if this is done then the result should be the same using either the proper method or the sloppy one.


How to cross-validate a deep learning model for highly imbalanced datasets?

Accuracy is not a good indicator of success with imbalanced data. The accepted answer is correct: F1 score is commonly used. Other options include roc_auc_score (see here) and average_precision_score (see here), both defined through scikit-learn.

If you're using Keras, I would recommend using class_weights (note this will not work well if you have a multi-label problem, although there are some workarounds, for example here).


Speed for different kernels in scikit-learn's SVM

One option is to speed up your code is to standardize features, removing the mean and scaling to unit variance.
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X = scaler.fit_transform(X.reshape(-1, 1))
y = scaler.fit_transform(y.reshape(-1, 1))

After I standardized the data, each model fit in less than 500ms.
Support vector machines (SVMs) need to calcuate the distance between each data. Standardize numbers make the calculations faster.


You can weight the loss of each class with a suitable value which is inversely proportional to the class size. One example would be to use: (Total number of data points)/(number of data points of the class). For loss you can use the standard Binary Cross Entropy (BCE) loss.

As for the metric, I would advise you to use AUROC (Area Under the Receiver Operating Characteristics). You can generate an AUROC for each class to see the performance of the model on these individual class and average them to get an overall sense of your models performance.

Edit: I noticed you have put a tag on f1 score so I would like to point out that AUROC should be preferred as the metric as they more robust to class imbalance as compared to f1 scores.


what should i do if my target variable is categorical when using decision tree? (many categorical variables)

This is called multiclass classification, and the encoding needed for the target variable depends on what package and model you're using.  You may be expected to one-hot encode (e.g. neural networks usually have an output neuron for each class), ordinal encode (e.g. most(?) sklearn multiclass classifiers), or leave them as strings (most R models, I'd guess?).


I had the same doubt when I was doing my Master's Degree. First of all, you don't include something as random as 'IDs'. This is where data preprocessing comes in. Let's take a dataset which has users and their movie genre preferences based on variables like gender, group of age, rating, blah, blah. With the help of information gain, you split at 'Gender' (assuming it has the highest information gain) and now the variables 'Group of Age' and 'Rating' could be equally important and with the help of gain ratio, it will penalise a variable with more distinct values which will help us decide the split at the next level.


Which is the "most properly working" Bert-Ner repository

While not NER specific, the go-to PyTorch implementation of BERT (and many other transformer-based language models) is HuggingFace's PyTorch Transformers. There is plenty of documentation to get you started.

They also have models which can directly be used for NER, such as BertForTokenClassification.

I am not sure if you were looking for something higher-level, but as the question doesn't specify, this is the answer I would give to someone looking to use BERT in PyTorch.


Guys I am trying to create a application to create training data

The error in the line pickle_out = open(X.pickle,'wb') is that the name of file must be in quotes. So the correct code would be:

pickle_out = open('X.pickle','wb')
pickle.dump(X,pickle_out)
pickle_out.close()

Do the same for Y also. Also you can do it more concisely in the following way:

with open('X.pickle', 'wb') as f:
    pickle.dump(X, f)


How to apply Machine learning model on time series to predict next time step

As your question is quite vague I just throw in some tools: MCMC, Kalman, LSTM, ARIMA, one-step-ahead and many more.
Have a look at this to get started.

Can you please provide a part of your data? There are a lot of tools but it depends a lot on your situation and data.


you can print the coefficients with:
coef(fm$finalModel,model No. e.g.1)


What are the top schools for online part-time Masters in data science/analytics programs? Is it much better to do the program in person?

Michigan University has a nice and good dual program that you get your Master in Data Science and also a MBA.
I'd take a look at MIT's Master of Business Analytics as well.

If you want to get a job, I don't think a Master is necessary. This Forbes article explains that pretty well https://www.forbes.com/sites/metabrown/2016/07/29/4-reasons-not-to-get-that-masters-in-data-science/#160f195a40c0


Did you evaluate the results in the training set? Or in the test set?

Those results are outstandingly good! Suspiciously good.

I think you tried your results in the training set only, so your results reflect overfitting on your data, which means your model learned the set, it was not generalized (which means it is unapplicable to any other dataset you may encounter in the future, which is not useful).

For comparing ROC between methodologies you should model them being careful for overfitting and try them on a test dataset (a dataset which you never knew before, which can be obtained partitioning your dataset).

In that way your comparison is not measuring which model learns by memory your data.


Some questions about feature hashing in the context of document classification

Throughout the hashing process those words may map to the same indices and for the example you gave, i'll say it's extremely likely you'll have collision.  How the hashing table chooses to handle that is up to the implementation.

Feature hashing may purpose is when you have a large amount of data and that data tends to be sparse.  Words tend to be a good example because word matrix of counts tend to be rather sparse.

The hashing trick is useful for large datasets because the more traditional ways of handling text data is essentially by making two passes on the dataset.  You basically have to go over the dataset once to create a dictionary and then make your transformation.  For large data this can be rather expensive process.  Feature hashing allows for a one time pass of the data.  This also enables the ability to do online learning, which brings me to this point.  I wouldn't say that feature hashing helps with new words rather that it basically handles it for you.   In traditional approaches you have to ignore the new word or create a new dictionary.  Feature hashing will simply just map it to an existing index or a new one if there's space.

The last point is that feature hashing rarely if ever improves performance of a model.  It also hurts the ability to be introspective.  You are going to lose information, and in fact, you can view feature hashing as a way to introduce noise.  However, research indicates that in large sparse data, the noise does not impact performance in major ways.

Hashing Function

Large Scale Hashing

Short idea behind hashing


If I understand correctly, you're trying to predict the target variable 'Full_Time_Home_Goals' based on the four features ['HomeTeam', 'AwayTeam', 'Full_Time_Away_Goals', 'Full_Time_Result'] (the first two being being categorical and one-hot-encoded).
In your test set, you only receive (or only use) the first two, and fill the last two (which are, naturally, the more important ones) with a value of 1.
Now, let's imagine that your model decides that the teams aren't important, and makes the prediction based solely on the two latter variables... but in your test set, those all have the same value - so they'll get the same prediction.
The lesson here is that when you're training a model, do not use features that are (mostly or completely) missing from the test set.


One-hot encoding is the way. For ARIMA models, no other configuration is possible. Whether it is good or not, it depends on the quality of your data, your variable choice, and a correct model specification. Unfortuntately, we can't say a priori whether the result is going to be good or bad.


COCO is a python class and getCatIds is not a Static Method, tho can only be called by an instance/object of the Class COCO and not from the class itself.

You can probably solve it by doing this instead:

a = COCO() # calling init
catIds = a.getCatIds(catNms=['person','dog', 'car']) # calling the method from the class

(answer based on reading the code from https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py)


How to extract features from the encoded layer of an autoencoder?

You are using a dense neural network layer to do encoding. This layer does a linear combination of the input layers + specified non-linearity operation on the input.

Important to note that auto-encoders can be used for feature extraction and not feature selection. It will take information represented in the original space and transform it to another space. The compression happens because there's some redundancy in the input representation for this specific task, the transformation removes that redundancy. Original features are lost, you have features in the new space.

Which input features are being used by the encoder? Answer is all of them. For how exactly are they used? Answer is you can check the weights assigned by the neural network for the input to Dense layer transformation to give you some idea. You can probably build some intuition based on the weights assigned (example: output feature 1 is built by giving high weight to input feature 2 & 3. So encoder combined feature 2 and 3 into single feature) . But there's a non-linearity (ReLu) involved so there's no simple linear combination of inputs.

If your aim is to get qualitative understanding of how features can be combined, you can use a simpler method like Principal Component Analysis. The factor loadings given in PCA method's output tell you how the input features are combined. If the aim is to find most efficient feature transformation for accuracy, neural network based encoder is useful. But you loose interpretability of the feature extraction/transformation somewhat.


Keras is used in academia (see google scholar citations of Keras as a proxy for academy adoption) and hobbyists (see github stars or google results for keras in www.kaggle.com as proxies for hobbyist adoption).

It was recently bundled together with Google's tensorflow.

It is also used in industry, at least for prototyping (I know because I use it!), but I have found no source to back this statement.


Including validation set in my code for a Linear SVM classifier returns a Type error

Based on the repo you linked in the comment section (github.com/prrao87/fine-grained-sentiment), method_class is meant to be a Class not an instance. Therefore method_class.predict is meant to be a static method not an instance method, Static method are declared without specifying a self (as they are not linked to an instance but a class) declaring your function as def predict(train_file: str, test_file: str, lower_case: bool) should fix your problem.
Some reference on the topic : https://www.makeuseof.com/tag/python-instance-static-class-methods/


K-Nearest Neighbours algorithm explanation needed

K Nearest Neighbors is a Classification Algorithm. Just as with every classification algorithm is important that the algorithm doesn't "remember" the answers and that the answer it gets can be generalized to all the population, not just learned from the database.

1. Why is the training process needed in KNN algorithm? In regression model the training process means to find optimum parameters for a selected function. Does the training in KNN mean that the similarity distances are calculated between TRAINING datapoints? Why is it good for a later prediction phase?

The traininig process is needed in KNN because (as I stated before), we want to avoid it from learning-the-database, the famous "overfitting".

As you stated, in a regression the training is for finding the optimum parameters, here too, but the idea of "parameters" slightly changes: Here the parameters are the selected points (individuals) and of course, the $k$, separating in train/test allows the model to be more general when the test data is unknown.

Yes, it means the KNN calculates distance only in the training set. When the optimum parameters are found in the training set, the results in the test set are better, more general.

Are similarity distances also calculated between TRAINING and TESTING datapoints? If so then why is it necessary at all that before that the distances between TRAINING datapoints were calculated?

The new datapoints (the ones from the TEST dataset) do not participate in a recalculation of KNN, the way the new points decide if they are blue or red is determined by the points which were calculated before (TRAIN dataset), by measuring which color the k nearest neighbors (of the new point) have.

For the new green point, with $k=3$, two were blue and one red, the new color is likely to be blue (and is assigned blue).

3. If the following code is executed then what does it exactly do? Does it mean that distances between TRAINING datapoints are calculated?

neigh = KNeighborsClassifier(n_neighbors = k).fit(x_train, y_train)

No, a more thoughful explanation would be that they are "memorized" in certain position and certain y_value. The calculation of distances is done when fitting TEST data.

4. If the following code is executed then what does it exactly do? The manual says that class labels for each data sample are returned. But how is it done? Does it mean that distances between TRAINING and TESTING datapoints are first calculated? If not then how can we find neighbours for "unseen datapoints" at all? We need to know distances.

yhat = neigh.predict(x_test)

As was explained in another point, the former code (fit) did not calculate distances, the code memorized the points. In the predict code, distances between x_train and x_test are computed, and for every member of x_test, the k nearest neighbors (from x_train) are found, and they vote to retrieve a class for this new datapoint (in x_test).

In the image, when $k=3$, they vote 2-1 to say that the green point (new one) is blue.


RStudio/RMD files to create a student version assignment from the instruction version assignment solution

How about learnr?

Here is the link to the package information:
https://cran.rstudio.com/web/packages/learnr/index.html


Cluster a categorical variable without breaking the existing categories

Without looking at the data, can't you just use your Domain Knowledge to create meaningful groups/attributes for the airlines data frame? Such as "low-cost carrier/premium", "foreign (non-USA) airline/US-Airline" , "long-distance carrier/domestic only", "founded in decade" ,  or whatever? With only 14 airlines you can assign appropriate values yourself, manually, without relying on some cryptic algorithm.

Besides, is your dataset identical to the dataset in the R package "nycflights13"?

If so, you could easily load it with library(nycflights13), after installing the package of course.


Custom keras dataset generator not accepted by fit_generator

I was following the instructions on a tutorial but it seems that there was a mistake or that the internal code of TensorFlow has changed since it was written. I was defining my class like this:

class DataGenerator(tf.keras.utils.Sequence):

and later on I tried also this:

class DataGenerator(tf.keras.models.Sequential):

(which, by the way, is completely different and is totally wrong)

Inspecting the code for some of the adapters, I saw this:

class KerasSequenceAdapter(GeneratorDataAdapter):
  """Adapter that handles `keras.utils.Sequence`."""

  @staticmethod
  def can_handle(x, y=None):
    return isinstance(x, data_utils.Sequence)

and also for my other suspect adapter:

class GeneratorDataAdapter(DataAdapter):
  """Adapter that handles python generators and iterators."""

  @staticmethod
  def can_handle(x, y=None):
    return ((hasattr(x, "__next__") or hasattr(x, "next"))
            and hasattr(x, "__iter__")
            and not isinstance(x, data_utils.Sequence))

where data_utils is defined as:

from tensorflow.python.keras.utils import data_utils

So, I changed the definition of my class to this:

from tensorflow.python.keras.utils import data_utils
class RandomFramesFromPathsToVideos(data_utils.Sequence):

and now it works.

I've got, so, two options. Using a subclass from data_utils.Sequence is one of them. In that case, we are talking about a "Keras generator" (handled in TensorFlow by KerasSequenceAdapter) in which you need to define two methods: __len__ and __getitem__

The other one is building a new class NOT DERIVED from data_utils.Sequence, and defining the methods __iter__ and __next__ (or simply next). In that case we are defining an standard Python generator, which will be handled by GeneratorDataAdapter inside TensorFlow

Please, bear in mind that a Keras generator is not the same thing as a Python generator. At first, that confused me a lot.


Cold start recommender system with features

Since you have access to user level features, try a k nearest neighbors recommender algo.
When a user signs into your site, find the k most similar looking users (based on a similarity metric such as cosine similarity), and recommend products that those users have purchased.


Machine Learning algorithm to predict an outcome where output is not known

What I was looking for is called ordinal regression or ranking learning (https://en.wikipedia.org/wiki/Ordinal_regression)

Edit: I will elaborate more on why this solved my issue
What I was having a problem with was finding a statistical model to appropriately predict the ordinality of multiple independent options. What I cared about was the order of the options and less so about what the value of the options was.

Ordered Logit/Probit describe such models. The provided wikipedia link explains them in far more detail than I can.


Loading collections of datasets - Python code examples

How to fetch Kaggle data from python code?

Install kaggle package
C:\Users\TalgatHafiz>  pip install kaggle

login to your Kaggle account
click on the icon in the upper right corner -> My Account
Scroll down to API section
Click "Create New API Token"
"kaggle.json" file is created and saved locally

Create ".kaggle" dir
C:\Users\TalgatHafiz>mkdir .kaggle
and move "kaggle.json" into that directory

see all active competitions by running the following command
C:\Users\TalgatHafiz>kaggle competitions list

Select one of the competitions that you signed up for, eg
https://www.kaggle.com/c/contradictory-my-dear-watson/data#
Scroll down. Right before "Data Explorer" section there should be API line:
"kaggle competitions download -c contradictory-my-dear-watson"
copy it

run these commands from the notebook
import kaggle
!kaggle competitions download -c contradictory-my-dear-watson

zipped data file is downloaded into your the same directory where your notebook is:
C:\Users\TalgatHafiz\conda\contradictory-my-dear-watson.zip
so now you can unzip and start using the data

If you still have questions please read
https://medium.com/@jeff.daniel77/accessing-the-kaggle-com-api-with-jupyter-notebook-on-windows-d6f330bc6953


does KNN have a hypothesis space? if yes, Is there a way to clearly define the hypothesis space with the instances?

The hypothesis in this case is a way to formally describe the model. The model could be a hyperplane, a set of conditions on input with a constant for each branch (decision trees) or whatever you like. The set of all possible hypothesis in a given representation is named hypothesis space. Knn prediction is realized using the first k instances from a training data set. So knn cannot represent a hyperplane because of the finite amount of datapoints. This is why choosing a representation restricts the hypothesis space. You are restricted to things which can be represented by instances and distance. SVM is the same because the support vectors are selected instances from the training dataset and the fitted surface is obtained using support vectors and kernelized distances from those.


How to tell a boosting model that 2 features are related and should not be interpreted stand-alone?

I am going to talk about some ways you could do it later but first I want to talk about whether you should!

If the relation that you describe exists XGB will be able to learn and detect it! There is no real benefit in "hard-coding" a rule into the algorithm, it won't speed up the training, it won't benefit accuracy, etc. Simply put the benefit of ML algorithms is that they are able to detect exactly these relationships and model them in the best possible way.

Now if you still insist that this is something that must be done, you can. The easiest way to achieve this would be feature engineering:

Introduce NAs -simply leave out hair length of male respondents and fill with NA
Create interaction factors - instead of having hair length and gender as a simple variables you could also code it in a way that represents the known interaction like this:

gender_hair = [male, female_short,female_medium,female_long] # example factor levels

But again if you compare models with those engineered features to a simpler model you will see no benefit I'd wager.


What are the differences between multivariate data visualizations and multidimensional visualizations?

I don't think there is a well-defined (or well enough defined) terminology for that.

Multi-variate means that it involves multiple (dependent or independent) variables. Which is pretty clear.

Multi-dimensional in mathematical sense means that the data points are from a multi-dimensional space (typically vector space). This is exactly the same as multi-variate. In this sense, your visualization is both multi-variate and multi-dimensional. And both the size of the markers and the color represent individual dimensions or variables, which is then equivalent. Also, time would be an individual dimension. Personally, I would go with this interpretation of the term multi-dimensional.

If you want to restrict the term multi-dimensional to spatial dimensions of the plot, then size and color of markers would not be dimensions in that sense. I personally find this interpretation confusing. Time then also would not be a dimension in this sense, unless you represent it by a 3rd axis in the graph. This, again, is confusing as many people actually often put time into the same category as spatial dimensions.

So, all in all I would stick to the mathematical interpretation of dimensions here so that multi-variate and multi-dimensional will be equivalent.


Ensemble Techniques - Bagging | Subset size

like said in a previous answer, the exact subsample parameter value depends on your data.

But a usual starting parameter that gets you good results in general, and doesn't hurt the data distribution much is 0.9.

Taking out 10% of your data at each iteration or newly constructed tree, makes your model generalize a little better. You can try out different variations of the 0.9 and see the results.


What are the best practices to anonymize user names in data?

I suspected you were using the names as identifiers. You shouldn't; they're not unique and they raise this privacy issue. Use instead their student numbers, which you can verify from their IDs, stored in hashed form. Use the student's last name as a salt, for good measure (form the string to be hashed by concatenating the ID number and the last name).


How does LightGBM deal with value scale?

Generally, in tree-based models the scale of the features does not matter. This is because at each tree level, the score of a possible split will be equal whether the respective feature has been scaled or not.

You can think of it like here: We're dealing with a binary classification problem and the feature we're splitting takes values from 0 to 1000. If you split it on 300, the samples <300 belong 90% to one category while those >300 belong 30% to one category. Now imaging this feature is scaled between 0 and 1. Again, if you split on 0.3, the sample <0.3 belong 90% to one category while those >0.3 belong 30% to one category.

So you've changed the splitting point but the actual distribution of the samples remains the same regarding the target variable.


This is a very ambitious project. First it's important to realize that ML cannot really solve this kind of problem in general, it can only help detect the posts which are likely fake news (see for example this other post about measuring credibility, i.e. the same question seen the other way around).

Assuming you work on the text of the message (I'm not competent about images), the first step would be to collect a corpus of data from all your sources and manually label all of it as fake or not (you could also decide to use a score of "fakeness" for instance). From there you could train a model: at first I would suggest something simple like Naive Bayes (it's the traditional model used for spam detection).

In any case it's important to keep in mind that what the model would learn is not whether the input text is a fake news or not, it would only learn to recognize the tell tale signs: for instance fake news tend to use scary words whereas real news tend to be worded more neutrally. But of course there can real news using scary words and fake news using neutral language, so it's not going to be perfect. In order to really reach the goal the only way would be to have the potential fake news checked by humans, who would be able to give a more informed judgement.

The last part of the question is more a legal question. As far as I know it's often very hard on social media to discover who is really the author, only the police can investigate this but they do it only in very serious cases.


How important is the channel order in deep-learning computer vision tasks?

Task dependent but could be important.

Lets find some arbitrary representation that separates luminance from chrominance (represented as a color 2D vector)? That way, you are detected bright objects independent of color, and colorful objects less dependent on brightness. Obviously under assumption that you are color agnostic. Check this paper where they applied exactly this augmentation


How to build a webscraper for indeed.com

Scraping a website is very particular to the case. To try and make this answer useful to other people I will go through some basic concepts as to how this can be done successfully. Below I will use your URL as an example to put everything together.

First we need a means to access the HTML content of the website. For static websites we can use BeautifulSoup. This is a very powerful package which allows you to navigate an HTML. To get the HTML we will use the requests packages.

from bs4 import BeautifulSoup
import requests

url = 'https://www.indeed.com/jobs?q=data+scientist+$20,000&l=New+York&start=10'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'lxml')

BeautifulSoup requires a parser, I have had a lot of luck using lxml, however html.parser is also very popular. The parser is what is used to access the HTML tags and identify its inner elements.

Most websites are quite hard to scrape because they will reuse the same name for multiple element tages. This makes it harder to get the elements and extract their values. This is either due to the developers laziness or their expertise to avoid people abusing their resources by running scrapers on their data. To circumvent this you can use this simple BeautifulSoup trick. You can chain your .find calls to hone into the part of the HTML you want to scrape.

Using the above example URL we can identify where in the HTML we can find the results on the page for each job posting.

len(soup.find_all('div', {'class': 'row'}))

  16

Now we will want to collect data from each of these HTML elements. The data we will want to collect is: job title, company name, job description, city, location, and salary.

Each record of data will be collected into a dictionary with all the desired data as listed above. These dictionaries will be collected into a list which will be used to create the pandas DataFrame.

To find the individual data points you can use your browser and inspect elements to find out the name of the element parameters in order to use BeautifulSoup to get the information.

data = []
for i in soup.find_all('div', {'class': 'row'}):
    job_title = i.find('a', {'data-tn-element': 'jobTitle'})['title']
    company_name = i.find('span', {'class': 'company'}).text.strip()
    job_summary = ''.join([j.text.strip() for j in i.find_all('span',
                                                              {'class': 'summary'})])
    location = i.find('span', {'class': 'location'})
    if location is not None:
        location = location.text.strip()

    salary_range = i.find('span', {'class': 'no-wrap'})
    if salary_range is not None:
        salary_range = salary_range.text.strip()

    datum = {'job_title': job_title,
             'company_name': company_name,
             'job_summary': job_summary,
             'location': location,
             'salary_range': salary_range}

    data.append(datum)

df = pd.DataFrame(data)
df.head()


Evaluating loss for non classifying convolutional neural network

My wild guess is to weight each layer a "wrongness" factor (e.g if instead of wanting to move the first pawn , it tries to move the queen , it is labelled as really "wrong" and then apply some kind of spacial locality (e.g if it doesn't move the first pawn but the second and on the very right case, it is not very wrong). But is it correct ?

You can relatively simply teach a policy network to predict human-like moves in your scheme using a database of moves. Actually your "wrongness" is probably well-enough represented by classification (your positive class might be "This is a good move") and the usual log loss that goes with it.

After you have trained a policy network, you will want to look in depth at the literature for game-playing bots. Your policy network might work quite well alongside Monte Carlo Tree Search, provided you have some kind of evaluation heuristic for the resulting position.

A reinforcement learning approach to learn from self-play would take you further, enabling the bot to teach itself about good and bad moves/positions, but is too complex to explain in an answer here. I suggest look into the subject after training your network and seeing how good a player you can create using just the policy network and a move search algorithm.

  And in general , how to compute the loss of a non classifying convolutional neural network ?

There are a few common options available for regression, such as mean square error (MSE) i.e. $\frac{1}{2N}\sum_{i=1}^{N}(\hat{y}_i - y_i)^2$ where $\hat{y}_i$ is your prediction and $y_i$ is the ground truth for each example. If you use this loss function, and want to predict values outside of range 0-1, remember to use a linear output layer (i.e. no activation function after the last layer), so that the network can actually output close to the values you need - that's about the only difference in network architecture you need to care about.

In the more general case of game-playing bots, it is usual (but not required) to calculate a predicted "return" or "utility" which is the sum of all rewards that will be gained by continuing to act in a certain way. MSE loss is a good choice for that. Although for zero-sum two player games where the reward is simply win/lose, you can use a sigmoid output layer (predicting chance of a win) and cross-entropy loss, much like a classifier.

For your specific case, you can treat your initial policy network as a classifier. This immediately gives you some probability weightings for the predicted move, which you can use to pick the predicted best play, or maybe to guide Monte Carlo Tree Search.

The kind of network that predicts utility or return from a position (or combination of position and action) is called a value network (or an action-value network) in reinforcement learning . If you have both a policy network and a value network, then you are on the way to creating an actor-critic algorithm.


Best approach for A/B testing two different recommendation systems

If user rates how good each recommendation then ABABAB approach is the best one. There could be case when system A becomes better than B overtime, then having 2 system does not make sense.


How to implement handmade features in a Keras Sequential model?

Very interesting  question , I am  wondering if there  are nough skilled  data science  entusiast   that  could   answer this question , because I am  interested in simmilar stuff as well.
Congratulations  for this  question  man


AI that maximizes the storage of rectangular parallelepipeds in a bigger parallelepiped

This looks very much like a variation of the bin packing problem. The bad news is that this is mathematically a hard problem, NP-hard in fact (which means that no polynomial-time solution is possible). The good news is that it is heavily studied, with multiple approaches to solving it, or at least optimising solutions to within reasonable bounds.

Which approach to optimisation will work best for you will depend on how physically accurate you want your model to be. Your voxel approach already makes some decisions in that regard, but you may also want to consider issues such as gravity, stability of any piles of structures you create, and access to items. These things add additional constraints that make modelling the problem and implementing an optimiser harder, so you might want to start with more unrealistic model initially.

One simple approach might be to use a simple heuristic e.g. pack largest objects first, fill close to edges first etc. Then attempt to swap or revise the order of packing of a few pieces and see if it makes an improvement. You can make those changes greedily, or look into making them more randomly but with a system that rewards better improvements. Approaches such as simulated annealing or genetic algorithms can be used as stochastic approaches, many others are possible. There are a very large number of global optimisers that might help with such a combinatorial problem.

Here is an example of solving a bin packing problem using Simulated Annealing.

Here is an example of solving a bin packing problem using a Genetic Algorithm


How to define a MLP with multiple outputs

Have four neurons in your output layer, depicting gain $A, B, C, D$ and apply relu activation function throughout all your layers since it sets all negative values to $0$ and passes the rest.

Also you can use a simple mean squared error as your loss function. This setup will allow you to predict non-negative values, greater than $0$ for your gains, which is, what I believe, you require.


Multiclass semantic segmentation with some classes possibly not present in some of the images

That problem is commonly called weak labels and is associated with weak supervision.
Weak labels are often not handled by the loss function. Most conventional loss functions assume perfectly labeled data points and have no mechanism to handle label uncertainty.
An alternative framework for weak labels is Confident Learning which

characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence.


There are different flavors of Naive Bayes, so the answer depends a bit on the use case.

One potential issue with outliers is that unseen observations can lead to 0 probabilities. For example, Bernoulli Naive Bayes applied to word features will always produce 0 probabilities when it encounters a word that wasn't seen in the training data. Outliers in this sense can be a problem. However, all these and similar issues of Naive Bayes have well-known solutions (like Laplace smoothing, i.e. adding an artificial count for every word) and are routinely implemented.

In Gaussian Naive Bayes, outliers will affect the shape of the Gaussian distribution and have the usual effects on the mean etc.

So depending on your use case, it still makes sense to remove outliers.


How to interpret a decision tree correctly?

Let me evaluate each of your observations one by one, so that it would be more clear:

  The dependent variable of this decision tree is Credit Rating which
  has two classes, Bad or Good. The root of this tree contains all 2464
  observations in this dataset.

If Good, Bad is what you mean by credit rating, then Yes. And you are right with the conclusion that all the 2464 observations are contained in the root of the tree.

  The most influential attribute to determine how to classify a good or
  bad credit rating is the Income Level attribute.

Debatable Depends on how you consider something to be influential. Some might argue that the number of cards might be the most influential, and some might agree with your point. So, you are both right and wrong here.

  The majority of the people (454 out of 553) in our sample that had a
  less than low income also had a bad credit rating. If I was to launch
  a premium credit card without a limit I should ignore these people.

Yes, but it would also be better if you consider the probability of getting a bad credit from these people. But, even that would turn out to be NO for this class, which makes your observation correct again.

  If I were to use this decision tree for predictions to classify new
  observations, are the largest number of class in a leaf used as the
  prediction? E.g. Observation x has medium income, 7 credit cards and
  34 years old. Would the predicted classification for credit rating =
  "Good"

Depends on the probability. So, calculate the probability from the leaves and then make a decision depending on that. Or much simpler, use a library like the Sklearn's decision tree classifier to do that for you.

  Another new observation could be Observation Y, which has less than
  low income so their credit rating = "Bad"

Again, same as the explanation above.

  Is this the correct way to interpret a decision tree or have I got
  this completely wrong?

Yes, this is a correct way of interpreting decision trees. You might be tempted to sway when it comes to selection of influential variables, but that is dependant on a lot of factors, including the problem statement, construction of the tree, analyst's judgement, etc.


How does Seaborn calculate error bars when using estimators other than the arithmetic mean?

This information is in the documentation of Seaborn.
They show a bootstrap confidence interval, computed by resampling units (rows in the 2d array input form). By default, in seaborn version 0.8.1 it uses 95% of confidence interval, which is equivalent to a standard error. This value is a parameter that can be changed.


Using PCA to cluster multidimensional data (RFM variables)

Judging from the plot, there are no clusters.

K-means requires continuous variables to work well. The data you have has discrete steps (which causes the grid pattern in your plot).

There is no benefit of using PCA here. Use it only for visualization. The scale -3:3 that you don't understand is from PCA. So you probably have not understood PCA enough either.


Let's see. I modified the string a bit, so that it can be saves as a dictionary.

g = {(8758148.0, 'CI Alpine Growth Equity Fund'): 'D    0.000016 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI American Growth Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI American Small Companies Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI American Value Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Canadian Equity Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Canadian Investment Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Canadian Small Cap Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Canadian Small/Mid Cap Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Global Bond RSP Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Harbour Fund'): 'D    0.0 Name: 8758148.0, dtype: float64' }

Now here is the code:

df = pd.DataFrame([i[0] for i in g.keys()], columns=['id'])
df['Name'] = [i[1] for i in g.keys()]
df['Value'] = [i.split()[1] for i in g.values()]
df

    id          Name                            Value
0   8758148.0   CI Alpine Growth Equity Fund    0.000016
1   8758148.0   CI American Growth Fund 0.0
2   8758148.0   CI American Small Companies Fund    0.0
3   8758148.0   CI American Value Fund  0.0
4   8758148.0   CI Canadian Equity Fund 0.0
5   8758148.0   CI Canadian Investment Fund 0.0
6   8758148.0   CI Canadian Small Cap Fund  0.0
7   8758148.0   CI Canadian Small/Mid Cap Fund  0.0
8   8758148.0   CI Global Bond RSP Fund 0.0
9   8758148.0   CI Harbour Fund 0.0


I haven't read the paper you linked, but I have followed a lecture notes by Richard Socher.

So, basically that mapping matrix is called weight matrices. There are two weight matrices W1 and W2 for input and output mapping. Thus for each word, vectors in both the matrices are updated via backpropagation.

To answer your question, a word is represented by one-hot sparse vector which has the size of Vx1 (V is size of vocabulary), with a value of 1 in one of the position. and when this vector is multiplied with the weight matrix W1 with size NxV, the corresponding embedding vector of size Nx1 (N is size of the required embedding vector) is used in the hidden layer.

Document has no one-hot represented vectors. So, basically there is a document matrix D of size Nxd (d is number of documents) where each column represents a document. In other words, the matrices W1 and W2 need the one-hot representation only for mathematical steps, other than that they are representing each word in each column just as the document matrix D.


difference between XGBRegressor and XGBClassifier

XGBRegressor is for continuous target/outcome variables. These are often called "regression problems."

XGBClassifier is for categorical target/outcome variables. These are often called "classification problems."


Recognize and predict a periodic pattern

If it is really close to random (background noise, bits from random number generators or hash codes), I don't expect there is much you can do, except create a large collector. (Or are you only allowed one collector?)

You may want to throw in a few random checks. Those won't prove anything but may give you some insight into the pattern.

If you want to create a successful model, you'd have to come up with some assumptions about parameters that could affect the success of your model, and create and test that model.

For instance, one of my assumptions would be that there are rows with higher probability.
One other assumption would be that there is some hysteresis effect (either positive or negative), so the probability may depend on the actual outcome of the previous event(s).


Define a Model for predicting missing values in Data Set

You could try a classification approach like random forests for categorical data.
For example in your brand name assumption, Take your existing data and treat the brand name as the response variable. So when you give the values of the other features in the entry where the brand name is missing it should try to predict what the brand name is.

The reason I am suggesting random forests is that it uses an ensemble approach. Please explore the working of Random Forests if you don't know the working behind it.


How are the channels handled in CNN? Is it independently processed or fused?

Let $n$ be a convolutional layer with dimensions $w' \times h' \times c'$. Then each of its $c'$ filters is connected to all $c$ filters (or channels*) of the previous layer.

I find it helpful to look at the number of weights here: A single filter of that convolutional layer $n$ with kernel size $k'\times k'$ will have $c \times k' \times k'$ weights. And since layer $n$ has $c'$ of these filters it has a total of $c \times k' \times k' \times c'$ weights.

In a toy example with a 3 channel input layer followed by a conv. layer with 5 channels this would look as follows (not showing the bias here for simplicity):

As you can see from the drawing each feature map of the conv. layer receives all input channels as an input (and the same would apply if this was not an input layer but a conv. layer with 3 feature maps).

*Note that it does not make a difference whether the previous layer is a conv. layer too or the input layer - in the first case you call its depth "filters" and in the second you call it "channels" but that does not change how it is connected to the following conv. layer.


Machine learning - 'train_test_split' function in scikit-learn: should I repeat it several times?

You can use KFold cross validation if you want to average the results of the training.

sklearn.model_selection.KFold

This will split your data in a specified number of folds (k) and train your data on all but one folds, then validate on the last fold. This operation is done k times and the results are averaged out.

Normally, how I do is:

Train/Test split
Model selection + Hyperparameters tuning using KFold on training set
Retrain the final model on the whole training set
Evaluate on the test set

Note that if you want to check whether your split was 'lucky' or 'unlucky', you can still change the seed, or not give a seed at all and compare the results with different runs.

[EDIT] As stated in the comments below, the seed is controlled by the random_state argument and is mainly there for reproducibility. If you want a different train/test split at each run, just leave the default as is. It's always good to check at least twice to see whether you've been particularly lucky or not but it never happened to me ! :)


How to perform tokenization for tweets in xlnet?

You need to do pip install sentencepiece for it to work.
By the way, you can also give the tweets as a list to the tokenizer. You don't need to tokenize them one by one.
tokenizer(tweets, max_length=max_len, padding='max_length', add_special_tokens=True)


How to replace words in a sentence with their POS tag generated with SpaCy efficiently?

You can not replace tags in a sentence because Python strings are immutable.

You can make another string with just the tags:

>>> import spacy
>>> nlp = spacy.load('en_core_web_sm')
>>> doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
>>> " ".join(token.tag_ for token in doc)
'NNP VBZ VBG IN VBG NNP NN IN $ CD CD'

That example is based on the spaCy documentation.

If the tokens are split by several different non-space delimiters (called "multiple infix tokenization" in SpaCy), you would have to track that and the code would be more complex.


In classification task, is it possible that a truly classified data has a higher loss compared to a miscallisfied data?

Assuming cross-entropy loss (as you specified in the comment), it can happens.
The reason is that the loss in cross-entropy is "activated" only for the probability of the ground truth (t_i is 1 for the ground truth and 0 otherwise).
Imagine a classification task with five classes, with the two following probability situations:
(a) 0.4    0.5    0.04  0.03   0.03

(b) 0.3    0.2    0.2     0.15  0.15
Assume that the true label is the first class.
In situation (a) the predicted class is the second one, but the probability of the first class (which is the only one which matters) is 0.4.
In situation (b) the predicted class is the first one (so the right one), but the probability of the first class is less than the situation (a).
Since cross-entropy cares only of the probability of the true label, in the first case you will obtain a lower loss because the probability is higher, even if the predicted class is the wrong one.
This phenomenon is more probable as the number of classes grows. For few classes, this is highly improbable or even impossible (think to the binary classification case, where this can't happen).
Cheers


1.) You should not change your test set, even if you downsample the training set.
2.) At least in the plots you show there is no clear indication of overfitting. Did you see what happens if you train for longer epochs, and tried different learning rates ?
3.) If you have some overfitting, normally you would simplify the model, and not reduce the size of the dataset. Also you can think about data augmentation to prevent overfitting.
4.) You can compare your solution to the solutions available at kaggle.
5.) With such a black-box solution (vanilla CNN) you will not get super high results.  Though I do not know the upper bound. You can compare to kaggle for that..
6.) To further improve you have to integrate some prior knowledge (e.g. facial key points).


Methods to improve neural network to distinguish red from blue?

I think the other comments about improved feature engineering and sharing more about your problem statement are correct.
I would just add that one helpful tip when thinking through these kinds of problems could be how you as an expert would classify these points by hand. If you would have trouble manually separating the two classes (which the distribution you have shown would lead us to believe), then almost any ML model will have the same trouble.
Without knowing more I'd say that this is potentially related to the concept of Bayes error rate (the amount of intrinsic error which cannot be further reduced with better modeling). If you flip a fair coin, sometimes you will get heads and sometimes you will get tails. There are no better features which will help you decide how to map that action (flipping the coin) to a value (heads/tails), there will always be some intrinsic error because the process itself is random.
In case you are interested: https://en.wikipedia.org/wiki/Bayes_error_rate