It's hard to say without knowing a little more about your dataset, and how separable your dataset is based on your feature vector, but I would probably suggest using extreme random forest over standard random forests because of your relatively small sample set.

Extreme random forests are pretty similar to standard random forests with the one exception that instead of optimizing splits on trees, extreme random forest makes splits at random. Initially this would seem like a negative, but it generally means that you have significantly better generalization and speed, though the AUC on your training set is likely to be a little worse.

Logistic regression is also a pretty solid bet for these kinds of tasks, though with your relatively low dimensionality and small sample size I would be worried about overfitting. You might want to check out using K-Nearest Neighbors since it often performs very will with low dimensionalities, but it doesn't usually handle categorical variables very well.

If I had to pick one without knowing more about the problem I would certainly place my bets on extreme random forest, as it's very likely to give you good generalization on this kind of dataset, and it also handles a mix of numerical and categorical data better than most other methods.


How to proceed 2 executions in 1 step in hive?

You can use HiveCLI Tool to run HiveQL with a given sql file.

  $HIVE_HOME/bin/hive -f /home/my/hive-script.sql

Please see official document: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli

What you need to do is to

Put your HiveQLs in a file as below
SELECT * FROM TABLE1;
SELECT * FROM TABLE2;
Use HiveCLI and run with above file


What does GlobalMaxPooling1D() do to output of LSTM unit in Keras?

No, it takes max across steps.

Suppose the output x of your Bidirectional(LSTM()) has shape  (batch_size, steps, hidden_size), then after GlobalMaxPooling1D() your max_pool would have shape (batch_size, hidden_size).


You could use a collaborative filtering approach, ie. train a network that learns your customer and offer embeddings in a latent space.
You could randomly initialize a customer embedding matrix (no_of_customers, dimensions_c) and an offer embedding matrix (no_of_offers, dimensions_o).

Your training data would be each click, a customer index, offer index, and a binary variable (0,1) which would be used to lookup the embedding vectors. Then you could concat these vectors and use them as input to a fully-connected layer, or any other architecture you wish to use.


Why does balancing the test dataset improve precision-recall curve?

Trying to answer

When dealing with class imbalance ROC is not a good criteria. There is a post on Kaggle about Credit Card Fraud Detection with experiment over the class imbalance, possible ways of mitigating the imbalance, better metrics and code in python for every experiment.

Since it is a really long post (actually it is a notebook on class imbalance and roc), but here I quote author's conclusion while Comparing PR curves to ROC:

  For a PR curve, a good classifer aims for the upper right corner of
  the chart but upper left for the ROC curve.

  While PR and ROC curves use the same data, i.e. the real class labels
  and predicted probability for the class lables, you can see that the
  two charts tell very different stories, with some weights seem to
  perform better in ROC than in the PR curve.

  While the blue, w=1, line performed poorly in both charts, the black,
  w=10000, line performed "well" in the ROC but poorly in the PR curve.
  This is due to the high class imbalance in our data. ROC curve is not
  a good visual illustration for highly imbalanced data, because the
  False Positive Rate ( False Positives / Total Real Negatives ) does
  not drop drastically when the Total Real Negatives is huge.

  Whereas Precision ( True Positives / (True Positives + False
  Positives) ) is highly sensitive to False Positives and is not
  impacted by a large total real negative denominator.

  The biggest difference among the models are at around 0.8 recall rate.
  Seems like a lower weight, i.e. 5 and 10, out performs other weights
  significantly at 0.8 recall. This means that with those specific
  weights, our model can detect frauds fairly well (catching 80% of
  fraud) while not annoying a bunch of customers with false positives
  with an equally high precision of 80%.

  Without further tuning our model, and of course we should do cross
  validation for any real model tuning/validation, it seems like a
  vanilla Logistic Regression is stuck at around 0.8 Precision and
  Recall.

  So how do we know if we should sacrifice our precision for more
  recall, i.e. catching fraud? That is where data science meets your
  core business parameters. If the cost of missing a fraud highly
  outweighs the cost of canceling a bunch of legit customer
  transactions, i.e. false positives, then perhaps we can choose a
  weight that gives us a higher recall rate. Or maybe catching 80% of
  fraud is good enough for your business if you can minimize also
  minimize the "user friction" or credit card disruptions by keeping our
  precision high.

My conclusion:

Even though your PR curve looks bad, given the class imbalance, that is the best metric and you should try to improve so your model is best fitted for your application

Observation:

You have a dataset with 100 samples and only 2 are from $C_1$, gathered right so far?

Supose now that from these 2, one is a complete outlier to its one class $C_1$, he provides more harm than good in your model fitting, and since he accounts for 50% of your $C_1$ samples he is going to make your model perform porly.


What algorithm can predict structured outputs of arbitrary size?

You better try graph embedding and for that I propose going through Horst Bunke from University of Bern who has been doing it for years. Just search him in google scholar and go through his publication network (co-authors,cited papers, citing papers,etc) for example this and this as more classic papers or this as an exact answer to your question. I am pretty confident you will find your solution as it's exactly his research direction. I am on mobile so referencing is difficult now but I can update my answer later.

The other point that I would like to mention is that graph embedding has two kinds:

Embedding the vertices (nodes) of one graph into a n-dimensional space i.e. each vector will be a node of the graph. (see this)
Embedding a dataset of graphs into a n-dimensional space in which each vector will be an entire graph.

You need the second one.

The other thing worth to mention is looking at the problem from Network Science point of view in which the statistical and structural of a graph is widely studied. There are plenty of graph/network measures which are less or more tolerante to the number of nodes e.g. Clustering Coefficient. Using them as features can be fed to a classifier to see how it works.

The last but not the least is what you actually asked! What you are looking for is called Graph Kernel. It is nothing but the definition of the ML-known concept Kernel but on graphs. So kernel methods can get benefit of that by skipping the feature extraction step (like what I suggested above) and directly work with structural inputs. For this purpose I would suggest that you equip yourself with the graph kernels e.g. this paper.

Good luck!


You could create a second label for your usernames according to whether they contain london or not (pseudocode below):
for idx, username in df['Usernames']:
    if 'London' in username:
        df['London'].iloc[idx] = 1
    else:
        df['London'].iloc[idx] = 0

Consequently given you want to go with correlation and that you are comparing binary variables, the metric to go with is Pearson correlation coefficient (pseudocode below):
import scipy.stats.pearsonr as rho
corr = rho(df['labels'], df['London'])


Categorise sentences based on their semantic similarity

There are several techniques that you could apply in order to cluster data if your input is a matrix of pairwise distances between elements. As usual, the best option depends on your specific data, so it is hard to answer to the question of what is the best one, but you could try any of the following ones:

The k-medoids algorithm is similar to the well-known k-means algorithm. After randomly choosing k of your sequences as initial cluster centers (initial medoids) and assigning each sequence to the closest medoid, you randomly reassign sequences to different clusters as long as the value of the cost function decreases.
Hierarchical clustering is another example of clustering algorithm whose input is a matrix of pairwise distances between sequences. In this case the output is a dendrogram.
Another option is to apply multidimensional scaling, a dimensionality reduction technique which input is a matrix of pairwise distances between sequences, to project your sequences into a 2D plane. Once you do that, you can apply any cluster algorithm you can think of, like for instance k-means.

As I said, there are many other options, but these ones are the simplest ones I can think of, and the ones I would start from.


Input for LSTM for financial time series directional prediction

I think there's a partial answer to your question, about the part: "the sequences it's fed are all parts of the same long sequence"

You can train your LSTM model in stateful mode. There's a clear explanation here: http://philipperemy.github.io/keras-stateful-lstm/

In short, in stateful mode, the model will remember what it is previously fed and will also use that information for training. That way, it can sense that the current input is a part of a long sequence.


From these 2 pictures alone, I feel that the algorithm is focusing on the table color. Your second table is black but the basin is silver. Can you add more black table images?


First perform data augmentation or normalization?

In the specific context of image/video problems, it may be okay to normalize the data before augmentation, because you already know that each feature (pixel) will have a value between 0 and 255.  As long as you normalize w.r.t. the theoretical min and max value, then the order of normalization/augmentation shouldn't matter.
But in general, data augmentation should always come first.  Otherwise, the normalized features may be incorrect after augmentation operators are applied.  After all, you can think of augmentation as gathering additional data.  The normalization computed on your original dataset may not be valid after gathering additional data.

To illustrate, suppose we have a dataset consisting of three 2x2 matrices like this:
$$
\begin{bmatrix}
0 & 0 \\
0 & 200
\end{bmatrix}
%
\begin{bmatrix}
50 & 100 \\
200 & 0
\end{bmatrix}
%
\begin{bmatrix}
100 & 200 \\
0 & 0
\end{bmatrix}
$$
And suppose we decide that one good way to augment the data is to flip one or more of the images horizontally.  If we do that for the third image before normalizing.  Then the dataset becomes:
$$
\begin{bmatrix}
0 & 0 \\
0 & 200
\end{bmatrix}
%
\begin{bmatrix}
50 & 100 \\
200 & 0
\end{bmatrix}
%
\begin{bmatrix}
100 & 200 \\
0 & 0
\end{bmatrix}
%
\begin{bmatrix}
200 & 100 \\
0 & 0
\end{bmatrix}
$$
And after normalization, the dataset will be:
$$
\begin{bmatrix}
0.0 & 0.0 \\
0.0 & 1.0
\end{bmatrix}
%
\begin{bmatrix}
0.25 & 0.5 \\
1.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
0.5 & 1.0 \\
0.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
1.0 & 0.5 \\
0.0 & 0.0
\end{bmatrix}
$$
This is the correct encoding of our dataset, and it can be transformed back into the original feature space without errors.

Now what happens if we normalize first, then apply our augmentation operators?
Again we start with our dataset of three matrices:
$$
\begin{bmatrix}
0 & 0 \\
0 & 200
\end{bmatrix}
%
\begin{bmatrix}
50 & 100 \\
200 & 0
\end{bmatrix}
%
\begin{bmatrix}
100 & 200 \\
0 & 0
\end{bmatrix}
$$
This time we normalize first.  Notice that the top-left feature has a range [0, 100] in our dataset, while all other features have a range [0, 200].  So after normalization the dataset becomes:
$$
\begin{bmatrix}
0.0 & 0.0 \\
0.0 & 1.0
\end{bmatrix}
%
\begin{bmatrix}
0.5 & 0.5 \\
1.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
1.0 & 1.0 \\
0.0 & 0.0
\end{bmatrix}
$$
Now let's augment the dataset like we did before, by flipping the last matrix horizontally.  The augmented dataset is:
$$
\begin{bmatrix}
0.0 & 0.0 \\
0.0 & 1.0
\end{bmatrix}
%
\begin{bmatrix}
0.5 & 0.5 \\
1.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
1.0 & 1.0 \\
0.0 & 0.0
\end{bmatrix}
%
\begin{bmatrix}
1.0 & 1.0 \\
0.0 & 0.0
\end{bmatrix}
$$
Oh no! The last two matrices are identical in the normalized space.  But we know that they should represent two different instances.  And if we try to map these matrices back into their original form, we would get incorrect results.  The last two matrices would be identical, even though we know they should be flipped.
Usually, the errors would not be this extreme, but this example show why it is dangerous to normalize before augmentation.  After augmentation, the features may not "line up" correctly with the original normalization.


Encoding categorical variables using likelihood estimation

I was learning this topic too, and these are what I found:

This type of encoding is called likelihood encoding, impact coding or target coding
The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean.
If you have classification task, you calculate the relative frequency of your target with respect to every category value.
From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.
If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.

The quote:

  It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

  Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

  So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

  It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

Another implementation of this encoding is here.
In R library vtreat they have implementation of impact encoding. See this post.
In CatBoost library they have a lot of options for categorical variable encoding including target encoding.
There is no such encoding in sklearn yet.

UPDATE: There is a nice package for sklearn models and pipelines! https://github.com/scikit-learn-contrib/category_encoders


Spark development on local machine with PyCharm

In my opinion, I recommend to use the following approach to develop spark jobs for a big Data Context :

First, develop your spark in local mode on your computer. Use a simple subset of data from your final dataset. This way allows you to work with your favorite IDE (jupiter notebook, PyCharm...) and small computing latences. This is really suitable for spark job development.
Secondly, once your are done with dev in local mode, you can transfer your code on a cluster using SSH clients or whatever. You just have to change your spark context relative instructions and the input/output sources. Now test your code on a small dataset again and correct the few issues that could appear.
Finally, once your code is cluster suitable, you can now use it on your whole Big Data like dataset.

Note : On a Hadoop based cluster, I recommend to store your data on HDFS ORC format. You can then smoothly structure it with Hive and easily read or write these tables with Spark.

Note : Concerning the SSH clients, I recommend you using mobaXterm (a ssh based client for windows) to easily access cluster and drag-and-drop your code. To get some subsets of data for local mode, you can use "Toad for Hadoop" client which gives you the possibility  to run SQL-like queries directly on your hive tables and extract data in CSV or Excel format.

Note : Spark Standalone mode is not suitable for more that few machines. Prefer yarn or mesos clusters approaches for big datasets.


Unable to figure out the linear embedding layer in the convolutional neural network?

Linear embedding layer must be just a fancy name for a dense layer with no activation.
'Linear' means there is no activation (activation is identity). And the embedding is rather a concept for a vector representation of the input data (e.g. word embeddings). I believe the elements from the second vector are simply added to the first one element-wise.


Arithmetic underflow can happen if the result of a calculation is a number smaller than absolute value than the computer can actually represent as a fixed length (fixed precision) binary digit. Instead returning the actual result of the calculation, the computer returns a zero.
Arithmetic underflow can happen in many statistical and machine learning models when many small likelihoods are multiplied together.
Taking the log of each likelihood increases the absolute value so arithmetic underflow is less likely to happen.


Control which features are used for every task in multioutput classification?

I don't think its possible.

First, let's see what's the difference in explicitly modeling separate trees for different tasks versus modeling them in joint manner.

Lets suppose we have 2 task with each n classes. In the later case, to be able to jointly model the correlations, one must create new classes which is a subset of all the permutations available from nC2 combination of the classes from the 2 tasks. Now, if it is true that one feature (say feature A) is beneficial for task 1 but not for task 2, then how would one decide whether to use the feature A while figuring out the final classes in both the task? Task 1 wants the feature but Task 2 doesn't, so this creates a deadlock which would prevent us from putting a bias in the tree model to not use feature A for the classification!

So, if you are certainly sure that a particular feature is not beneficial for a particular task the then the simpler and more effective way would be to model them separately as you mentioned.


How to output my uploaded text in Shiny in R at the position of the screenshot?

In your code you use the Sidebar layout which allows only two columns, instead you should use the grid layout to position your elements where you want them, see https://shiny.rstudio.com/articles/layout-guide.html.
You probably need something like this (not tested):
ui<- shinyUI(fluidPage(
  fluidRow(
    column(3,
      # content left
    ),
    column(5,
      # content centre
    ),
    column(4,
      # content right
    )
  )
)


What is the lightest browser to launch a jupyter notebook?

Try 32-bit Chrome or Firefox, it may significantly lower memory usage.

Remember, it is surprisingly difficult to estimate physical memory consumption of given app: https://dzone.com/articles/windows-process-memory-usage-demystified

Also - unused RAM is wasted RAM. Do not blame browser for taking and using, what's available. When overall memory usage will raise, browser can drop some internal cache and free that memory.


Why do we need to discard one dummy variable?

Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form here "A categorical variable of K categories, or levels, usually enters a regression as a sequence of K-1 dummy variables. This amounts to a linear hypothesis on the level means."

This is already discussed at this very nice stats.stackexchange answer.

I was told there is an advanced course by Yandex in Coursera that covers this subject in more details if you still have doubts, see here. Note you can always audit the course content for free. ;-)

Another nice post if you want a thorough explanation with lots of examples with statistical perspective and not being limited to only dummy coding, see this from UCLA (in R)

Note that if you using pandas.get_dummies, there is a parameter i.e. drop_first so that whether to get k-1 dummies out of k categorical levels by removing the first level. Please note default = False, meaning that the reference is not dropped and k dummies created out of k categorical levels!


Multiple classification algorithms are predicting always exactly with the same scores. Is that normal? If not, what should I suspect?

Same accuracies and losses for multilabel classification usually refer to the fact that model just chooses majority class. Please check the confusion matrix. Definitely, it is not a normal occurrence and you need to deal with it.


How to train ML algorithm with multiple values in target data?

Because your test cases may have values from L3 as input, this strikes me as more of a recommender system; consider encoding each possible value in L3 as a new binary column (thinking of them as products that each row/user has purchased / expressed interest in), and look into different kinds of recommender systems to see what seems most appropriate for your data.

(I'm not too familiar with recommenders.  AIUI, these are all clustering algorithms, but with different notions of distance and focusing on either the products or the users.  sklearn does not appear to have any built-ins specifically for recommendations, but you could use it for an underlying model.  There are also some other Python packages specifically for recommenders, but I can't recommend any [see what I did there?].)


Yes, and that benchmark is called validation data. The idea is to split your data in some training data, which is what you are going to fit to your models in order to estimate the parameters in them (and make them learn), and some validation data, which is the data you are going to use in order to evaluate your model.

What you can do is compute the error on your validation data (which the models haven't been trained on) for the different models you want to use, and you would like to keep the model with the lowest validation error.


R: for each row in DF1 subset/count corresponding rows in DF2

Here is a solution using data.table package.

DF3 <- data.table(merge(DF1, DF2, by = "CATEGORY")) # merge tables by category
DF3$TIME_OK <- (DF3$TIME >= DF3$START & DF3$TIME <= DF3$END) # create a column to check time
DF3 <- DF3[DF3$TIME_OK == T,] # subset table
DF3 <- DF3[,.(NO_OF_OBJECT = .N), by = c("CATEGORY", "DATE", "TIME")] # aggregate results

Basically, it merges both data frames and subsets only relevant time period before aggregating results.


Explanation behind the calculation of accuracy in deep learning model

_, predicted = torch.max(output.data, 1)

On the first line above, it uses torch.max() to find all cases in output.data along the dimension - 1(i.e along each row). In addition, torch.max() returns (values, indices), so only the indices are stored in a variable called predicted while the values are not stored.

total_train += target.nelement()

The second line above just tracks the total size of the train set into a variable called total_train based on the number of elements in target

correct_train += predicted.eq(target.data).sum().item()

The third line above takes the predicted variable, which contains all indices of when the output is greater than 1, checks it against target.data, and sums up all items that are equal. In other words, it is counting how many of the predicted values are equal to the target data.


Yes, its called graph-api.
https://developers.facebook.com/docs/graph-api/reference/v7.0/page/feed
You should be aware that scraping data from facebook not using this api its
violation of the regulations of fb laws and could you be ended with ban. So be careful.


Perform clustering on individual data in categories

As always clustering is about what is the meaning of distance, since that encodes at least some part of your question. So you question is which channels are similar, where similarity is defined on users. Usually you do not assume some nesting on your instances, but what you have is basically a nesting of users in channels. So you have to incorporate this kind of nesting into the distance / similarity function.

I would start with some observation on similarity function. We denote similarity between instance $i$ and $j$ with $d(i,j)$. Usually this function obeys the following condition $d(i,j)\ge0$ for any $i$ and $j$. Note that we usually have equality only on identical data points. The main consequence of this observation is that we can have basically an identity at user level. What you want is an identity at channel level.

One way would be to define the similarity function like:

$$d_c(i,j) = I_{c_i\ne c_j}d(i,j)$$

where $I_{c_i\ne c_j}$ is $1$ when channel of instance $i$ is different that channel of instance $j$. The main effect is that now all the points from the same channel are considered identical since the distance between them is zero. When the identity function is $1$ the distance is given by your real business distance which should be constructed by you and answer your question.

If you use such kind of function in a hierarchical clustering basically it will start to find first some clusters which are identically with your grouping on channels and later on it will join clusters which are similar. This kind of approach will work even with a kmeans algorithm and perhaps with most of the clustering approaches.

A slight different approach would be to define your $I$ function to return $1-\lambda$ when cluster is different and $\lambda$ when channels are equal, and have $\lambda$ a positive value close to $0$. This will not guarantee that all the clients will go into the same cluster, but it gives you the benefit that you have a slider which you can use to fine tune the compromise between all user of the same channel goes into the same cluster and a distance measure which is more robust to outliers.

A totally different approach from an implementation point of view would be to define a more complex function directly on channel samples. This would be similar with how hierarchical clustering works since in order to joins to clusters it should have a distance function which would measure inter clusters similarity. See more on linkage-criteria. For example average linkage clustering. Note that I said that the approach is different only algorithmic. I would bet that the results would be similar with the first approach.

A totally different approach would be to use a more robust criteria. It is known that sample average is not robust since one point can blow away the estimation. Median instead is much stable. You can use a median or a trimmed mean to have a more robust aggregation values. This would have the advantage that the clustering would be much faster since you would work with channels instead of clients and the running time for computing the clustering would be reduced.

And finally, another approach which comes to my mind would be to go further with comparing channels, but this time using a distance which would be based on statistical tests. I will give you a scenario to clarify. Suppose that your users have an attribute named Weight, which as expected would be a continuous variable. How could you define a distance between the Weight of users of one channel and Weight of users of other channel? If you can assume a distribution, like a Gaussian on that weight you can build a two sample t-test on the two samples which are the two clusters. If you can't assume a distribution you can employ a homogeneity / independent test like two sample KS test. KS test is valuable since does not assume any distribution and is sensitive to both changes is shape and location. If you have nominal attributes you can employ a chi-square independence test. Be careful with what you can use from that tests. In order to have equal contribution for each attribute used in distance function you have to you p-values. Also note that if the test is significant it will have a small p-value, since the null hypotheses for both tests is the independence assumption, which can be translated as same marginal for both samples. So, smaller p-values means bigger distance. You can use $1-\text{p_value}$ or even you can try $\frac{1}{\text{p_value}}$.


How to use GridSearchCV with RidgeClassifier

The problem is that Y has shape (15780,1). The shape of Y should show (15780,). The explanation for this is a bit long, so I just give here the links-

Difference between shapes (R,1) and (R,)

Solution for IndexError: too many indices for array


Trouble with Scrapy and Python: ever present unexpected indent

After moments more of work, I was capable of solving my own problem. The error turned out to be a wild 'tab' mark that I had lost in the code there. Also, my 'allowed_domains' section was wonky. This is all corrected in the code below

 from scrapy.contrib.spiders import CrawlSpider, Rule
 from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
 from scrapy.selector import HtmlXPathSelector
 from craigslist_sample.items import CraigslistSampleItem

class MySpider(CrawlSpider):
name = "test"
allowed_domains = ["salem.craigslist.org"]
start_urls = ["http://salem.craigslist.org/search/jjj"]

rules = (
    Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
)

def parse_items(self, response):
    hxs = HtmlXPathSelector(response)
    titles = hxs.xpath('//span[@class="pl"]')
    items = []
    for titles in titles:
       item = CraigslistSampleItem()
       item["title"] = titles.xpath("a/text()").extract()
       item["link"] = titles.xpath("a/@href").extract()
       item["email"] = titles.xpath("A/@anonemail").extract()
       items.append(item)
    return(items)

Unfortunately, I am still not managing to acquire the email, phone number, or description data - however as these problems are not directly related to the question I posed its fair to say that it is answered.


values passed to user-defined distance function by KNeighborsClassifier is wrong

Neighbors-based methods are known as non-generalizing machine learning methods, since they simply “remember” all of its training data (possibly transformed into a fast indexing structure such as a Ball Tree or KD Tree).

(emphasis mine.  Source: https://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors )

You are using algorithm='ball_tree', which roughly speaking uses Euclidean balls to cluster the points, making use of the ball distances for bounds on the distances between actual datapoints (wiki).  So, I suspect the "a" value you're seeing is actually one of the balls' centers.

For the Levenshtein distance, you probably should use the 'brute' algorithm.


In a recommender system, how can you normalise the similarity between two arbitrary users?

This normalization means if the number of mutual voting of $u$ and $v$ is low and not enough proportional to total number of voting (here from $\frac{25}{10000} = \frac{1}{400}$ to $\frac{50}{10000} = 0.005$) you should apply this fact into the similarity in some way. Because, your observation is not enough and you can't talk about it confidently.

In other words, it can be confident factor which you apply into the similarity measure.


The concept of Ensemble Learning is very applicable to your scenario: given two machine learning models that can predict well (or even just better than a random choice), combine the outputs of both models (with another ML model) to produce a unified model (let's call it Model C) that is more predictive than the individual models alone.
A straightforward application to your scenario could involve using the outputs of Models A and B along with some additional features from the original feature set as inputs to Model C. This larger set of features would allow for model C to learn for example when to the signal from Model A is more informative than B based on the current context provided by additional features from the original set.
It may also be possible to do this within a deep learning framework by adding both models into the same graph and adding a layer that ensembles the signals from the two models for an output prediction.


Is training one epoch using mini-batch gradient descent slower than using batch gradient descent?

You are correct, there is more overhead to process the same amount of data because you do more weight updates and maybe preprocessing the batches in your generator will take more time as well. However, since you are doing more updates and if your batch size is big enough, the gradients of your mini-batches will approximate the gradients of your full batch fairly well and you will have gotten much closer to the optimum of your full dataset by splitting your full batch into mini batches than you would have gotten by doing one non-stochastic weight update.


50% accuracy on multiclass classification

Assuming it really is a one form many class problem and not a multi-label problem.

An accuracy of .5 would mean that half of the instances were classified correctly. That would also mean that the model is able to generate the correct class half of the time. For your test data, results can still vary for new data.

If that is a good score depends on more than the number of classes. Quite often one class is represented more than others. The ZeroR-score is a good baseline for that.

Suppose you have 10 classes and Class A makes up 30% of the population. A random proces would achieve an accuracy of .1, a proces that simple says "Class A" all the time, would post a .3 accuracy.


Prove data points not enough? Estimate required number of data points?

It depends on what you mean by "needed". More training data = lower variance with the same model complexity. So when you have a small training dataset, your models will have to be very simple (like linear regression), so to have best generalization. And very simple models tend to badly represent reality.

The answer is: you need so many points, such that your validation error is below desired threshold. 30 is probably much too low for any more complex machine learning model than linear regression with one, maybe two features.


What's the default Scorer in Sci-kit learn's GridSearchCV?

From the User Guide:

By default, parameter search uses the score function of the estimator to evaluate a parameter setting. These are the sklearn.metrics.accuracy_score for classification and sklearn.metrics.r2_score for regression...


Train classifier on synthetic images to recognise real images

Neural networks require a lot of data for training. This is one of their largest drawbacks, and is very hard to get around this.

Using artificial data is not a solution. We don't really understand what the network is learning and we can't tell if it learned anything that is particular to the synthetic images.

What you can do is to use some sort of data augmentation. Rotate, scale, translate your images many times, and create a larger dataset.

Since it's such a small dataset, and if more data is really not possible to obtain, you should consider other approaches to help the classification. Either manually design some features to help a bit the classifier or do the classification with more traditional methods.


TL;DR
While the data comes from the NYTimes and seems legit, the presentation is intentionally misleading and the subsequent assertions are baseless. I say "intentionally" because an unbiased and reputable analysis would not propagate such major allegations from the data they have presented. The data does not prove nor disprove voter fraud, so the unfounded assertions are simply disinformation.

Before diving into the data
We are going to see many more such threads/posts in the coming months and years, so I need to address this bit before delving into the question.
Consider the following whenever you view anything online:

Fake news is much more viral than true news! Always air on the side of caution.
Is the source credible? In this case, you have no reason to believe the random twitter user "CulturalHusbandry @Aphilosophae".
Do other credible sources corroborate the findings? Your confidence in the findings should increase with every independent credible source that come to the same conclusions. In this case, we do not have a credible source to begin with.
Is the language charged and the rhetoric conspiratorial? This is unfortunately becoming a prominent red flag for disinformation campaigns that are developed to sow doubt and distrust in democratic institutions.

How this Twitter thread is disinformation (step-by-step)
1. False premise
He/she first establishes the fact that, in total, batches of votes do not conspicuously favor one candidate or another (they do so via the tweets below). The distribution of the D/R ratio in vote batches over time is not reflected in this aggregate plot. However, this could mislead viewers to believe that they should expect 50/50 Democrat/Republican consistency throughout the vote counting process, which cannot be assumed from this information. The "anonymous data scientist" seems to encourage this logical fallacy by using this as his/her premise and posting his/her follow-up tweet to the plot (image below).

2. Misleading visualizations
He/she then establishes that the initial counts are "random" and noisy as one might expect from a 50/50 split. However, these batches do not have associated sizes with them! A "batch" may be 100K votes or it could fewer than 10. An actually informative portrayal of the data would have made the size of the circle/point reflect the size of the batch (bubble chart) rather than a scatter plot. Although this plot may technically be accurate, this can still be quite misleading, particularly because the "anonymous data scientist" made no note of this deficiency (lack of batch size information). But for the sake of argument, let's go with it.

3. Unsubstantiated claims and ignoring legitimate possibilities
Now they start talking about an "anomaly" and introducing conspiratorial allegations. They note the jump toward D, given the D/R ratio per batch in Milwaukee at some point in time. There are many legitimate explanations for this.
For example, a high/dense population such as a city (likely Democrat-leaning) would of course take more time to collect/count ballots, so they may not have started their reporting until after low/sparse populations (likely Republican-leaning). This can cause a sudden jump in favor of Democrats.
Also, the majority of the early ballots are in favor of Republican. This is to be expected; of course the candidate who consistently attempted to delegitimize mail-in ballots would receive fewer mail-in ballots from his followers. By contrast, Democrats were encouraged to vote by mail (and they did), making the later mail-in vote counts lean Democrat. This is another factor that could explain why the initial vote counts are skewed toward Republican in most states when compared to the final counts.


How to create a model to suggest similar words in realtime?

Autocompletion algorithms usually build on either tree or hash structure for the sake of efficiency. One famous approach is the Ternary Search Tree. Use this source to have an initial idea. It is also a comparably space-efficient solution. You can find more complex systems build on the Ternary Search Tree.
Another good solution is Trie. Trie data structure is a good alternative of Ternary Search Trees, as an example implementation check this.
These both are handy at the front end of the application since they are simple but powerful approaches. Other than that, they have a good balance of time and space complexity.
Moreover, a fast-autocomplete library may help you. Since, in your case, you may have two names for the same job, it has synonym functionality.
In general, tree structures are what you are looking for. You can do this task even using Binary Trees. However, I would suggest you consider the case that in reality many times there appears spelling problems, so in that case, you either will not be able to suggest a word or the suggestion will be incorrect. For such problems, you can also have a spelling check algorithm that first checks if such a word exists, if not using spell check you can have proposed words and you can suggest the autocompletion based on those suggestions. E.g. instead of 'Java' one may type in 'Jsva' and your algorithm will find the correct word and make the suggestion accordingly. This is a one-of-the best algorithms that does auto-completion using ML techniques. You can also build a simple one using Levenshtein distance. (However, spelling error cases mostly discarded in the systems because of its being costly to fix them)


Convolution Neural Network (CNN) is the state of art algorithm for your problem. The standard terminology for this kind of problem is "feature detection".

Here is a good resource for the basics. http://cs231n.stanford.edu/

Your biggest problem probably is going to be the size of your training images. 1,000 is most likely not going to be enough. You should try to augment your data by rotating and translation etc.


Your problem originates in the fact that your testing set consists of examples of just 6 possible classes, while your model is trained with a training set of 9 different labels.

When converting the test set to one-hot-encoding, you need to add the number of possible labels as the 'n_values' parameter. Otherwise by default it relies on the number of classes in the test set (which is 6 in you case).


Which classifier performs better when using 'class_weight'?

Very good question! I think the plain answer is: no. To my experience this usually depends on the data. Even different boosting methods/tools will perform differently. It often depends on try and error.

What can make a difference is using L1/L2 regularization, since it can shrink features which are „not useful“ to make a good prediction.

I cannot think of a theoretical reason here. I‘m curious if someone comes up with a more theory based answer!


Positive or negative impact of features in prediction with Random Forest

With decision trees you cannot directly get the positive or negative effects of each variable as you would with say a linear regression through the coefficients. Its just not the way decision trees work. As you point out, the training process involves finding optimal features and splits at each node by looking at the gini index or the mutual information with the target variable. But no parameters are learnt during the process which we could use for such analysis.

A common tool that is used for this purpose is SHAP. In fact, there is a specific explainer for decision trees based models which is the TreeExplainer. With SHAP you can get both the contribution of the features in pushing towards one or another value of the label, and also an overall view of the contribution of all features.


Based on the paper you set k but want to sample greater than k and apply the log calculation.

I would recommend using the foreach function. Where you could say foreach() apply further reach than k.

where the following aglo from the paper could be interpreted as follows.

Algorithm 2 k-means||(k, ) initialization.

1: C ← sample a point uniformly at random from X
2: ψ ← φX(C)
3: for O(log ψ) times do
4: C 0 ← sample each point x ∈ X independently with probability px =·d2(x,C) φX(C)
5: C ← C ∪ C0
6: end for
7: For x ∈ C, set wx to be the number of points in X closer to x than any other point inC
8: Recluster the weighted points in C into k clusters

instructions:

get points of k = 2
points [(1,1),(1,2),(2,2)]

centroid is placed in middle for this example and

c1 = [(1,1),(1,2)]

This was achieved with the over sample of Euclid foreach() point that satisfies the over sampling requirement.

see example here which uses for each:

# Cluster the data into two classes using PowerIterationClustering
model = PowerIterationClustering.train(similarities, 2, 10)

model.assignments().foreach(lambda x: print(str(x.id) + " -> " + str(x.cluster)))

So you will need to write the distance into the lambda.(if you provide code easier to help you).

K distance = k distance on perimeter which is the bottom red line in diagram

 .foreach(lambda x: kdistance[get average] + then check prob(k prime) of k)

share some code and sample data please


Reduction of multiple answers to single variable

The variable represents the answer to the first question.

One straightforward way is to allow for all possible categories in this variable. For example, if there are 5 options in this answer, you will have to treat it as a categorical variable with 2^5 = 32 categories.

However, the number of categories increase exponentially with the number of options (check boxes) provided for the answer. In that case, it might be better to restrict the number of categories to, for example, 5. This can be done by leaving the top 4 choices/ options (by count) as they are and treating every other choice as "other".


XLNET how to deal with text with more than 512 tokens?

BERT also has the same limit of 512 tokens.
Normally, for longer sequences, you just truncate to 512 tokens.
The limit is derived from the positional embeddings in the Transformer architecture, for which a maximum length needs to be imposed. The magnitude of such a size is related to the amount of memory needed to handle texts: attention layers scale quadratically with the sequence length, which poses a problem with long texts.
There are new architectures that specifically deal with longer sequences, like Longformer, Reformer, Big Bird and Linformer.


Classification engine with multiple Textual independent fields

If I combine all the fields value in my second Example and then form a single vector from that and then pass those vectors for classification, would that be the ideal way of doing it?

In short, yes. You can convert each input variable into a vectorised feature separately, then concatenate those vectors into one long vector of features per example. Most statistical machine learning models don't care (or "understand") that the feature data has been formed in different ways, it is just treated numerically.

You do need to pay attention to each input data type in order to decide how best to convert to a usable feature. There is no single "best" way to do this, and this is partly an art - you add the "science" part of Data Science by testing your ideas and taking measurements, e.g. accuracy.

These are some quick thoughts on how you might prepare the data into features from each column:

Comments. You have already converted this, you are using a variant of "bag of words" to convert text into a fixed length numerical vector. This is a common approach and hard to beat. If you have a lot of data you can look into more sophisticated models that take account of word order, such as Recurrent Neural Networks, but that's a whole new subject area for now.
Gender. You can convert this into a simple vector, using one hot encoding - this will have two or more columns.
City. You could one hot encode this also. It might also be worth grouping cities and having a smaller feature vector. E.g. group by state/country before one hot encoding.
Age. Either group and one hot encode, or scale to a smaller number,
by dividing by e.g. 50 (this helps algorithms like SGD by keeping distance metrics similar between different feature types). I suspect grouping in typical demographic splits (e.g. 0-11, 12-17, 18-24, 25-34 etc) and one hot encoding would work well for sentiment analysis, because that would capture some generalised differences between uses of text expression.


Is sequential pattern mining possible with machine learning?

Seq2Pat: Sequence-to-Pattern Generation Library might be relevant to your case.
The library is written in Cython to take advantage of a fast C++ backend with a high-level Python interface. It supports constraint-based frequent sequential pattern mining.
Here is an example that shows how to mine a sequence database while respecting an average constraint for the prices of the patterns found.
# Example to show how to find frequent sequential patterns
# from a given sequence database subject to constraints
from sequential.seq2pat import Seq2Pat, Attribute

# Seq2Pat over 3 sequences
seq2pat = Seq2Pat(sequences=[["A", "A", "B", "A", "D"],
                             ["C", "B", "A"],
                             ["C", "A", "C", "D"]])

# Price attribute corresponding to each item
price = Attribute(values=[[5, 5, 3, 8, 2],
                          [1, 3, 3],
                          [4, 5, 2, 1]])

# Average price constraint
seq2pat.add_constraint(3 <= price.average() <= 4)

# Patterns that occur at least twice (A-D)
patterns = seq2pat.get_patterns(min_frequency=2)

Notice that sequences can be of different lengths, and you can add/drop other Attributes and Constraints. The sequences can be any string, as in the example, or integers.
The underlying algorithm uses Multi-valued Decision Diagrams, and in particular, the state-of-the-art algorithm from AAAI 2019.
Another source that might be relevant is SPMF
Hope this helps!
Disclaimer: I am a member of the research collaboration between Fidelity & CMU on the Seq2Pat Library.


Fitting probabilities in scikit-learn RandomForestClassifier

Well, based on how this algorithm works, here is why :
You want to build 200 trees, with a max depth of 5 levels. Each tree is a decision tree, train with a subset of your data. To answer your question, it doesn't fit to the value 2 because that's not how it works. What's happening is that 52 trees predicted the value 0, 10 predicted the value 1, and 138 predicted the value 2. So most of the trees predicted the value 2 You asked for the probability (using predict_proba(X_test)). If you want only one value, don't use predict_proba but predict(X_test). But it will only be the reflect of the probabilities you got.


Have you tried regression here is a nice project to get you started could you give some feedback on the output?


The renown algorithm for stemming is Porter stemming algorithm. Hence, you can use stemmer = nltk.stemmer.PorterStemmer() for the stemming. you can test it using stemmer.stem('poorly').

Moreover, you can see this post for more details.


Fitting glm without explicit declaration of each covariate

This has already been answered in stackoverflow here and here.

The main idea is to create the formula from a string with as.formula:

xnam <- paste("x", 1:25, sep="")
fmla <- as.formula(paste("y ~ ", paste(xnam, collapse= "+")))


Finding best cluster after running K-Means clustering

Generally, library that you use for k-means clustering will report "within cluster sum-of-squares" value after running k-means procedure. A smaller value means the cluster is tightly bound (most messages are similar to each other). You can use this value to order the cluster IDs.


Explanation of why Neural Networks are non convex

From the CV thread Cost function of neural network is non-convex?:

How does this relate to our neural network? A cost function $J(W,b)$
has also a number of local maxima and minima, as you can see in this
picture,
for example.
The fact that $J$ has multiple minima can also be interpreted in a
nice way. In each layer, you use multiple nodes which are assigned
different parameters to make the cost function small. Except for the
values of the parameters, these nodes are the same. So you could
exchange the parameters of the first node in one layer with those of
the second node in the same layer, and accounting for this change in
the subsequent layers. You'd end up with a different set of
parameters, but the value of the cost function can't be distinguished
by (basically you just moved a node, to another place, but kept all
the inputs/outputs the same).

From NonConvexity of neural networks:

I don’t remember where I heard this arguement but it was a
pretty interesting one and I couldn’t remember the entire
reasoning and so as I began to think about it, I figured some of it
out and I believe it is pretty reasonable. We have all argued that
neural networks are difficult to optimize  because they are
non-convex(or non-concave) for that matter. What we don’t talk
about so much is why is it that they are non-convex. It probably has
simpler answers where you will argue something about the
non-linearities involved. This post has to do with a much more
interesting answer.
To understand the reasoning, we have to argue that if the function has
atleast two local minimas such that their mid-point is not a local
minima, the function is non-convex. In formal terms, $$\text{If
> }\exists x,y \in Dom(f) \hspace{5pt }s.t.\hspace{5pt} g(x) = g(y) = 0
> $$ and $$\forall z\in [x,y] \text{ s.t. } g(z) \neq 0  $$ where $g(x)$
is a subgradient at $x$ the function is non-convex.
This is pretty simple to follow from the Mid-point value theorem and
the definition of convexity.
Consider a neural network  $F(\cdot)$ with a few layers and name three
layers $L_1, L_2$ and $ L_3$ Consider nodes $a$ and $b$ in $L_1$ and
$c$ and $d$ in $L_2$.Let the parameters connecting node $i$ to $j$ be
called $w_{ij}$.
Thee arguement to the function can be thought of as $A=[\cdots w_{ac},
> w_{ad}, \cdots , w_{bc}, w_{bd}\cdots]$ and the function evaluation as
$f(A)$. Consider $B=[\cdots w_{ad}, w_{ac},\cdots, w_{bd}, w_{bc},
> \cdots]$. I can claim that there exists a $B$ i.e. where $ w_{ad},
> w_{ac}$ and $ w_{bd},  w_{bc} $  are swapped respectively, there
exists an ordering of the other weights(permute the other edges) such
that the function evaluation remains same. It is pretty easy to see
how it can be true.
Look at the following picture where the reg edges come into one node
and the green into another.

For reference consider

The first red edge coming out of the top left node as $w_{ac}$
The gree edge out of the same node as $w_{ad}$.
The red edge out of the second node in the same layer as  $w_{bc}$
The green edge out of the second node as   $ w_{bd} $.

Connect all edges coming in to $L_2$ from $L_1$ into node $d$ to node
$c$ and vice versa and then shift the origin of  all outgoing edges
from node $c$ to $L_3$ to node $d$ and vice versa. What you have
acheived is the network that you would have achieved by holding the
two nodes by your hand and manually exchanging their positions while
keeping the connecting edges intact. And Voila! now you have $A, B \in
> dom(F)$ where $F(A) = F(B) $.
Now look at them after the edges are shifted.

[..]What this effectively means is that if the neural network was
indeed convex, there would be a value which could be given to all
nodes in a layer and yet the network would have represented a local
optima. This is higly absurd and is very apparent if you consider a
classification network where the final outputs should not have the
same value in all nodes given some value in the input layer(which is
also a part of the function arguement).
To generalize this to any neural network, you will have to look at the
activation functions and their properties. The claim you will be
trying to verify is that the average of two permutations of an optimal
point need not be optimal.
I guess this was an interesting read. A very interesting thing this
says is that
A neural network cannot have a unique optima if it has atleast two
distinct node values in the same layer. Amazing , isn’t it ?


Extracting data from a data frame with conditions

There is a good summary on StackOverflow from the user EDi that I'm referring to. Basically, what you want to do is group the values by year and then select the maximum frequency of each group:

# with dplyr
require(dplyr)
df %>% group_by(year) %>% summarise(freq = max(freq))

# data.table
require(data.table)
dt <- data.table(df)
dt[ , max(freq), by = year]

Check out the linked answer for further possibilities.


In k-fold-cross-validation, why do we compute the mean of the metric of each fold

It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).

But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.

(this is my interpretation, there might be other reasons)


Data augmentation solutions for tabular/structured data

It's hard in general, because you don't know the joint distribution of the data, let alone joint distribution with the label -- or else you wouldn't need a classifier. Without that you can't confidently sample new, valid instances.

In the image case it's 'easy' because we know certainly that a rotation or shear or scale of an image produces another valid image, and we know its label is the same.

If you're willing to make the assumption that points in your input space between two instances with the same label are also a) valid inputs and b) have the same label, then you can apply things like SMOTE (https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html) to generate more data according to that assumption. If that assumption is wrong, it'll hurt performance.


I think your best bet is a gazetteer approach. If you go to this link, you will find plenty of datasets containing lists of existing first names. This should help you detect a big majority of first names using regular expressions.

Now, there are some preprocessing steps you could take:

remove any digits, punctuation, etc.
if you have upper cases, you could split the username such that "MichaelScott" becomes "Michael" and "Scott". Now, in this case, you end up with two things that could be the first name. You could use the assumption that the first one is the first name and the second one is the surname.

Once you've done this, look at usernames where no first names were detected. Now, it could very well be that the username doesn't contain any first names (i.e. user123213543), or the user is bit cheeky and could use special characters or digits to write their first name (i.e. $am, B0b, etc.). If these situations exist, you could create strategies to deal with them.


Conditional Entropy and Mutual Information - Clustering evaluation

Mutual information does favor many small clusters. Nectar these tend to be "pure". That is why variations wish as normalized mutual information and adjusted mutual information (AMI) are used instead.

  Vinh, N. X.; Epps, J.; Bailey, J. (2009). "Information theoretic measures for clusterings comparison". Proceedings of the 26th Annual International Conference on Machine Learning - ICML '09. p. 1. doi:10.1145/1553374.1553511.


Using a feature in prediction even if it gets zero as p-value?

For the correlation problem, this basically sums up why I dislike univariate feature selection. Just because a feature has low correlation with your response in isolation, does not mean that the variable itself is not informative. The variable could relate in a non linear way (so transforming it could be useful), it could work well with other variables in a multivariate setting, and also could be used to generate other variables through feature engineering.

With respect to a "low p value", note that a lower p value suggests that the variable is significant, that is, the effect of the variable on the response is statistically different from 0. So if you were to use p values as a means of feature selection, you would select variables with smaller values. However, this is not at all a good way to select variables if your goal is to improve predictive performance. P values are meant to test very specific statistical hypotheses related to statistical inference, not prediction. Removing variables with high p values will likely lead to a loss in predictive performance because p values don't even answer the right question in this context. Also, realize that by conducting multiple hypothesis (looking at individual p values for coefficients) you further increase your chance of a false positive beyond what you set your significance level at (i.e. the cutoff that you use to deem a variable to be significant).

If you want to do feature selection within the context of a linear model, look into the LASSO/Elastic Net. These models will shrink the coefficients of variables to zero if they do not improve the fit of the model. Otherwise, if you only have a handful of variables then go with model 2 since it is the highest scoring model (on this specific test set anyway). Possibly repeat model evaluation on different partitions of the dataset to see if model 2 is consistently better than model 1. This would allow you form confidence intervals which could be useful in deternining if the extra column gives a significant performance increase over a simpler model.

Finally, I recommend not using AUC or accuracy for comparing the performance of models. These metrics are not proper scoring rules and may lead you to selecting models that don't discriminate between your classes as well as others.


How to encode a class with 24,000 categories?

Yes, using one-hot encoding on 24k features requires 24k input nodes. However this should not be a problem for Keras (or any other deep learning library). Natural language processing often uses one-hot encoding on words with a vocabulary size in the same ballpark.

If you are using a "deep" model, one of your hidden layers should take care of reducing the dimensionality of your data. A separate pre-processing step is usually not needed.

The training time should not be unreasonable.


What to infer from a given scatter plot?

It is not easy to say based on this graph. Maybe if you could apply a transformation, it was easier to see a trend. I would have at least plotted them on log-log, log-linear, etc. to get better idea.


How to build classifier if only 1 image per class available

One image per class in way too less data for building any classifier. Rather than image classification, image similarity matching seems a better approach and so I would suggest you a Siamese network for one-shot learning.

A Siamese network is a class of neural network that contains two or more identical network and are used to differentiate between the input images by maximizing the minimum distance among the images. This technique is helpful for what we call as One-shot learning.

But if you only want to build a classifier, here are a few steps you can think about:

Try to increase your dataset size by adding a few more images to you training set.
Apply heavy Image-augmentation.
Use a Pre-trained model to try transfer learning.


Reporting test result for cross-validation with Neural Network

Now when I am reporting the test result, I set the epoch number for training on my training data a fixed number of times, and when my network is doing the best on the test set, I stopped the training, saved that model for future use, and report that result. My question is about this last part. Am I doing something wrong on reporting this result?

Technically yes this is incorrect process for reporting an unbiased test metric. This could be a bad over-estimation of performance if cv results are noisy and vary randomly epoch-to-epoch. You should in theory treat the early stopping epoch number same as your other hyper-parameters, discover a good value from the lower-level cross-validation and stick with it.

There is a problem with my suggestion though - the early stopping epoch number is sensitive to training data set size, and you just increased the size when you changed from lower level cross-validation to the higher level one. So you might get an unbiased measure at the expense of significantly worse results.

First, take a look at your learning curves. Just how sensitive are the cv results to epoch number? If they are not sensitive - no obvious over-fit over a reasonable range of epoch numbers - then just pick a mid-range fixed number of epochs that applies to all folds. That way you will have your unbiased estimate, and probably not compromised on model quality.

Alternatively, you may just have to be happy knowing that your test estimate is biased, but you have reduced the variance significantly by using 10-fold CV, and have only searched one hyper-parameter at the top level (not all three). It may only be slightly biased, and still a good estimate. The smoother and less jittery the learning curves are, the more you can get away with this - but sadly you won't get a measure of the bias, so you'll never be 100% sure.


How word2vec can be used to identify unseen words and relate them to already trained data

Every algorithm that deals with text data has a vocabulary. In the case of word2vec, the vocabulary is comprised of all words in the input corpus, or at least those above the minimum-frequency threshold.

Algorithms tend to ignore words that are outside their vocabulary. However there are ways to reframe your problem such that there are essentially no Out-Of-Vocabulary words.

Remember that words are simply "tokens" in word2vec. They could be ngrams or they could be letters. One way to define your vocabulary is to say that every word that occurs at least X times is in your vocabulary. Then the most common "syllables" (ngrams of letters) are added to your vocabulary. Then you add individual letters to your vocabulary.

In this way you can define any word as either

A word in your vocabulary
A set of syllables in your vocabulary
A combined set of letters and syllables in your vocabulary


Keras exception: ValueError: Error when checking input: expected conv2d_1_input to have shape (150, 150, 3) but got array with shape (256, 256, 3)

[EDIT:]
Your problem is definitely in the generators, in that you do not set the target size, and its default it (256, 256) - as seen in the documentation fro flow_from_directory:

flow_from_directory(directory, target_size=(256, 256), color_mode='rgb', ...)

  target_size: Tuple of integers (height, width), default: (256, 256). The dimensions to which all images found will be resized.

Try setting the target size parameter to (150, 150) and I think it will work. That default seems to be overwriting your preprocessing.

It must be in your generators - I ran the following code and a model trained as expected:

from keras import models, layers, metrics
import numpy as np

model = models.Sequential()
   ...: model.add(layers.Conv2D(32, (5, 5), activation='relu', input_shape=(150, 150,
   ...:  3)))
   ...: model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
   ...: model.add(layers.Conv2D(64, (3, 3), activation='relu'))
   ...: model.add(layers.MaxPooling2D((2, 2)))
   ...: model.add(layers.Conv2D(64, (3, 3), activation='relu'))
   ...: model.add(layers.MaxPooling2D((2, 2)))
   ...: model.add(layers.Conv2D(128, (3, 3), activation='relu'))
   ...: model.add(layers.MaxPooling2D((2, 2)))
   ...: model.add(layers.Flatten())
   ...: model.add(layers.Dense(300, activation='relu'))
   ...: model.add(layers.Dense(10, activation='softmax'))

In [10]: model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 146, 146, 32)      2432
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 73, 73, 32)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 71, 71, 64)        18496
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 35, 35, 64)        0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 33, 33, 64)        36928
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 16, 16, 64)        0
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 14, 14, 128)       73856
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 7, 7, 128)         0
_________________________________________________________________
flatten_1 (Flatten)          (None, 6272)              0
_________________________________________________________________
dense_1 (Dense)              (None, 300)               1881900
_________________________________________________________________
dense_2 (Dense)              (None, 10)                3010
=================================================================
Total params: 2,016,622
Trainable params: 2,016,622
Non-trainable params: 0
_________________________________________________________________
In [17]: model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics = ['
    ...: acc',metrics.categorical_accuracy])

# Create some fake data to match your inputs. Each label seems to be 10 points: (1, 10)
In [11]: fakes = np.random.randint(0, 255, (100, 150, 150, 3))
In [24]: labels = np.random.randint(0, 2, (100, 10))

In [25]: model.fit(fakes, labels, validation_split=0.2)

Epoch 1/10
80/80 [==============================] - 2s - loss: 62.8913 - acc: 0.1125 - categorical_accuracy: 0.1125 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 2/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 3/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 4/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 5/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 6/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 7/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 8/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 9/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 10/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00


How should continuous outputs be in convolutional neural network?

Convolutional networks can be used for regression tasks too. The difference corresponds to the output layers of the dense networks. In classification tasks you use sigmoid or softmax depending on your task. In regression tasks you can simply use linear activation function as the non-linearity of the last layers. Consequently, if you have the ages as the outputs of your network and if they are in a same scale as the inputs you can simply use them as they are right now.


Constructing a Maximum Entropy Classifier for Sentence Extraction

Conjugate gradient descent is a variation on gradient descent. Gradient descent is a method of finding the minimum of a function by taking steps that reduce the error between the data and the model parameters. Conjugate gradient descent extends gradient descent by searching a plane, instead of a line. The plane is defined as a linear combination of the gradient vector and the previous descent step vector. Conjugate gradient descent is very good at finding the solutions to a set of sparse linear equations.

Gradient descent and variations are general methods for finding the best parameters. The best practice is to define your specific model and then call on a separate gradient descent package to search for optimal values (i.e., hyperparameters) for the model.


Text classification with multiple documents per labeled datapoint

Why don't you make a person id and add this to your model?

If I understand you correctly, you do:

$$y=\beta X$$,

where each row in $X$ are combined docs per person and $y$ is a vector of true/false, right?

You could try:

$$ y= \beta X + \gamma z$$,

where each row in $X$ is only one doc now and $z$ is a vector of ids per person (so a factor).

Might be worth a try.


How to handle string features in a neural network

Does the string add any useful data? No. You should replace categories by id's for sure, and manufacturers as well. Why would having uncorrelated groups of strings stop you from giving each an id?

So Google = 0, Apple = 1, etc. But that doesn't mean that a category can't have id 0 or 1. This is a different input, so these should start at 0 too: Books = 0, Electronics = 1. You still have to normalize these id's for your network ofcourse (value = id / maxId).

This might be useful: https://github.com/cazala/synaptic/wiki/Normalization-101


Glasses matter. So you will need those photos in the training dataset.
Yes. You will need to feed images of all ages for multiple people. This will help the model generalize how people age
Alignment should not matter
Background should not matter


My understanding of the question is that you are using a basic MLP, feed-forward neural network, to work with data that is a 3-dimensional vector which you are calling $(x,y,z)$ - although note that most notation you read would call the whole observation $x$ which would be a vector $(x_1, x_2, x_3)$. You want to accept an input of 1st, 2nd, 3rd observations of this data (a window on some possibly longer series), and predict the 4th observation.

For the given architecture, the prediction is made immediately on the supplied input - a MLP has no "memory" for previous inputs - so you will need 3 x 3 = 9 inputs to represent each (x,y,z) in the series. For a MLP it doesn't really matter how you arrange this input - e.g. you could have $(x^{(1)}, y^{(1)}, z^{(1)},x^{(2)}, y^{(2)}, z^{(2)},x^{(3)}, y^{(3)}, z^{(3)})$ or $(x^{(1)}, x^{(2)}, x^{(3)},y^{(1)}, y^{(2)}, y^{(3)},z^{(1)}, z^{(2)}, z^{(3)})$ or any other arrangement as long as you are consistent for every training example and prediction.

You don't usually want or need to somehow combine the x, y and z values to create a pre-processed layer with 3 inputs, or create a special layer that combines $x^{(1)}, x^{(2)}, x^{(3)}$. You might do that if your understanding of how the sequence worked allowed you to come up with features that modelled something important. However, the more usual approach is to allow the neural network to figure out how the multiple inputs combine from the training data.

You may want to look at alternative architecture of Recurrent Neural Network (RNN), which has more options for dealing with predictions of series. For a RNN, you would have 3 input neurons, and you would run the network 3 times in sequence and read the prediction after running it the third time.

You have noticed correctly that a MLP has no "understanding" internally that your x in (x,y,z) at step 1 is the same type of variable as the x in steps 2 and 3. The MLP would have 9 separate inputs and would ignore any direct relations, or even that there is a sequence being predicted. Whilst in a RNN, both those details are built into the model as core assumptions. In some cases that could be an important advantage to the RNN. But it is still worth trying the simpler MLP first and testing whether its predictions are good enough for purpose.


There are many approaches to evaluate the uncertainty of your regression algorithms.
There are model dependent approaches as which the bayesian approach allows you to have an predictive distribution instead of an output which allows you to evaluate your confidence interval. You can apply this approach on linear models as well as on DNN.
There are also model free approach as bootstrapping or quantile regression that allows you to evaluate your uncertainty while considering the prediction algorithm as a black-box model


Adding a new custom column in a python data frame

This turns out to be fairly simple! There is a handy method called repeat on a datetime index. Here are the steps:

import pandas as pd

Define a date range, supplying start and end

jan = pd.date_range(start="1-Jan-2018", end="31-Jan-2018")    # could specify any year

Now provide how many times you want to repeat each date and create the repeated column

num_repeats = 24
repeated_jans = jan.repeat(num_repeats)

Let's create the random dummy dataframe as a base

total_dates = num_repeats * len(jan)    # 24 x 31 = 744
df = pd.DataFrame(np.random.randint(0, 10, total_dates))

This is how we add a column - the name of the column can be anything

df['repeated_jans_lalala'] = repeated_jans

Have a look at some dates:

print(df.iloc[[0, 24, 48, 71, 72]])    # multiples of 24...we can see one repeated date

    0 repeated_jans_lalala
0   7    2018-01-01
24  4    2018-01-02
48  3    2018-01-03
71  6    2018-01-03
72  3    2018-01-04

If try adding the column to a dataframe with a different number of rows, we get an error:

df_error = pd.DataFrame(np.random.randint(0, 10, 743))    # require 744!
#df_error['repeated_jans'] = repeated_jans                         # raises ValueError

If you want to change the way the dates look, you can use the strftime method on the dates

jans_fancy = jans.strftime('%d-%B-%y')
df['fancy_jans'] = jans_fancy

df.head()

0   repeated_jans_lalal  fancy_jans
0   7   2018-01-01       01-January-18
1   2   2018-01-01       01-January-18
2   8   2018-01-01       01-January-18
3   9   2018-01-01       01-January-18
4   7   2018-01-01       01-January-18

If you don't want to show the actual year, just leave out the %y part!


How do Data Scientists integrate Predictive Modeling with SQL?

There is no need to download data in CSV format, in most cases it is actually bad practice. Consider cases where data size > 1GB and is updated daily. This would add a considerable overhead and is not easy to automate.

Instead you can take a look at various R packages that exist to fetch data from your SQL database. Using RODBC, SQLite, sqldf or other available libraries you can import data to R, run queries, create tables, update tables and pretty much anything else you will need.


Is it always better to using stacked LSTM than single LSTM?

From https://machinelearningmastery.com/stacked-long-short-term-memory-networks/:

"Stacking LSTM hidden layers makes the model deeper, more accurately earning the description as a deep learning technique ... The additional hidden layers are understood to recombine the learned representation from prior layers and create new representations at high levels of abstraction. For example, from lines to shapes to objects ... A sufficiently large single hidden layer Multilayer Perceptron can be used to approximate most functions. Increasing the depth of the network provides an alternate solution that requires fewer neurons and trains faster. Ultimately, adding depth it is a type of representational optimization."

Increasing the number of layers/hidden units in a neural network doesn't necessarily result in overfitting. Too few will result in low training and test accuracies; too many will result in high training accuracy but low test accuracy (overfitting). Somewhere in the middle there will be the right amount of hidden layers and units for the problem. Some complex problems like NLP require a number of stacked hidden LSTM layers http://ruder.io/deep-learning-nlp-best-practices/.

This thread might be useful: https://ai.stackexchange.com/questions/3156/how-to-select-number-of-hidden-layers-and-number-of-memory-cells-in-an-lstm


Help with breaking up a column in PowerBI to make a bar chart

You need to clean up the data. Load it into the Power Query editor.

Split the column with the variables by the delimiter ;
now you have a lot of individual columns, some have values, some are empty
select all columns except the newly split ones and use Unpivot > Unpivot Other Columns on the transform ribbon.
Now you have one row for each combination of document name and variable
Rename the columns as you see fit, delete the Attribute column and load the data.

Create a chart with the variable column in the legend.


Dimension Mismatch Error during dot product in Python

Firstly user_vecs, item_vecs are matrices, not vectors, since both the shapes are of two dimensions. If seems like you are trying to multiply both the matrices, so you can simply use user_vecs.dot(item_vecs). No need to transpose item_vecs. You can also use np.matmul(user_vecs, item_vecs).  Refer to this thread for more details.


You can use confusion matrix present in caret package.


How to model the probability of detecting an image, given it is seen multiple times

An approach that is not specific to the image domain is to use a probabilistic data structure like a Count-Min Sketch. A Count-Min Sketch data structure can accumulate information to estimate the observed frequency of an input value based on the past set of input values by using multiple hashing functions over the input.


Check their reference to the following page for PIP install:

https://github.com/MaxHalford/Prince


Suggest DataSet for Sentiment Analysis on Dutch languages


Which is better: multi-output model or separate models for similar tasks?

If both the tasks consolidated classes set is small,using the multi-output network will be useful. But if your consolidated classes set is very large then I would suggest you go with different networks.
Personally, I would prefer to train individual models as it is hard to optimize multi-output networks.


How to import image data into python for keras?

The docs for ImageDataGenerator suggest that no augmentation is done by default. So you could instantiate it without any augmentation parameters and keep the rest of your code for handling your directory structure:

train_datagen = ImageDataGenerator(rescale=1./255)

You are also allowed to write your own custom data generator and pass it to model.fit_generator(). Here is a nice tutorial.

Or if your data fits in memory you could write some simpler code possibly using keras.preprocessing.image.load_img to load all the images into an array and pass them to model.fit instead.


Hyperparameter Tuning in Machine Learning

In kNN algorithm, you only try to find a suitable value of parameter k. And some models may have many parameters that can be modified. Normal parameters are optimized by loss functions and Hyperparameter tuning allows you to set various parameters to get the best model. You set them before training. Its 2 methods: GRID METHOD AND RANDOM SAMPLING might work well.

Grid Method:
Impose a grid on possible space of a hyperparameter and then go over each cell of grid one by one and evaluate your model against values from that cell. Grid method tends to vast resources in trying out parameter values which would not make sense at all.

Random Sampling Method:
In random method, we have a high probability of finding a good set of params quickly. After doing random sampling for a while, we can zoom into the area indicative of the good set of params.

Random sampling allows efficient search in hyperparameter space. But sampling at random does not guarantee uniformity over the range of valid values. Therefore, it is important to pick an appropriate scale.

You can read this: Hyperparameter Tuning of Deep Learning Algorithm


First of all, it is not a general rule that continuous values should be discretized. Especially in the context of regression, it is actually very much wrong.

Regarding discretization, I believe you may have a wrong definition. Discretizing means transforming a continuous interval, say $[0,10]$ into sub intervals, $[0,1), [1,2),...,[9,10]$, and then attributing the same value to points in the same interval, for instance, $\forall x \in [0,1) \mapsto 0.5$.

I'm not sure what is your point have having value $\in \mathbb{R}$. If it's because you don't know what the interval you have is, then you can simply find the minimum and maximum of your data points, and assume these to be the total interval. Then you can discretize that interval in however many pieces you want.


Why is the number of samples smaller than the number of values in my decision tree?

Yes, it seems to display unique samples, the others have been duplicated by the bootstrap sampling.

There's the 0.632 rule - when you have N items and you take a random sample of size N with replacement (as bootstrap does), you only get 63.2% of the samples from N, the rest are duplicates.

That roughly matches what you've seen: 0.632 * (507+545) = 665 unique samples.

You can also try it with some Python code:

samples = np.arange(507 + 545)

bootstrap_sample = np.random.choice(samples, size=len(samples), replace=True)

print(len(np.unique(bootstrap_sample)))

This always prints values closely around 665.


I think the 2-layer neural network can be used to classify two or more class problem. It does not depend on the number of nodes of each layer. The problem you want to solve (i.e., the dataset you work on) decides how many classes you have.
For more, see here, here, here or here.


Preprocessing of 3D CAD files for Keras Conv3D input

I found a way that works now.

Keras Conv3D can be fed with a numpy array of the voxelization and the corresponding class of the file.

So first the files have to be voxelized with a tool like binvox.

Afterwards those files can be read into a numpy array with binvox-rw-py:

np.int32(binvox_rw.read_as_3d_array(f).data)

In combination with the classification, the files can be used as input for a Conv3D layer in Keras.

A complete example can be found here.


In this context, masking means replacing the token with a special [MASK] token. The network does not have the information of what the original token was, the only way how it could potentially figure out what it was is by looking at the context.
It is not the loss function that guarantees that the model learns something meaningful, it is architecture design that does not allow the workaround that you suggest.


No, you can't infer that.
I assume you have the same training set with the same predictors (symptoms). The only difference in the training set is the binary class label for each patient.

The smaller tree just means:
- with the given symptoms, it might be easier to distinguish between the people who have the condition 1 and the ones not having it. (since you have the same certainty with fewer information)
This distinction seems harder with medical condition 2, that's way you have to consider more symptoms to be pretty sure about your classification. So if condition 2 is a very mild condition where it is hard to diagnose even for an expert if someone has it then it would result in a deep tree.


ax = sns.scatterplot(x="total_bill", y="tip", data=tips)

With this format, Seaborn support Pandas DataFrame.
"total_bill" and "tip" are the name of the columns. "tips" is the pd.DataFrame.
You can plot on default Axis by
sns.scatterplot(x="X_Colname", y="Y_Colname", data=DataFrame)

#Can also pass the desired Axis to customize size etc.
sns.scatterplot(x="X_Colname", y="Y_Colname", data=DataFrame, ax=ax)


Is there any model agnostic way to calculate the weight importance for neural networks given a set of inputs?

You should look into fisher matrices. Elastic weight consolidation finds weights which are important for dataset 1 when training dataset 2 by finding the fisher matrix and slows the tuning of them down by adding a regularization term to the loss function. It's non trivial though and I got a lot of trouble implementing it in PyTorch.
The paper which proposed elastic weight consolidation is here: https://arxiv.org/pdf/1612.00796.pdf


Using Machine Learning techniques for text-analysis

Take a few words you know are linked with e.g republicans and with democrats.
Extract their word embedding. That is the vector representation of the word which will be high dimensional.
This will assume you have trained word embedding, either pretrained, or even better trained them yourself on your data. The word embeddings could have been trained/extracted through e.g LSTM training or matrix factorisations.

On the word embeddings, do dimension reduction(PCA as an example) to 2d. Plot! Those who are close together will be "similar" for the objective they are trained for.

Pretrained word embeddings exist to download. However, they might not be the optimal embeddings for your objective, but better than nothing still.


How to sample a statistically uniform dataset

There are a variety of sampling options:

Sample directly from the data which perfectly models the empirical distribution.
Fit a kernel density estimation (kde). Sample from the estimated kde function.
Create a histogram of the data, aka bin the data. Then treat each histogram as a probability mass function (pmf). Sample from bins proportional to their frequency.

You can create variations of the data or distributions:

Apply a transformation to the data. For example, a log transformation will transform skewed distribution to be more normal.
The histogram values could be changed to any shape.

Then sample from the changed data or distributions.

The most extreme option would define a uniform distribution across the data domain and sample from that distribution. The green box is a uniform distribution.


A huge list of open data sets is listed here:

Publicly Available Datasets

Including Amazon, KDnuggets, Stanford, Twitter, Freebase, Google Public and more.


How to get a confidence score for predictions?

No matter the model, you can always use the non-parametric bootstrap to construct a confidence interval for any parameter, including predictions (which are actually random variables themselves but are reported as expectations). Here's the general procedure:

Let $N$ denote the number of observations in your training data $X$, and $x_j$ denote the specific observation whose prediction, $\hat{y}_j$, you want a CI for.
Let $K$ denote some number of resampling iterations (Must be $\ge 20$ for a CI with coverage $\ge 95\%$)
For $i$ in $K$, draw a $N$ random samples from $X$ with replacement. Denote this $X_i^{*}$
Train a model on $X_i^{*}$ and use this model to form a prediction on $x_j$. Call this $\hat{y}^{*}_{ji}$
Estimate distributional parameters for $\hat{y}_j$ from your sample. A $100 -
 \alpha$ CI is given by the $\frac{\alpha}{2}$ and $100 - \frac{\alpha}{2}$ percentiles of $\hat{y}^{*}_{j}$.


A duplicate of this question in stackoverflow. Anyhow just to recap quickly, you can do it with:

scatter_kws={"s": 10}

And in your case it is simply:

sns.lmplot(x="x", y="y",data=data,  fit_reg=False,scatter_kws={"s": z*1000})


Any "rules of thumb" on number of features versus number of instances? (small data sets)

Multiple papers have opined that

only in rare cases is there a known distribution of the error as a function of the number of features and sample size.
The error surface for a given set of instances, and features, is a function of the correlation (or lack of) between features.

This paper suggests the following:

For uncorrelated features, the optimal feature size is $ N-1 $ (where $ N $ is sample size)
As feature correlation increases, and the optimal feature size becomes proportional to $ \sqrt N $ for highly correlated features.

Another (empirical) approach that could be taken, is to draw the learning curves for different sample sizes from the same dataset, and use that to predict classifier performance at different sample sizes. Here's the link to the paper.


Time Series Forecasting with Neural Network (mixed data types)

I guess I answered the question in the comments, so here goes.

Most ML models cannot deal with categorical values. A common way to solve this is to use one-hot encoding, also known as dummy variables. For very possible value of your categorical variable you create a column which is 0 unless this row has this category, then it is 1. It is possible to remove one of the categories since it is a linear combination of the other dummy variables (if all are 0, the last one must be 1).

The downside of this method is that it increases the dimensionality of your feature space. If you have enough data to support this or not that many categories that is not a problem. There are other alternatives, like taking the average feature of every category and adding that to your features as opposed to the categorical feature.


How to user Keras's Embedding Layer properly?

You've brought up some very good points. Let's walk through all of this:

A word embedding is a mathematical representation of a word.
This is needed since we cant work with text as plain input.

In order to get these word embeddings, there a different ways, methods and settings on how to calculate them. cbow, skip-gram and so on.

There are different pretrained word embeddings out there e.g.

Fasttext 2017, trained on Wikipedia; cudos to Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas
Word2Vec 2013, trained Google News dataset; cudos to Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean
Glove 2013, Wikipedia 2014 + Gigaword 5; cudos to Jeffrey Pennington, Richard Socher, Christopher D. Manning

This is just an excerpt of the most well-known ones. As you can see, they used different data sets - thus different word vocabulary and word embeddings respectively.

  I don't understand what is the real difference between these
  approaches regarding the desired outcome? Maybe the internal-only
  solution is not a semantic representation? What is the point of
  applying embedding layer to an external matrix of which the rows
  already have fix length?

Keras is an awesome toolbox and the embedding layer is a very good possibility to get things up and running pretty fast. Convert the text into one-hot/count matrix, use it as the input into the word embedding layer and you are set.

On the other hand if you use pre-trained word vectors then you convert each word into a vector and use that as the input for your neural network. This approach would give you more flexibility when it comes to feature engineering.

As mentioned above, pre-trained word vectors were given mostly general text data sets. You might run into the point where you have some special kind of data (e.g. Tweets) where people write or behave differently. So you might look into training your own embeddings, on your own dataset - at the end of the day it depends on your task/problem and the metrics that you are tuning towards.

  Moreover, what is the purpose/effect of the trainable parameter of the Embedding layer?

As you said correctly, it is to retrain the weights of the embeddings with the data set you use.

  I cannot understand what is the point to explicitly sign the end of sentence in a sentence based input

One of the most important things in NLP is feature engineering. It is the same as you sitting in school and learning a language, what needs to be considered, vocabulary, it's grammar and rules. Things that makes it easier for you as a human being to understand the language. The same is applied here. You can see it as one part of feature engineering, it all sums up to the bigger picture.

  And finally: how could a model predict the translation of a word which is not represented in the training set?

Either you convert the word that couldn't be found to an <UNK> token (unknown word) which then represents its group. However it requires having the <UNK> token in the trained word embeddings. Or you could use fasttext binary file, which calculates word vectors on the fly for unknown words.


Are deep - learning toolkits targeted for certain areas or all-purpose tool kits?

Yes, different toolkits are suited for different purposes given they contain different algorithms. This is due to the lack of a general AI (an AI that is intelligent in all respects). Modern AI has different algorithms which are better suited for different tasks. For example a CNN framework is much better for images than a NN would be, and much better than encoders. However, a LSTM framework would be better suited to capture temporal dependencies in the data.

However, if two frameworks are using the same algorithms they should both be relatively the same. Differences will arise due to the underlining codes being different as can be seen between Keras and TensorFlow. Although, this should be limited.

Worry more about the algorithm you are choosing for your task than the framework. Coding up most machine learning algorithms is not that hard if you want to customize it.


Scikit learn linear regression - learning rate and epoch adjustment

A linear regression model $y=\beta X+u$ can be solved in one "round" by using $(X'X)^{-1}X'y=\hat{\beta}$. It can also be solved using gradient descent but there is no need to adjust something like a learning rate or the number of epochs since the solver (usually) converges without much trouble.
Here is a minimal example in R:
x0 <- c(1,1,1,1,1)
x1 <- c(1,2,3,4,5)
x2 <- c(8,4,3,1,8)
x <- as.matrix(cbind(x0,x1,x2))
y <- as.matrix(c(3,7,5,11,14))

x
y

# (X'X)^-1 X'y
beta1 = solve(t(x)%*%x) %*% t(x)%*%y

# R's regression command
beta2 = summary(lm(y ~ x[, 2:3]))

# Gradient decent
m <- nrow(y)
grad <- function(x, y, theta) {
  gradient <- (1/m)* (t(x) %*% ((x %*% t(theta)) - y))
  return(t(gradient))
}

# define gradient descent update algorithm
grad.descent <- function(x, maxit){
  theta <- matrix(c(0, 0, 0), nrow=1) # Initialize the parameters

  alpha = .05 # set learning rate
  for (i in 1:maxit) {
    theta <- theta - alpha  * grad(x, y, theta)
  }
  return(theta)
}

# results without feature scaling
print(grad.descent(x,2000))
beta1
beta2