Is deduction, genetic programming, PCA, or clustering machine learning according to Tom Mitchells definition?

A while back, when presenting ML basics to experts of other domains, I also looked into the formal definitions; Mitchell's seems to be the most official/prominent is usage. However, I found that the interpretation can be also somewhat subjective. In addition, the lines between Machine Learning, Deep Learning, Artificial Intelligence and Statistics in general, do sometimes become a little hazy in various fields of application - the terms are also often abused and used as buzz-words (just like Big Data). I think the biggest guinine struggle is perhaps the interpretation of each of the terms that Mitchell introduces: Experience (E), Tasks (T) and Performance (P).

Perhaps it would be worth keeping another definition of machine learning in mind (my current favourite); that of Kevin Murphy from his book Machine Learning: a Probabilistic Approach (page 1):

  ... we define machine learning as a set of methods that can automatically detect patterns in data, and then use the uncovered patterns to predict future data, or to perform other kinds of decision making under uncertainty (such as planning how to collect more data!).

This is certainly a lot broader, and may help accept the greater everyday toolbox as machine learning, instead of tight constraints.

For these reasons, I would say my answer below is more of a discussion than a factual answer.

I'd happily be corrected by readers possessing better informed opinions or facts!  :-)

Inverted Deduction

Of the examples you list, I am least familiar with inverted deduction. It is described by Mitchell himself (in Section 10.6 of his book) and   this helpful Quora post, as logically equivalent to induction! The latter link includes one paragraph describing this symbolist method, which makes the idea quite accessible to someone more similar to modern connectionist approaches:

  The most successful form of symbolic AI is expert systems, which use a network of production rules. Production rules connect symbols in a relationship similar to an If-Then statement. The expert system processes the rules to make deductions and to determine what additional information it needs, i.e. what questions to ask, using human-readable symbols.

At this point, it helped me distinguish between mathematical induction and mathematical deduction. The quote above sounds a bit like a expert designed recommender system. Given that deduction implies to refer a universal hypothesis from axioms or known facts (I imagine Sherlock Holmes!), we start with some rules that come from experience (instead of starting with just data). The idea with inverted deduction is that we have some observations and can induce relationships/rules that support the data/observations, and from that we can improve our system. We accept this to be weaker than pure mathematical deduction, and the starting point differs to Mitchell's own definition of machine learning, but if machine learning is an iterative cycle, is our starting point in that cycle critical to the definition?

Putting inverted deduction in the framework of a machine learning  problem (as per our definition with T, E and P), we have the following definition from Mitchell to help:

  ... assume as usual that the training data D is a set of training examples, each of the form $\langle x_i, f(x_i) \rangle$. Here $x_i$ denotes the $i^{th}$ training instance and $f(x_i)$ denotes its target value. Then learning is the problem of discovering a hypothesis $h$, such that the classification $f(x_i)$ of each training instance $x_i$ follows deductively from the hypothesis $h$, the description of $x_i$ and any other background knowledge B known to the system.

He explicitly writes that the Task is to extract the hypothesis (the target function) rules that explain the system as a whole. Experience is the observations plus background knowledge (including the experience of the experts who create/define the rules). Performance remains as the accuracy or strength of the final set of rules. It is especially the interpretation of experience as data as well as expert/background knowledge, which differs slightly from simple observations.

As a facit: induction allows us to make statements about relationships, mapping input to outputs on the known range of our observations, but it is not as watertight as deduction, which by definition must be a universal truth (and therefore becomes a theorem). This makes induction sound a lot like machine learning (and therefore inverted deduction!)- just as neural networks learn relationships very well, but only over their input domain i.e. not universally.

Genetic Programming

Straight off the bat, I would say this does fall into machine learning by the given definition. Models are iteratively assessed on their ability to perform a predefined task, then further training data is used to improve the performance on that task.

Given the usual approach of allowing many models to train, then selecting the best models and allowing them to somehow be fused (equivalent to genetic cross-pollination in biological reproduction). The resulting child-model would thereafter (hopefully) contain the best traits of the two best parent-models. Just as can be the case in biology, or within Reinforcement Learning and exploration step (as opposed to exploitation), the resulting child-model also is sprinkled with a bit of luck/randomness in the form of a mutation. Here is a summary article. The resulting child-model may well be degenerate i.e. less performant than the parents.

  OP: What is experience in this case? Maybe "iterations" / "epochs"?

Given the above rough description, I would call a training case as all the work done between the consolidatory steps (breeding and mutation: producing the child-model). This is also similar to the paradigm used within meta-learning (a great lecture from Ilya Sutskever), whereby a task is defined as the sample dataset, plus information about the task itself. This is inspired by the idea of allowing algorithms to "learn to learn". For me, this is very close to the idea of genetic algorithms and evolutionary models.

PCA

I'm not convinced PCA falls into a machine learning algorithm; however, it is such a commonly used pre-processing method, I would be surprised if it didn't get a mention in any machine learning book.

If I had to force PCA into Mitchell's definition, this would be my best shot: The Task, T, would be simply that of PCA; to reduce the dimensionality of the dataset. The Performance metric, P; I would interpret as the percentage of the variance that is contained with the first $k$ principal components - e.g. just fix $k = 3$, and run the algorithm with more and more data. More data would allow this to be performed more accuractely, i.e. with higher levels of confidence and the variance should be described more densely be the denser set of input points. So, let's say P is given by:

$$
P = \sum_{i=1}^{k} C_i
$$

where $k = 3$, for example and $C_i$ is the $i^{th}$ principal component's normalised coefficient. This sum therefore integrates to 1 over all $k$.

If the value of P actually decreases after adding more data, I would perhaps argue that we have actually learned more about the sample distribution with the extra data, even though our performance metric technically indicates worse results.

Despite this attempt to shoehorn PCA into the definition - I am still uneasy about calling this machine learning (see the second paragraph in my conclusion).

Clustering

I agree with you that the only real experience is the number of samples. To add to any doubts, clustering (like PCA), is an unsupervised method - there are typically no labels on the data.

If experience, E, is then the number of data points, the metric, P, would be the inherent error measured used to iteratively improve the clustering result. For example, in the case if KNN, it could be the sum of squared distances between points one cluster and those of a different clusters. Allowing the model to train with additional data points (assuming the ability to overfit), it would not produce a worse value for P: either the same or better.

In my opinion, clustering is included in many machine learning books, not because it falls nicely into Mitchell's definition, but because it is a widely used and extremely useful tool. That being said, it can also be used in the a non-parametric setting, e.g. using a Dirichlet process  as a prior and allowing the number of clusters to grow as more data is provided. I find that idea to be easier to merge into Mitchell's notion of a model improving a metric given more experience!

The fact that clustering is indeed defined as un-supervised learning, i.e. without any sample labels, just makes the idea here more interesting to me.

Conclusion

I agree with @Anony-Mousse, in that PCA and clustering really don't give the impression of a learning algorithm. I think it is because I conflate statistical learning with human learning. General discussion around machine learning and AI tends to use anthropomorphism: e.g. "AlphaZero learned from scratch how to play a wide range of Atari games"... If we replace AlphaZero with Alice or Bob, it is clear we are talking about a human. This way of describing algorithms leaks into methods like PCA and clustering - people say how PCA learns representations of the variance in the data, even though we struggle to align Mitchell's definition of machine learning to PCA. After the explosion of machine and deep learning since ca. 2012, we see there are still many undisputed terminologies floating around - something that is unavoidable for a rapidly devloping area.

To address the discssion in the comments to Anony-Mousse's answer: Arthur Samuel coined the term machine learning, as a computer being able to learn without being explicitly programmed. Here, in its original context, I believe it meant to learn as a human does, i.e. to exhibit the use of intelligence (paraphrased from quotes in this nice article). Humans learn a great deal from observations, even without knowing the correct answer (the case without labels). Given this, and drawing the direct comparison to unserupervised learning, one could argue that any process by which a machine is able to improve some metric is indeed a learning machine. So clustering as an example, is learning, as it improves a metric and derives new insights into data (without labels). In the case of PCA, the metric is unclear - it must be derived from the usefulness of the results for the task at hand: beauty is the eye of the beholder.

In the end, it doesn't matter too much to me, as long as we know what we are talking about and can make progress in research and applications thereof. My only concern is that this confusion makes teaching the subject more difficult and raises the barrier to entry for newcomers, who have to deal with sometimes contradictory terminology.


What is an intuitive explanation for the log loss cost function?

Ok, so here is how it works. Say you want to classify animals, and you have cats, dogs and birds. This means that your model will output 3 units in the form of a vector (call it a list if you prefer).

Each element of the list represents and animal, so for example

Position 0 represents how likely the input is to be a cat
Position 1 represents how likely the input is to be a dog
Position 2 represents how likely the input is to be a bird.

Now imagine you get an input that is a bird, in a happy world, your algorithm should output a vector like

[0, 0, 1]

That is, the input has 0% chance of being a cat, a 0% chance of being a dog and a 100% chance of being a bird.

I reality, this is not so simple, and most likely your output would be something like this

[0.15, 0.1, 0.75]

This means 15% chance of being a cat, 10% chance of being a dog and 75% chance of being a bird. Now, notice that this means your algorithm will still consider the input as a bird, so in terms of classification, sure the output would be correct... but it would not be as correct as it had predicted 100% chance of a bird.

So, the intuition is that the logloss measures how far away you are from perfection, where perfection would be identifying the correct label with a 100% chance and the incorrect labels with a 0% chance.

Final word of advice: Do NOT be afraid of math, you will really need to get the grasp of it at some point, do not let the sum terms  intimidate you, after all they just represent loops in programming.

UPDATE

Let's dive into the math, specially to demystify it.

The logloss formula is given by

$$ LogLoss = - \frac{1}{n} \sum\limits_{i=1}^n [y_i \cdot log(\hat{y_i}) + (1-y_i) \cdot log(1-\hat{y_i}) ] $$

Where

$n$ represents the number of examples, in our case i will use 2 examples.
$y_i$ represents the correct answer for example $i$
$\hat{y}_i$ represents our prediction for example $i$

So, for our example we have two examples, (remember that the examples are denoted by $y_i$) which are

[0, 1, 0] # Example 1: This means the correct answer is dog
[1, 0, 0] # Example 2: This means the correct answer is cat

Now, let's go for our predictions, remember that the predictions are denoted by $\hat{y}$ let's say they are

[0.1,  0.6,  0.3]
[0.85, 0.05, 0.1]

And let's apply the scary formula here. First notice that $$\sum\limits_{i=1}^n$$ just means sum all the elements from i=1 to $n$, in our case, $n$ is the number of examples, and we have two of those.

So, for $i=1$ we have

$[y_i \cdot log_e(\hat{y_i}) + (1-y_i) \cdot log_e(1-\hat{y_i}) ]$

For $i=1$

term1 = [0, 1, 0] * log([0.1, 0.6, 0.3]) + (1-[0, 1, 0]) * log(1 - [0.1, 0.6, 0.3])

For $i=2$

term2 = [1, 0, 0] * log([0.85, 0.05, 0.1]) + (1-[1, 0, 0]) * log (1-[0.85, 0.05, 0.1])

And finally we have

log_loss = (-1/2) * (term1 + term2)

Using sklearn log_loss, the answer is 0.3366

Now, do not get too lost in the math here, just notice that it is not THAT hard, and also do understand that the loss function here tells you essentially how wrong you are, or if you prefer, it measures the "distance" from perfection. I strongly recommend you to code the logloss by yourself (normally numpy is a good option to do so :)


Why real-world output of my classifier has similar label ratio to training data?

What can be the reason of such behavior?

A classifier only tries to capture the features-label relationship as accurate as it can, it does not learn nor does it guarantee that the ratio of predicted labels to be close to the true ratio. However, if sampled classes (even balanced) are good representatives of true class regions and classifier finds good decision boundaries, the closeness of ratios will happen naturally. Therefore, the explanation is that the sampled negative class is not a good representative of its true occupied regions, meaning some regions are not sampled well, and/or classifier is finding bad decision boundaries. Here is a visual illustration (drawn by myself):

In the case of good training set, predictions resemble the real-world ratio (2:1), even though model is trained on a balanced set (1:1). In the case of bad training set and/or bad decision boundary, predictions are completely incompatible (1:2) with the real-world ratio (2:1).

  What should I do to solve it?

If the problem is related to bad negative representatives

Use more negative samples in the training set, i.e. imbalanced training (you can nullify the effect of imbalanced samples with class weights), or
Train with balanced classes but change the decision threshold (a post-training solution). That is, instead of assigning instances with output > 0.5 to positive class, use a harder-to-pass threshold like output > 0.8 to decrease the number of positive predictions.

If the problem is related to classifier performance, we should come up with a better classifier which is an open-ended endeavor.

However, in my opinion, you should not select models based on the ratio of positive predictions. You should ecide based on a metric like macro-f1 (or any other one). Therefore, by using a validation set, a model that produces more positive samples and has a higher macro-f1 should be preferred over a model that produces less positives but has a lower macro-f1.

EDIT:

As @BenReiniger pointed out in another post, a hidden assumption here (specially in the sketch) is that classes are "clearly separable". This assumption becomes more justified in higher dimensions. As an example, dogs and cats are clearly separable based on their images (high dimensions) compared to their length (one dimension).


Several clarifications:

you can program with object-oriented (OOP) concepts in R, even though OOP in R has slightly different syntax from other languages. Methods do not bind to objects. In R, different method versions will be involved based on the input argument classes (and types). (Ref: Advanced R)
you can also replace nans with mean / any stat. / value in R using a mask if you store the data in a dataframe (See SO post)
There is no problem using them interchangeably. I use R from Python using the package RPy2. I assume it is equally easy to do it the other way round.

At the end of the day, any language is only as good as how much the users know about it. Use one that you are more familiar with and try to learn it properly using the vast online resources online.


Possibility of working on KDDCup data in local system

I think that you have, at least, the following major options for your data analysis scenario:

Use big data-enabling R packages on your local system. You can find most of them via the corresponding CRAN Task View that I reference in this answer (see point #3).
Use the same packages on a public cloud infrastructure, such as Amazon Web Services (AWS) EC2. If your analysis is non-critical and tolerant to potential restarts, consider using AWS Spot Instances, as their pricing allows for significant financial savings.
Use the above mention public cloud option with R standard platform, but on more powerful instances (for example, on AWS you can opt for memory-optimized EC2 instances or general purpose on-demand instances with more memory).

In some cases, it is possible to tune a local system (or a cloud on-demand instance) to enable R to work with big(ger) data sets. For some help in this regard, see my relevant answer.

For both above-mentioned cloud (AWS) options, you can find more convenient to use R-focused pre-built VM images. See my relevant answer for details. You may also find useful this excellent comprehensive list of big data frameworks.


What are some situations when normalizing input data to zero mean, unit variance is not appropriate or not beneficial?

A detailed answer to the question can be found here.

  [...]are there times when it is not appropriate or not beneficial?

Short answer: Yes and No. Yes in the terms, that it can significantly change your output of e.g. clustering algorithms. No, on the other hand, if these changes are what you want to achieve. Or to put it in the words of the author of the mentioned source:

  Scaling features for clustering algorithms can substantially change the outcome. Imagine four clusters around the origin, each one in a different quadrant, all nicely scaled. Now, imagine the y-axis being stretched to ten times the length of the the x-axis. instead of four little quadrant-clusters, you're going to get the long squashed baguette of data chopped into four pieces along its length! (And, the important part is, you might prefer either of these!)

The take-home-message of this is: always think carefully about what you want to achieve and what kind of data your algorithms prefer - it does matter!


I just hacked together a very basic helper in python
it requires that all images are stored in a pyton list allImages.

import matplotlib.pyplot as plt
category=[]
plt.ion()

for i,image in enumerate(allImages):
    plt.imshow(image)
    plt.pause(0.05)
    category.append(raw_input('category: '))


Transfer Learning Question: Extending the Functionality of a Multipose-Estimation Machine Learning Model?

I worked in a similar kind off object detection problem.so, my suggestion is Transfer Learning.

i)
The optimal way is to go with transfer learning.It allows us to build accurate models in a timesaving way.With transfer learning, instead of starting the learning process from scratch, you start from patterns that have been learned when solving a different problem. This way you leverage previous learnings and avoid starting from scratch.

A pre-trained model is a model that was trained on a large benchmark dataset to solve a problem similar to the one that we want to solve. Accordingly, due to the computational cost of training such models, it is common practice to import and use models from published literature (e.g. VGG, Inception, MobileNet).

Several pre-trained models used in transfer learning are based on large convolutional neural networks (CNN).

When you’re repurposing a pre-trained model for your own needs, you start by removing the original classifier, then you add a new classifier that fits your purposes, and finally you have to fine-tune your model according to one of three strategies:

1.Train the model.
2.Train some layers and leave the frozen.
3.Freeze the convolutional base.

Approaches for Transfer learning process:

1)Select a pre-trained model.
2)Classify your problem according to the Size-Similarity Matrix.
3)Fine-tune your model.

ii) Tensorflow offers a vision model named as PoseNet.

PoseNet is a vision model that can be used to estimate the pose of a person in an image or video by estimating where key body joints are.
From here you can freeze the model and fine tune it. orelse you can use other pretrained image model (e.g. mobilenet , resnet).

iii) The estimation directly from the raw images first request you to identify or detect the people , so it relays on another model.


Pyspark.read from MariaDB fetches duplicate header rows instead of data

Sometimes adding an alias to subquery confuses PySpark. Try:
.option("dbtable", "select fname, lname from SYNTH_EHR_NORM.PATIENTS limit 20") \


Whole time column in csv file convert into UTC (epoch) using python

One of the easiest ways to solve the problem of plotting your time-series graph is by using pandas. You can use this code

data = pd.read_csv('YOUR_FILENAME_HERE.csv')
data['datetime'] = data['date'] + ' ' + data['time']
data['datetime'] = pd.to_datetime(data['datetime'])
data.index = data['date_time']
data.drop(['date', 'time', 'datatime'], inplace = True, axis = 1)
data['x1'].plot()


For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task.

Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element    of the matrix is the magnitude  of similarity between  and . This magnitude of similarity can be calculated by using any similarity function, most popular being the Cosine Similarity.

In spark this can be done by:

Create a k X n matrix, where n items are described as k-dimensioned vectors. For representing each item as a k dimension vector, you can use ALS which represents each entity in a latent factor space. The dimension of this space (k) can be chosen by you.

This k X n matrix can be represented as RDD[Vector].

Convert this k X n matrix to RowMatrix.
Use columnSimilarities() function to get a n X n matrix of similarities between n items.

Finally , whenever an itemi is being viewed, you can recommend other m items obtained by sorting the items in row i by the decreasing value of similarities and picking top m.

More details can be found here.


By default, the image is loaded as a color Image i.e. 784*3 = 2352

Load image as grayscale i.e. use parameter color_mode="grayscale"
No need to of np.vstack(), simply reshape to (-1,784)


You are asking about Data Dredging, which is what happens when testing a very large number of hypotheses against a data set, or testing hypotheses against a data set that were suggested by the same data.

In particular, check out Multiple hypothesis hazard, and Testing hypotheses suggested by the data.

The solution is to use some kind of correction for False discovery rate or Familywise error rate, such as Scheffé's method or the (very old-school) Bonferroni correction.

In a somewhat less rigorous way, it may help to filter your discoveries by the confidence interval for the odds ratio (OR) for each statistical result.  If the 99% confidence interval for the odds ratio is 10-12, then the OR is <= 1 with some extremely small probability, especially if the sample size is also large.  If you find something like this, it is probably a strong effect even if it came out of a test of millions of hypotheses.


Standard correlation coefficient of various datasets

First part:
The question you have to ask yourself is "Given I know the x-value, what knowledge do I gain about the y-value". And a small side note: you may only consider linear relationships.
Consider the first row. In the first image, if you know the x-value, then the y-value is y = x, i.e. you know the y-value exactly. In the following two images, if the x-value is high, then the y-value is high too, but there is a random component too. Thus, the correlation coefficient is positive (i.e. knowing x helps in estimating y), but not 1 (i.e. there is no exact equation y = a*x. In the middle image, there is no relationship between the x- and y-coordinate - they are purely random, so no correlation. In the right three images, it is the same story as in the left three images, but the sign is flipped.
Now consider the second row: in the first three cases, given you know the x-value, you can always infer the y-value, e.g. y = x, y = 0.5 x, and y = 0.1 x. There is no random component in y, so if you know x, you also know the exact value of y, thus the correlation coefficient is 1. It doesn't matter if the formula is y = 1 x or y = 0.1 x (i.e. the slope of the line doesn't matter!) all that matters for the correlation coefficient is that there is such a linear coefficient which leads to an exact match.
In the third row, knowing x does give some knowledge on y. For example in the middle plot, you have a relationship of y = x^2 + random, so there is indeed some relationship between x and y, but it is non-linear. Thus the correlation coefficient is zero. There is no way for you to say "a high x-value also leads to a high y-value" or "a high x-value leads to a low y-value".

Second part:
I assume this plot is generated e.g. with Seaborn's pairplot() function, though of course different functions exist for creating this kind of plot. It is important to note that the plots on the diagonal are fundamentally different plots than the other elements.
The off-diagonal plots are two-dimensional scatter plots, i.e. you draw a point for each data sample.
The diagonal plots are histograms of each feature, and not scatter plots. This is because a scatter plot of twice the same variable would always give you a straight line y = x.
The histograms are actually quite useful as you can get a feeling for the data and make a guess about the underlying distribution.
For example, your total_rooms variable is heavily skewed, most houses have very few rooms, but there are a few outliers with huge numbers of rooms. I thus wouldn't trust the mean value of total_rooms much and rather use the median - of course this is highly dependent on what kind of analyse you are doing.


Nested features with one to many relationships

This is called "bilevel optimization" where one problem is embedded (nested) within another. If there are more than one objectives at one or both levels, it is then called "multi-objective bilevel optimization". There is a rich literature on this problem, including the book Multilevel Optimization: Algorithms and Applications.

Finding solutions to minimize the objectives often does not require machine learning, just linear programming.


Using python to identify common paths that users follow among different activities

You can rank the paths by frequency using the database (SQL), constraining the start-, end- points as necessary once you fix the length of the window. If you let the path length be variable then you will not be able to do it all in SQL. In that case you can learn the transition probabilities between states then solve a weighted shortest path problem, where the distances are log likelihoods. Or you can use a heuristic like A* search. I do not know of any library that will do all this since it is very ad hoc, but you can do the visualization in networkx.


Converting a column to a DataTimeIndex and query by day

Solution-

df.loc['05-26-2019']

Borrowed from-

https://stackoverflow.com/questions/36871188/how-to-access-pandas-dataframe-datetime-index-using-strings


Can we learn f(x)=1/x using a neural network exactly?

Since the activation function itself can be an inverse function, the answer is yes.


What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

Maybe I am not understanding your question properly, but k-means clustering is not sensitive to the zeros. More accurately zeros are valid values for features to cluster on.

While PCA may enable you to reduce the number of features to train on, it may complicate the clarity of the clusters from k-means. PCA may be beneficial in its own right, but AI would resist the urge to feed PCA data into k-means simply to reduce the number of zeros in your data.

HTH


Should one normalize the frequency values when feeding it as an input to machine learning model?

As generic advice, any algorithm that uses distances might be affected by scaling or normalization. Thus, if you use one of them, you need to think about it twice and consider how this will alter your problem.

I will take the answer from the CrossValidated StackExchange since another user has already explained it there.

  If your variables are of incomparable units (e.g. height in cm and
  weight in kg) then you should standardize variables, of course. Even
  if variables are of the same units but show quite different variances
  it is still a good idea to standardize before K-means. You see,
  K-means clustering is "isotropic" in all directions of space and
  therefore tends to produce more or less round (rather than elongated)
  clusters. In this situation leaving variances unequal is equivalent to
  putting more weight on variables with smaller variance, so clusters
  will tend to be separated along variables with greater variance.

ttnphns's answer


Logistic Regression - ROC curve plots Sensitivity vs Specificity instead of (1-Specificity)

As you can see, the specificity in the x-axis goes from 1.0 to 0.0 (backwards).

So the effect is the same, you are seeing 1-Specificity


Monitoring machine learning models in production

The changes in distribution with respect to training time are sometimes referred to as concept drift.
It seems to me that the amount of information available online about concept drift is not very large. You may start with its wikipedia page or some blog posts, like this and this.
In terms of research, you may want to take a look at the scientific production of João Gama, or at chapter 3 of his book.
Regarding software packages, a quick search reveals a couple of python libraries on github, like tornado and concept-drift.
Update: recently I came across ML-drifter, a python library that seems to match the requirements of the question for scikit-learn models.


If you only care about word similarity, then apply Occam's Razor and use word2vec. There is no need to increase model complexity if not going to be used.
Also, the quality of embeddings is primarily increased through the size and diversity of the corpus. The algorithm has a much smaller effect on the quality of the embedding.


If you are trying to predict the target of winner or loser, then it is a binary classification problem.
If you frame it as a binary classification, you'll have to reorganize the data as tidy data such that each row is an individual observation and each column is a feature.


Why would it be bad to fit a regression model to a binary classification problem?

It would work, afterall ML is a lot about engineering and hacking the things together. However, it would perform not so well, as for example logistic regression. If you compare the a linear line and logistic regression you will notice that the gradients of their respective loss functions for points near the decision boundary ("threshold") and far away from it are quite different. While for logistic regression points, that lie close to the decision boundary or are wrongly classified the absolute value of the gradient is larger. As a consequence the optimization, if a gradient approach is chosen, will be impacted by these points the most. Thus, the curve will be fitted to improve the classification.

Whereas in linear regression the distance of each point to the linear line is to be minimized, so independently of if the point is already classified correctly, it will impact the optimization. This leads to the optimization to be influenced a great deal by points that are already classified correctly and don't really need to influence the adjustement of the linear function. They suppress the importance of the points that are actually wrongly classified and that should impact the optimization at large.


Imputation of missing values and dealing with categorical values

I think like @El Burro suggested, you I believe you should focus on feature transformation mainly. Use different techniques for different features. For straightforward features, such as occupation or gender for example, use one-hot encoding, while for others  you can use some kind of hierarchical mapping-clustering (e.g. map values to groups defined by you, for example if those urls linked to products, make 30 different groups with similar types of products and map the urls to these groups. Then you can use again one-hot encoding for these mapped features).

For the textual features you mentioned, I'm pretty confident you could drop some of them. If you really want to exploit some kind of information generated from text though, for starters you could either do some tf-idf feature extraction, so as to generate features for each text-snippet or consider topic modeling (take a look at gensim for python if interested) and represent each text as a mixture of topics.


One of the standardized process for doing data analysis / data mining is the CRISP DM (Cross-industry standard process for data mining)
 which is one of the most adopted norms in the industry.

The process is very straightforward and starts by an important task which is the Business understanding of the issue or data you are dealing with.

This means that a prior research should be done if the analyst is not very familiar with the industry or the problem.

What makes this standard very popular is the fact that you can iterate various times between the step of formulating the problem ( writing in simple terms), data exploration and data visualization.

Here is a a very rich document that you can follow.

It would take a bit of time to go through all of the steps, however what could be quit interesting is the practice part, where you adapt your own client's problem into a simple question and start your exploration phase right away.


Versatile data structure for combined statistics

It sounds like you would like the Boost Accumulators library:

  Boost.Accumulators is both a library for incremental statistical
  computation as well as an extensible framework for incremental
  calculation in general. The library deals primarily with the concept
  of an accumulator, which is a primitive computational entity that
  accepts data one sample at a time and maintains some internal state.
  These accumulators may offload some of their computations on other
  accumulators, on which they depend. Accumulators are grouped within an
  accumulator set. Boost.Accumulators resolves the inter-dependencies
  between accumulators in a set and ensures that accumulators are
  processed in the proper order.


Depending on the images you used but I would not necessarily convert them to greyscale as there could be vital information in the colors of the image when it comes to making a prediction.

Feature scaling is not necessary for any of those algorithms but they could perform better with it.

Yes you should plot the Receiver Operating Characteristic and the Area Under the Curve. This is a metric you should be using as your problem is about Binary Classification.

The rest looks good.


This task looks similar to what is called text segmentation, in particular topic segmentation. I don't know any python package to do it but apparently Google gives a good few results for "semantic text segmentation python" (I'm not sure that this is the best phrase, you might want to try variations).

Note: this is still an active NLP research topic as far as I know. I don't know how fast reliable python packages are written and maybe there are some for this, but I wouldn't be too surprised if there were only research prototypes available at this stage.


Investigate why data is missing? After finding out reasons, what should I do next?

Add the binary column but start with summary statistics by group (missing/not missing) for all of the x’s. That may reveal an implied ‘why’, eg the values are missing whenever some other x is always above/below some value.


What does this symbol means, what operator is it?

$\mathbb{E}$ means expected value.

The subscript is there to clarify which random variable is the expected value taken.


So we have that a word to vector model has been trained on a certain corpus to be able given words as inputs (one hot encoding is one way for the input) to represent the word as a vector of a usually high dimensionality of numbers.

Does it carry information? Yes, we could say that it carries the information about the words. Similar words have similar vectors. The point is that us as humans we do not define the similarity but the neural network defines the similarity for us based on the corpus that we are providing.

Does it carry meaning? No, because this representation does include grammatical or syntactical information and therefore the answer would be that it does not carry any meaning.


Android: NLP library for date recognition in string

Maybe too late, but you could have a look at HeidelTime. It is a Java library that can detect dates, times, durations and time sets in texts for many languages. The downside of HeidelTime is that it seems to be not maintained since 2018. Also, you need the TreeTagger tool in order to use it. But the README of HeidelTime explains the necessary steps quite detailed.

Another alternative or addition to HeidelTime would be defining several regular expressions of common date and time formats and find these in your input text.


Features used by MAUI key phrase extraction tool

The references are in the link that you give: here you can find references to two papers which describe the method:

Domain-Specic Keyphrase Extraction
KEA: Practical Automatic Keyphrase Extraction

Fyi it seems there's no neural nets involved (the question is tagged with neural-network)


LSTM: many to one and many to many in time-series prediction

The choice is mostly about your specific task: what do you need/want to do?

Many-to-one (single values) models have lower error, on average, since the quality of outputs decreases the more further in time you're trying to predict. Many-to-one (multiple values) sometimes is required by the task though.

An alternative could be to employ a Many-to-one (single values) as a (multiple values) version: you train a model as (single), then you use it iteratively to predict multiple steps. I personally experimented with all these architectures, and I have to say this doesn't always improves performance.

Based on my experience, Many-to-many models have better performances. For example, I had to implement a very large time series forecasting model (with 2 steps ahead prediction). The best model was returning the same input sequence, but shifted forward in time of two steps. It appeared that the model was better at keeping the predicted values more coherent with previous input values. It was a seq2seq RNN with LSTM layers.

EDIT:

The Loss doesn't strictly depend on the version, each of the Losses discussed could be applied to any of the architectures mentioned. A problem for multiple outputs would be that your model assigns the same importance to all the steps in prediction. This is something you can fix with a custom MSE Loss, in which predictions far away in the future get discounted by some factor in the 0-1 range. In that way your model would attribute greater importance to short-range accuracy. Let me know if that's helpful.


What are the disadvantages of having a left skewed distribution?

There are issues that will depend on specific features of your data and analytic approach, but in general skewed data (in either direction) will degrade some of your model's ability to describe more "typical" cases in order to deal with much rarer cases which happen to take extreme values.
Since "typical" cases are more common than extreme ones in a skewed data set, you are losing some precision with the cases you'll see most often in order to accommodate cases that you'll see only rarely. Determining a coefficient for a thousand observations which are all between [0,10] is likely to be more precise than for 990 observations between [0,10] and 10 observations between [1,000, 1,000,000]. This can lead to your model being less useful overall.
"Fixing" skewness can provide a variety of benefits, including making analysis which depends on the data being approximately Normally distributed possible/more informative. It can also produce results which are reported on a sensible scale (this is very situation-dependent), and prevent extreme values (relative to other predictors) from over- or underestimating the influence of the skewed predictor on the predicted classification.
You can test this somewhat (in a non-definitive way, to be sure) by training models with varying subsets of your data: everything you've got, just as it is, your data without that skewed variable, your data with that variable but excluding values outside of the "typical" range (though you'll have to be careful in defining that), your data with the skewed variable distribution transformed or re-scaled, etc.
As for fixing it, transformations and re-scaling often make sense. But I cannot emphasize enough:
Fiddling with variables and their distributions should follow from properties of those variables, not your convenience in modelling.
Log-transforming skewed variables is a prime example of this:

If you really think that a variable operates on a geometric scale,
and you want your model to operate on an arithmetic scale, then log
transformation can make a lot of sense.
If you think that variable operates on an arithmetic scale, but you
find its distribution inconvenient and think a log transformation
would produce a more convenient distribution, it may make sense to
transform. It will change how the model is used and interpreted,
usually making it more dense and harder to interpret clearly, but
that may or may not be worthwhile. For example, if you take the log of a numeric outcome and the log of a numeric predictor, the result has to be interpreted as an elasticity between them, which can be awkward to work with and is often not what is desired.
If you think that a log transformation would be desirable for a
variable, but it has a lot of observations with a value of 0, then
log transformation isn't really an option for you, whether it would
be convenient or not. (Adding a "small value" to the 0 observations
causes lots of problems-- take the logs of 1-10, and then 0.0 to
1.0).


Consolidating multivariate time-series information from many data sets

You could put all Information into 1 table with 30*7 features, since your timeline is the same for all tables.
Then, in order to predict the under/over budget, you can build a model with 30 output-nodes, where one node indicates whether a certain project was under/over budgeted.


How does sklearn random forest decide feature threshold at node splitting exactly?

After some searching, I found this source code:
https://github.com/scikit-learn/scikit-learn/blob/7813f7efb5b2012412888b69e73d76f2df2b50b6/sklearn/tree/_splitter.pyx#L467
I think, but am not 100% sure as my code comprehension is imperfect, that:

Yes
x_0 = (a+b)/2, where a is the nearest neighbor to the left and b is the nearest neighbor to the right


Is there a way to get y_pred values from saved Keras model?

If I've figured it out correctly, the answer is no. The point is that your saved model solely contains the network architecture and the parameters it has. What you want relates to the recall phase where you have to provide input to get output. This means that you need input data to be fed to your network in order to get output. What you want can be done using another approach. First, load your network and feed your data to your model. After that, get the outputs and store the inputs and outputs alongside each other using Numpy save method or maybe .h5 format.

The flow for achieving the y_pred can be like the following sequence of actions:

Load your model.
Feed your data to your model and get y_pred.
define a Numpy array of inputs and a Numpy array of outputs.
Store inputs, real outputs and y_preds using the methods which are available.
Later, when you want to make your confusion matrix, you can load your inputs and outputs, and the real outputs to make your matrix.


How long does it typically take to train a MNIST data on a Mac Pro?

Did you try to do a model summary and check the size of your model. Yes, i may take hours as you are adding lot of layers. Try starting with 10 or 20 epochs and check the accuracy of your model gradually. May reduce your training time.

When you do the above exercise its better to have a validation data set, rather than fitting the model on the training data set time and again.


Understanding the Shuffle and Split Process in a Neural Network Code


If your training accuracy is significantly higher than your validation accuracy, you are overfitting. You might want to try increasing the amount of dropout or using batch normalization.
Your test set is very small, leading to high variance in your results. Your model might actually be better than it seems due to this variance. Try cross validation so you can test on more examples.
Make sure your train and test set are randomly split. Otherwise your train and test examples might look different, leading to lower performance on the test set.
Do error analysis to try to identify what mistakes the model tends to make. Does it have trouble finding bar codes at certain angles, at a certain size, or on a certain background color? Error analysis by itself won't help but it might help you identify why your performance is worse than you want it to be.
Use a more powerful model (more parameters). Counterintuitively, bigger neural nets that have more potential to overfit sometimes generalize better than smaller neural nets. We don't yet understand why this is the case but the phenomenon is well known so you might want to see if it's the case here.


Keras Fit Function (R): Train Regression Model with multiple Labels

x is an array/matrix of the inputs/features/independent variables.

y is an array/matrix of the output(s)/target(s)/dependent variable(s).

Your x array should contain the features extracted from the dataset of people's faces (however you may choose to do that). You can even include the gender and ethnicity alongside the features extracted from the dataset, if this information will be available to you while you deploy the model.

Your y array should contain your target variable, i.e., the age associated with each face.

Response to EDIT:

Unfortunately, I actually haven't used Keras with R. But judging by the error you are seeing (and the prefix of the function), I would say it is most likely compatibility issues between R lists and whatever Keras expects underneath. Keras is implemented in Python, so it is probably expecting a Numpy array or just a Python list. I suspect you will have better luck creating an R data.frame for your input features.

This tutorial I found might be helpful as it uses Keras and R to build a CNN.


Handling outliers and Null values in Decision tree

Outliers: In decision tree learning, you do splits based on a metric that depends on the proportions of the classes on the left and right leaves after the split (for instance, Giny Impurity). If there are few outliers (which should be the case: if not, you cannot use any model), then they will not be relevant to these proportions. For this reason, decision trees are robust to outliers.

Null values: You have to replace them (unless the software you use already does that for you, which is not generally the case).

Edit about outliers:

What I have said in outliers is only about classification trees. However, it is certainly not true in regression tress. Regression tree split criterium depends on the averages of the two groups that are splitted, and, as the average is severly affected  by outliers, then the regression tree will suffer from outliers. There are two main approaches to solve this problem: either remove the outliers or build your own decision tree algorithm that makes splits based on the median instead of the average, as the median is not affected by outliers. However, basing the tree algorithm on the median will be very slow, as computing the median is way slower than computing the average.


Spectral clustering is an algorithm for data clustering that uses the spectrum (eigenvalues) of the similarity matrix of the data to perform dimensionality reduction before clustering in fewer dimensions.

The similarity matrix includes the relative similarity of each pair of points in the available dataset.


Fractions or probabilities as training labels

Cross-entropy loss still works with probabilities in $[0,1]$ as well as $\{0,1\}$. Most importantly, $\hat{y} = y$ is still a stationary point (although it will not equal $0$).

It is also the case that the possible improvement in loss (and immediate gradient) is larger for $\hat{y} = 0.99, y = 0.89$ than for $\hat{y} = 0.4, y = 0.5$. If you use a sigmoid output, then the gradient at the logit $\hat{y} - y$ still applies - the larger gradient of the loss function scales inversely to the lower gradient of the sigmoid at that point.

So, in short, yes use binary cross-entropy loss for single-class probabilities, even when they are not strictly in $\{0,1\}$.


What does the below phrase in the lstm blog mean? - Data Science

I think, with "series", the authors refer to time series data in their dataset. In this case, it should be x, y, z, angular velocity and maybe other features for each participant.

"Splitting data into fixed windows of 2.56 seconds (128 data points) with 50% overlap." I would translate to:
Every 2.56 seconds aggregate each series of the data over the next 5.12 seconds.

You can imagine windowing basically as a timestamp-based data aggregation technique that is often used for stream data processing to reduce data and fix granularity of the data. The stream analytics windowing function documentation contains visualizations, which should help you to understand what is meant. I think, in the blog, the authors mean what Microsoft calls "hopping windows".

I read that windowing is a time series feature engineering topic. So maybe, it might help, if you learn more about feature engineering.


the ratio of training set, validation set and testing set in K-fold cross validation

For K-fold cross validation, you would not need a separate validation set. In your example of 100 samples, if you have train:test set ratio to be as 80:20 and follow the 5-fold CV algorithm, the train set (80 samples) will be split into five 16 sample subsets ($s_i$) as in $\{s_1, s_2, s_3, s_4, s_5\}$. At each iteration, the model will select any 4 subsets out of the 5 subsets, based on your selection criteria, and use the left-out set as the validation set.
Once the training is completed, you can test the performance of the final model with the held-out test set (20 samples) to get an unbiased estimate of the performance of the model.
It is not necessary to have same number of samples in validation set and test set.


How to further Interpret Variable Importance?

The documentation offers a couple options. To plot the individual trees in your forest, one can access them like model.estimators_[n].tree_ and then plot them with export_graphviz as explained in the documentation, or you can follow this example that directly prints the structure in text format.

However, I would say this is not the best idea, because a feature can occur in different trees and nodes with different split points. You probably get a better intuition about your features from partial dependence plots that try to isolate the effect of one variable on your response variable.

As a bonus, here is a good article about more alternatives to gain insight into your model (not all applicable to random forests).


Attention Mechanism: Why use context vector instead of attention weights?

The attention weights themselves do not carry any information about the encoded sentence. It only tells you something like: the $n$-th and $m$-th word carry important information for generating the next word, but not what the words actually are. Moreover, it is a variable-length vector (of the same length as the input size), it would be hard to do anything with it in the decoder.


Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred?

It could be partially due to the number of topics you selected but the fact that two words rank high for a given topic doesn't necessarily mean that the two words will frequently occur in the same document. Consider a topic that generally corresponds to "medicine". You could have a number of medical documents associated with heart disease and a number of medical documents associated with injuries. They may all be linked by common terms such as "diagnose", "treatment", "patient", etc. But you wouldn't necessarily expect that words like "heart" and "aorta" would appear in documents containing words such as "sprain" and "fracture", even though they are all associated with the general topic of medicine. If one were to choose a larger number of topics, those words might be separated into more specific topics (e.g., "heart disease" and "sports injuries").


There are no rules to infer neural network hyper-parameters from a problem description. With only number of features, number of examples and the fact you have a binary classification problem, it is far too little to even make an educated guess.

First, are you sure you are ready to build a deep learning model for your data? Have you looked at the data, or scatter plotted it reduced to two dimensions using PCA or t-SNE, to get a feel for how easy your data is to separate (easy to separate raw data implies using simpler/shallower model)? You could try a basic model such as logistic regression or SVM in order to establish a benchmark so you can tell whether the deep model was doing anything useful.

Assuming you are ready to go ahead, the time-honoured approach is to try out variations, and measure the results using cross-validation. You can do this methodically, by for instance starting with one hidden layer with 64 neurons, and either adding/removing layers or adding/removing neurons in each layer. Generally these searches don't cover all possible variations, but stop once you have tried those that seem interesting and have reached a reasonable result for your problem.

Do note that other hyper-parameters will affect results and can interact with your choices for numbers of neurons and layers. You cannot isolate your choice of network depth from other choices, such as optimisation method, activation functions, regularisation. When starting to build a model, it is just as reasonable to spend a long time exploring these other factors as it is to look at network size/shape.

It is very easy to get a deep NN to over-fit your data. Cross-validation is therefore a necessity as you explore hyper-parameters for your problem. One reasonable method of searching against number of neurons is to increase them until you start to notice over-fitting, then adjust regularisation to stop the over-fit. At that point, with all other factors remaining the same, there is probably no need to explore networks with many more neurons (although deeper/shallower networks may still be worth exploring, and if you do so, you will once again want to explore number of neurons per layer).


How can I randomly sample the space of consistent neural networks for given data?

I am trying to formalize your question before discussing it.
If I understand correctly, you ask for the following:
For $X \subset \mathbb{R}^{n}$ and $Y \subset \mathbb{R}^m$, let $f:X \rightarrow Y$ be a map.
Let $w \in \mathbb{R}^q$ be weights. We consider a neural network $g: \mathbb{R}^{n}\times \mathbb{R}^q \rightarrow \mathbb{R}^m$, and let $g^{(w)}: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}, x \mapsto g(x,w)$ be the neural network parametrized by $w$.
Now you want to sample from the set $\underline{W}(f,g,X):= \{w \in \mathbb{R}^{q} \mid f = (g^{(w)})_{\mid X} \}$.
However, I think constructing $\underline{W}(f,g,X)$ is very difficult in general.
The following question arises:
Do you already have some $w \in \underline{W}(f,g,X)$ ?
If not, note that $\underline{W}(f,g,X) = \emptyset$ is possible! (its easy to construct an example for that)
Note also that all known universal approximation theorems have some requirements on $f$, and only state that $f$ can be approximated by some neural network. However, for fixed architecture, it might be that there is no $w \in \mathbb{R}^q$ with $f = (g^{(w)})_{\mid X}$ nor that $f$ can be approximated by $(g^{(w)})_{\mid X}$ (e.g. in terms of the uniform-norm).
If you have some $w \in \underline{W}(f,g,X)$, there are certains trivial permutations (e.g. permuting the nodes of a fully-connected layer, or some channels). Apart from that, I am not aware of a full description of $\underline{W}(f,g,X)$. And without further details or constraints, I think its there is no general answer at the moment.
I hope this helps!


Trouble understanding the partial differentiation used in reinforcement learning

Your understanding of what's going on seems to be correct, just one little clarification: $u$ should be the model parameters of the deterministic policy $\mu(s,u)$ and not a distribution itself, same as $w$ are the model parameters of $Q(s,a,w)$, but that's probably what you meant (or I might be unfamiliar with the formulation).

Concerning your actual question, the update step implied by $\frac{\partial l}{\partial u}$ is supposed to make the deterministic policy $\mu(s,u)$ to get closer to the optimal $a$, which maximizes $Q(s,a,w)$. As $a = \mu(s,u)$, we have a composite function on our hands

$$ Q(s,a,w) = Q(s, \mu(s, u), w)$$

When updating the actor paramters $u$ such that $Q$ is maximized, we need to make a step in the direction of the gradient of $Q$ with respect to $u$ which, since it is a composite function, is computed using the chain rule

$$ \frac{\partial Q}{\partial u} = \frac{\partial Q}{\partial a}\frac{\partial a}{\partial u}$$

The notation is a bit sloppy, replacing $\mu$ and $a$ and so on, but that also seems to be the case in the literature. So what is going on intuitively consits of two parts:

moving $a$ in the direction of $\frac{\partial Q}{\partial a}$ will increase $Q$, e.g. in 1D if $\frac{\partial Q}{\partial a} > 0$ increasing $a$ would increase $Q$ and if $\frac{\partial Q}{\partial a} < 0$ increasing $a$ would decrease $Q$
moving $u$ in the direction of $\frac{\partial a}{\partial u}$ will increase $a$, in 1D the example would be the same as above

If you multiply these together and update $u$ according to the product you end up moving $u$ such that $Q$ increases by either increasing or decreasing $a$, which is exactly what you want to do.


Can I treat text review analysis as a regression problem?

Yes, regression makes sense.

Many MLer working on similar tasks (e.g. the Yelp challenge) use classification instead of regression because they collapsed the label space to 2 (positive or negative) or 3 (adding neutral). For predicting numerical scores 1~5 regression makes more sense.


How to bypass ID column without being used in the training model but have it as output - Azure ML

You have two options to do this, one less elegant than the other:

You can drop whatever column you don't want to use and add them together after getting the scores. Though you might need to reorder columns etc.

Or...

There is a block called Edit Metadata which can be used to do what you need. Select whatever columns from the column selector you don't want to use and use the option Fields > Clear Feature from the block properties. These columns won't be used in the calculations but will be present when you visualise the Score Model output together with the rest of the columns and the scores. Moreover, columns in that output statistics will be labelled as Feature type if they were used in the calculations. So you straightaway know which columns were actually used in the calculations or not. Block help is quite useful here.


Prediction Algorithm for Data with high Randomness

You can use seller mean buying price, std buying price, max buying price, min buying price, median buying price PLUS include recently user buying power to suggest the totally new product to the user given the current data that's best I can recommend although extensive data can lead to better suggestions.


GradientChecking, can I blame float precision?

The fact that there is a sweet spot is a common issue in numerical differentiation. The issue is that using a high epsilon value will give you high error due to the fact that the numerical derivatives have error $O(\epsilon)$ (or $O(\epsilon^2)$ if you use centered difference), and using a very low epsilon will result in cancelling errors (see catastrophic cancellation question) due to the fact that you are substracting two numbers that are very close. It seems that $\epsilon = 0.01$ is a fair trade-off between these two extreme situations in your case.


Why margin-based ranking loss is reversed in these two papers?

In this paper, $d$ denotes "dissimilarity" which should be minimized for positive samples.

In this paper, $d$ ($g$ in the paper) denotes "similarity" which should be maximized for positive samples (or equivalently $-g\left(T^{(i)}\right)$ should be minimized)


convert some observations into variables

Assume your table can be put into a pandas DataFrame object data with 4 columns as above. One way to achieve what you want is to do a GROUPBY using ID and Location. Then gradually assign values to each row of new table:

    newdata = pd.DataFrame(columns=['ID', 'Location', 'Feat1', 'Feat2', 'Feat3', 'Feat4', 'Feat5', 'Feat6'])
    grouped = data.groupby(['ID', 'Location'])

    for index, (group_name, d) in enumerate(grouped):
        newdata.loc[index, 'ID'] = group_name[0]
        newdata.loc[index, 'Location'] = group_name[1]
        for feature, amount in zip(d['Feature'], d['amount']):
            newdata.loc[index, feature] = amount


How to create a training set and classify them as positive or negative

The only way to obtain a high-quality dataset in your specific domain is to do it manually. There exists no other method that can give you the sentiment labels for texts in arbitrary domains. If there would exists such a method, why would you even bother to create your own model.

You should probably find/hire people that can do this work for you. Adding the sentiment meta-data to the texts could be as easy as creating an additional column in excel. But you could also smoothen the process by creating a little app that shows the sentence and let the user decide what sentiment it is (e.g. by swiping to left or right). Find a bunch of people that are willing to classify 10 sentences a day and your dataset will grow steadily. Even better, apply a little gamification and give people points or a rank and give an award to the best performing user to boost their performance.

Keep in mind that if these texts are really domain-specific you probably need domain-experts to do the classification.

When having enough training data you could learn a (simple) that predicts the output. Then you could let the human classifiers focus on the sentences with low confidence, assuming that high confidence automatic classifications are correct.


How to calculate TPR and FPR for different threshold values for classification model?

To calculate TPR and FPR for different threshold values, you can follow the following steps:

First calculate prediction probability for each class instead of class prediction.

Sorting the testing cases based on the probability values of positive class (Assume binary classes are positive and negative class).

Then set the different cutoff/threshold values on probability scores and calculate $TPR= {TP \over (TP \ + \ FP)}$ and $FPR = {FP \over (FP \ + \ TN)}$ for each threshold value.


As far I know, no formula.

But have some approaches(can be guidelines), which will help you out in analyzing your requirements.

"training inputs", higher the training data, better the results we can expect.
"number of features", If the number of features are high, model performance can be decreased due to over fitting. We dont have any formula for optimal features identification. If I'm correct l1(lasso) regularization, will do it the same internally and optimize the number of features by setting coefficient values as 0.
Epoch, generally we observe consecutive losses of "n" epochs. If it's not reducing, then we can terminate further training and you best model so far we observed. If model is still not acceptable, then we have to retrain the model with certain changes like review  features selection process, hyper-parameter tuning, algorithm selection and so on.


What's the correct objective function for cosine similarity of two vectors to be 1 or 0?

There are a wide variety of loss metrics to compare two vectors (e.g., Triplet loss, Lifted Structure Loss, N-Pair loss, and Angular Loss).

Cosine distance (1-cosine similarity) can be used. Typically, a loss metrics needs to be minimized so distances metrics should be used.


Characterisitcs of the data set for a binary classification problem

Unfortunately you're not going to be able to do much without at least 200-300 records. You're going to be limited to simple (i.e. mostly linear) models until your dataset expands to at least 1,000. Anything less than 1,000 will require very thorough cross validation, and if you're not careful you'll be at risk of building a model that easily overfits.

@EricLecoutre makes a great point that you should use Amazon's Mechanical Turk. It usually costs just a penny or two per record and could save you a lot of time.


What does it mean to 'gradient check a few dimensions for every separate parameter'?

As you might know, biases are used in the CNN only when there is no batch normalization layer is included because this automatically adds a type of bias to the activation outputs.

So, this is what I understood:

Let us assume a layer of CNN (without batch norm) with 50 kernels each of size of 5 x 5. So, total weight updates will be 5 x 5 x 50 = 1250 and total biases updates will be 50 (a bias for each kernel).

For the part "...people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector..." :-

If you merge all these parameters into one single vector and randomly sample the parameters to check gradients for, you may end up checking very few or none of the gradient updates for biases as they make a very less portion of that vector (only 50 out of 1300).

For the part "One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter..." :-

On the other hand, you can choose some kernels and take all parameters (weights and biases) of only those kernels to check gradient for and so in this case you will definitely be checking gradients of some biases which they say is a good way of doing gradient checking.


Q Learning Neural network for tic tac toe Input implementation problem

One idea I have is that i have implemented the input neurons incorrectly.

There is no single correct way to implement the solution to this problem. There are less efficient and more efficient ways for specific problems.

  Use 18 input neurons and have the first 9 being the state before placing a move, and the next 9 being the state after placing a move.

This should work. The first set of 9 neurons will receive a representation of the current game state. The second set of 9 neurons will receive a representation of the intended action.

For predicting Q values from a state, action pair, this is a reasonable approach. Staying with the same architecture of 18 inputs, you could also have the second set of neurons be one-hot encoded to where the agent will to place the 'X' or 'O'.

In the game Tic Tac Toe however, you can look for more compact representations if you like. You have already noticed that because the game is simple and deterministic, that you can represent the agent's action as simply "desired next state". And in fact, the current state does not actually matter to a player, other than to enforce the rules of what are valid next states.

You won't be implementing the rules of the game into the agent - they are part of the environment. Therefore, you can do away with the initial state altogether in your estimate, and work with the end state of each move - this is called the afterstate in the RL literature, and you will find it used a fair bit in deterministic games, or even in non-deterministic games where the randomness happens before the action choice (e.g. in Backgammon). An afterstate representation is more efficient because it encodes the fact that you don't care what route a player took to get to certain board position, you just care about the value of that position as the game continues.

Having said all that, if your goal is to learn basic RL, then you don't need to be looking for the most efficient solution, just one that works. Don't expect your NN-based learner with state-action logic to be the most efficient learner however.

  Another question I have in terms of when you teach the neural network, would you feed it the old state and then the state after both you and the opponent have made a move?

Not as inputs to the network at the same time, no. In your (state, action [=next_state]) representation, your action representation should be the board state for the current player's move and before the other player takes any action. The resulting next state however, will be after the other player takes their action.

If you want to train two separate bots against each other, then each would see the current state, then it would choose an action, then it would either get the reward for winning, or the opponent would take a turn. If the opponent won, then the first bot should receive the (negative) reward. If the opponent's move was not final, then the first bot should see the state after the opponent's move and get to choose its next action.

Again, this is inefficient. For a win/draw/lose game like Tic Tac Toe, you don't need two separate agents, each with their own learning algorithm, in order to train through self-play. You can instead alter Q-Learning slightly to work with the minimax algorithm. In brief this means alternating between the agent selecting actions that maximise the expected reward (for player 1) or minimise it (for player 2).

However, like before, your 2 networks set up should be able to work, and is quite interesting dynamic - you could try different learning parameters, different NNs etc, and see which learns to win quickly (but don't forget starting player has an advantage for early random play, so you'd want to switch which learning algorithm was used for which player to get a fair assessment).

The difference again is in terms of efficiency - a single network inside a modified RL with minimax will typically learn faster that two separate networks.

I have implemented a simple tabular (i.e. no neural networks) Q-Learning based agent for Tic Tac Toe in Python. It uses afterstate representation and modified Q learning with minimax, and learns purely online through repeated self-play.


Why isn't leaky ReLU always preferable to ReLU given the zero gradient for x<0?

One reason that ReL Units have been introduced is to circumvent the problem of vanishing gradients of sigmoidal units at -1 and 1.

Another advantage of ReL Units is that they saturate at exactly 0 allowing for sparse representations, which can be helpful when hidden units are used as input for a classifier. The zero gradient can be problematic in cases where the unit never activates in a gradient based scenario when the unit is initially not activated.

This problem can be alleviated by using leaky ReL Units. On the other hand, leaky ReL Units don't have the ability to create a hard-zero sparse representation which can be useful in certain cases. So, there is a bit of a trade-off and, as in general with NN, it depends on the use cases when which unit performs better. In most cases, if the initial settings can make sure that the ReL Unit is activated (e.g. by setting the biases to small positive values) one would expect ReL and leaky Rel Units to perform very similarly.

Also, leaky RelU (if parametric) introduces another parameter (the slope for $x<0$) that needs to be learned during training and therefore adds more complexity/training time.


Cross Validation how to determine when to Early Stop?

First, I would like to emphasize, that cross-validation on itself does not give you any insights about overfitting. This would require comparing training and validation errors over the epochs. Typically you make such comparison with your eye and you can start with one train/validation split.

Question 1: By getting validation error N times, you develop a reasonable (whether it is very or not very good is a question) understanding of how your network will perform (= what error it will give) on the new unseen data.

Often you do cross validation as a part of grid search of hyper-parameters. Then averaging errors at step 6 is mainly for choosing the best hyper-parameters: you believe that the hyper-parameters are best if the corresponding network produces the smallest average validation error. Simultaneously this error is your estimation on what error the final model will give you on the new data.

If you want, you can proceed with your exploration and compare validation errors (for one and the same hyper-parameter set) with each other, calculate standard deviation in order to get further insights.

The concept "model generalizes well" is more related to the absence of overfitting. To make this conclusion, you need to compare train and validation errors. If they are close to each other, then the model generalizes well. This is not directly related to cross validation.

Question 2:  The only purpose to take the whole dataset is to train the model on more data. And more data is always good. On the other side, if you are happy with one of the models produced during cross-validation, you can take it.

Question 3: You can take the number of epochs as one of the parameters in grid-search of hyper-parameters. This search usually goes with cross-validation inside. When at step 7 you take the whole data set, you take more data. Thus, overfitting, if at all, at this stage can only be reduced. If it bothers you, that each chunk is small, replace K-fold cross validation with, for example, K times 50/50 train/test splits. And I would never worry about leave-one-out. It was developed for small (very small) datasets. For Neural Net to be good, you typically need large or very large dataset.


On Kaggle, LightGBM is indeed the "meta" base learner of almost all of the competitions that have structured datasets right now. This is mostly because of LightGBM's implementation; it doesn't do exact searches for optimal splits like XGBoost does in it's default setting (XGBoost now has this functionality as well but it's still not as fast as LightGBM) but rather through histogram approximations. The result is a slight decrease in predictive performance for a much larger speed increase in training. This means more opportunity for feature engineering/experimentation/model tuning (all of which are key to winning Kaggle competitions) which inevitably yields larger increases in predictive performance (despite using histogram approximations).

CatBoost is not used as much because on average, it it found to be much slower than LightGBM. That being said, CatBoost is different in its implementation of gradient boosting which at times can give slightly more accurate predictions, in particular if you have large amounts of categorical features. I have never used CatBoost and so I encourage you to read that paper. Regardless, because rapid experimentation is vital in Kaggle competitions, LightGBM tends to be the go to algorithm when first creating strong base learners.

In general, it is important to note that a large amount of approaches I've seen involve combining all three boosting algorithms in a model stack (i.e. ensembling). LightGBM, CatBoost, and XGBoost might be thrown together as three base learners and then combined via. a GLM or neural network. This is done to really squeeze out decimal places on the leaderboard and so I doubt there is any theoretical (or practical) justification for it besides competitions.


If you are famialiar with R, you can use this library: VennDiagram
Without glimpse of your data its hard to help more, but with this library you can build simple Diagrams like this :

Or something insane like this:

You can find boht examples in the library code examples.
Hope this helps!


ideal algorithms to demonstrate overfitting or underfitting

Another example you could use is the separation boundaries in decision trees classification problem. In the picture below, you can see that the training error continues to go down (lower the better) when max_depth increases, whereas the testing error is not as good. This is because the model has carved out specific pink areas (for 'x') as the separation boundary based on the training data. When these boundaries are applied to another set of data, they become poor separation boundaries. So these boundaries cannot be generalised to other testing data sets.


With using R, You could look at trees / randomforests. Since you have correlated variables, you could look into Projection pursuit classification trees (R package pptree). And there soon will be a ppforest package. But this is still under  development.  You could also combine randomforest with the package forestFloor to see the curvature of the randomforest and work from there.


I am not sure that it will answer the question exactly, but it appears that the glmnet package (R : glmnet, Python : glmnet) do something similar : enforcing sparsity trough iteration.
A small exemple of what you can get : coefficients as a function of the L1 norm. You can see the number of non-zero coefficients (degrees of freedom) evolving .


Yes, it is possible. You don't really need a new column, you could just convert your existing columns to their representative weights.

I would suggest for you to:

Convert your [0,1] to [-1,1];
Multiply by a float weight in the range [0.0, 1.0].

I suggest you change the range from [0, 1] to [-1,1] because of the following example where it is not possible to differentiate the two:

attribute_value = 0
weight = 1.0
final_value = attribute_value × weight = 0.0

attribute_value = 1
weight = 0.0
final_value = attribute_value × weight = 0.0

However, if you change the range, you get:

attribute_value = 0
attribute_value_new = -1
weight = 1.0
final_value = attribute_value_new × weight = -1.0

attribute_value = 1
attribute_value_new = 1
weight = 0.0
final_value = attribute_value_new × weight = 0.0

A full example would be:

X = [
    [0, 1, 0, 1],
    [0, 0, 1, 1]
]

X_transformed = [
    [-1,  1, -1, 1],
    [-1, -1,  1, 1]
]

column_weights = [1, 0, 0.5, 0.25]

weighted_X = [
    [-1, 0, -0.5, 0.25],
    [-1, 0,  0.5, 0.25]
]


Formulate multivariate multistep time series forcasting using traditional machine learning, NOT deep learning

You can always reduce the forecasting problem to a tabular regression problem and then apply any tabular regression algorithm you like. Here's a good explanation of how this works.


import matplotlib.pyplot as plt
import numpy as np
file = plt.figure()
plt.plot(np.arange(0,60+1), X)
plt.xlabel('t')
plt.ylabel('x(t)')
plt.title('title')
plt.show()
file.savefig("MyFile.pdf")


Find best machine learning for predicting category of products

This seems like an ideal problem for deep learning. Your outputs will be the categories and you can pick the 3 activations (categories) with the highest probabilities.

You will need to vectorize your description features, you can do this using one of these techniques.


Searching documents at scale: how to maintain cleaned documents?

Think of this problem as a pipeline of steps to automate and re-run, and not just the ML step at the end:

Read in raw documents.
Stem words, remove stop words.
Perform TF-IDF
Train model on cleaned up data.

Now when it comes time to score:

Load your saved model.
Repeat steps #2 and #3 from above.
Send the cleaned data into your model for a prediction/recommendation.

Here is an example in Python of something like that:

http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_text_feature_extraction.html


how much text data is required for a meaningful use of word2vec

As word2vec is a neural network, it benefits from very large datasets.  The Kaggle dataset is 50,000 reviews * ~5 sentences per review, so about a quarter million sentences.  As they note, they get approximately the same results using bag of words and word2vec.  One thing which is of note, since the review data comes from the internet, the sentences are much more loosely structured than what you would encounter in a corpus of newspapers, which typically go through grammatical review.  A great dataset for training word2vec on structured language is the wikipedia dataset:
https://en.wikipedia.org/wiki/Wikipedia:Database_download


How to explain get_weight with autoencoder in keras?

Does the layer contain two matrices, one for the actual weights and one for the biases?

There could be one bias value for each of the columns in your weight matrix, depending on how you built your model.


Depends on your programming skills. Here is the summary:

OpenCV is a great tool originally developed in C++ and after a while a Python interface was added. In industry (and also many academic research group) C++ is the popular language for Computer Vision as the nature of data needs efficiency in computation. So starting with python is an easy way to learn Computer Vision and OpenCV capabilities but when you are applying for a job in industry or academia, they most probably need you to get your hands dirty with C++.


For this answer, I have assumed that you prefer open source solutions to big data visualization. This assumption is based on budgetary details from your question. However, there is one exclusion to this - below I will add a reference to one commercial product, which I believe might be beneficial in your case (provided that you could afford that). I also assume that browser-based solutions are acceptable (I would even prefer them, unless you have specific contradictory requirements).

Naturally, the first candidate as a solution to your problem I would consider D3.js JavaScript library: http://d3js.org. However, despite flexibility and other benefits, I think that this solution is too low-level.

Therefore, I would recommend you to take a look at the following open source projects for big data visualization, which are powerful and flexible enough, but operate at a higher level of abstraction (some of them are based on D3.js foundation and sometimes are referred to as D3.js visualization stack).

Bokeh - Python-based interactive visualization library, which supports big data and streaming data: http://bokeh.pydata.org
Flot - JavaScript-based interactive visualization library, focused on jQuery: http://www.flotcharts.org
NodeBox - unique rapid data visualization system (not browser-based, but multi-language and multi-platform), based on generative design and visual functional programming: https://www.nodebox.net
Processing - complete software development system with its own programming language, libraries, plug-ins, etc., oriented to visual content: https://www.processing.org (allows executing Processing programs in a browser via http://processingjs.org)
Crossfilter - JavaScript-based interactive visualization library for big data by Square (very fast visualization of large multivariate data sets): http://square.github.io/crossfilter
bigvis - an R package for big data exploratory analysis (not a visualization library per se, but could be useful to process large data sets /aggregating, smoothing/ prior to visualization, using various R graphics options): https://github.com/hadley/bigvis
prefuse - Java-based interactive visualization library: http://prefuse.org
Lumify - big data integration, analysis and visualization platform (interesting feature: supports Semantic Web): http://lumify.io

Separately, I'd like to mention two open source big data analysis and visualization projects, focused on graph/network data (with some support for streaming data of that type): Cytoscape and Gephi. If you are interested in some other, more specific (maps support, etc.) or commercial (basic free tiers), projects and products, please see this awesome compilation, which I thoroughly curated to come up with the main list above and analyzed: http://blog.profitbricks.com/39-data-visualization-tools-for-big-data.

Finally, as I promised in the beginning, Zoomdata - a commercial product, which I thought you might want to take a look at: http://www.zoomdata.com. The reason I made an exclusion for it from my open source software compilation is due to its built-in support for big data platforms. In particular, Zoomdata provides data connectors for Cloudera Impala, Amazon Redshift, MongoDB, Spark and Hadoop, plus search engines, major database engines and streaming data.

Disclaimer: I have no affiliation with Zoomdata whatsoever - I was just impressed by their range of connectivity options (which might cost you dearly, but that's another aspect of this topic's analysis).


Advantages of one shot learning over image classification

In the case of face detection in particular it can get very use case specific:

1a) Let's say I am facebook, and I have a billion possible faces of users to tag.  With the one-shot approach, I am storing a representation of each new face as you suggest, I assume it is something like One-shot Learning with Memory-Augmented Neural Networks. Now I have a problem with a 'memory' that there are billion faces, and that is a lot of computation to do for lookup. The representations learned must increase in size w.r.t to the dataset in order to be able to distinguish between instances, so I get hit again.

1b) With Deepface, which uses the siamese architecture you mention I have the advantage that to classify a new user, I just have to run it on maybe 1k images of friends around the network where that image comes from. I do not have to retrain the network if a new human is found, since after the initial training, there is not such a problem with doing a comparison between images, as the network is doing a one-vs-one, not a one-vs-all query. The network is not learning classes, but representations and the differences between representations that are most indicative of two examples being different classes.

2a) Now let's say I am a modeling agency with 800 employees, and I have 1000 examples per person. In that case, the one shot learning approach is feasible, especially if  you have downloaded a pretrained network, and do the fine-tuning as you suggested. There is close to no cost for any projected growth (the one-shot learning paper trained on a 4million example dataset).

2b) In this case it may be feasible to finetune a network with just a final layer with 1000 logits (800 for employees, and a 200 for growth). This would be totally infeasible for the billion or even million classes (individual humans) - but still probably not advisable.

In short, it depends on how many examples per class you have and the number of classes you have. Many architectures out there only work when you have millions of images for relatively few classes (this applies to all the imagenet entries). Others are designed to learn from 10k examples or less, with 50 examples per class, and yet others are designed for not explicitly using the notion of classes, when even that becomes too big.

For a comprehensive review of face detection check out Deep Face Recognition: A Survey. New on arxiv 4/18/2018


How to deal with categorical feature of very high cardinality?

This is an old question. I am surprised that I don't see anyone mentioned Mean Encoding (a.k.a Target Encoding). It is very popular in supervised learning problems. Besides, I have seen people use frequency or the cdf of the frequency (to avoid noise generated by heavy-tailed pdf), and they achieved pretty good results with lightGBM. However, i cannot really explain why it works rigorously.


Column sum in SPSS (with filter and grouped by date)?

Normally such operations are done in Excel (click into an empty cell and type =Sum( select the column of which you want to calculate the cumulative sum and then close the bracket ). Finally, hit enter). Note that the sum is only a global metric. It will not be a local measure for each observation that you have.

If you want to get these sums in SPSS you can use the descriptive statistics or you can also use the following syntax CREATE CUMMULATIVESUM=CSUM(OLDVAR). This will generate the cumulative sum for your observations. The entry for your last observations will be the sum of all observations.


problems with installing tensorflow using anaconda

I can't comment because I'm new, so I put this as an answer but really should be a comment. I have come across this link that might help you:

https://towardsdatascience.com/tensorflow-gpu-installation-made-easy-use-conda-instead-of-pip-52e5249374bc

They say to use this command to install tensorflow with conda even on Ubuntu:

conda create --name tf_gpu tensorflow-gpu

Hope it helps!


Split time series by python or by keras?

Assuming they can both do what you want them for (can perform the correct splits for your needs), I would recommend simply using the one that fits best with your pipeline. E.g. the data is being fed straight into a Keras model, just do as much as possible in Keras -> use the TimeseriesGenerator. Otherwise, stick to Sci-kit Learn.

The both generate the splits, meaning they only actually create each split as you loop through the data. This saves memory.

Looking at the source code of the Keras variant, it doesn't seem to support running on the GPU if you combine e.g. with your Tensorflow graph - meaning there really isn't a big difference between the two, just the APIs/functionality.


How can I infer no target in a target classification problem based on deep learning?

What you are looking for is anomaly detection, or novelty detection.

This cannot always be solved by labeling some images as "no letter", since all kind of images having no letters may not be available for labeling, or maybe too costly to exhaustively label.

This link has some directions you can look at for a start.


Which is best FaceNet or dlib_face_recognition_resnet_model_v1?

In my experience with the natively LR, tinyface dataset, dlib's resnetv1 model failed to extract embeddings for a number of faces from images with high gaussian blur. However, FaceNet was able to vectorize those same poor-quality LR images.


error length > 1 and only the first element will be used?

I can't reproduce that error message, so I'm not sure what's going wrong.  That said, the typical way to filter a vector in R is to subset it with a conditional like so:

tst<-c("","foo", "bar", "", "baz")
tst[tst!=""]
[1] "foo" "bar" "baz"


How do I find the common values in two different dataframe by comparing different column names?

You can use pandas.merge() to get the common rows based on the columns. Try this:

df3 = pd.merge(df1, df2, how='inner', left_on='UniqueID', right_on='ID')

However, this will give you all the columns. After that you can use

df3.drop([column_names], inplace=True)

to drop the columns that are not required.


Welcome! I think you asked in the right place.

If you have familiarity with Python, you might look at the numpy and Pandas libraries.  Numpy implements fast array and matrix manipulations, and Pandas arranges numpy objects into tables.  Along with scipy, they make the basis of the Python numerical computing stack.

Without more detail it's difficult to share a code example that'll do what you want, but if you run into any issues feel free to ask.

Edit: if your dataset is REALLY big and won't fit in memory, there are ways around that too, just specify.


Is backpropogation used in convolutional neural networks?

Both Full Connected Neural Networks and Convolutional Neural Networks use backpropagation for training.

What you said is right, both are feed forward neural networks, which means that the connections in the neural network start from left (input) and move towards right (output). But, it is feed forward only during inference time. During training i.e. when you are trying to find the weights/bias of the neural network for the task at hand, you have to use backpropagation.

Training has both forward pass and backward pass.
Inference has only forward pass.

Also, when we say feed forward it means that there are no loops in the network. L1->L2->L3->L4 and so on. But, if you see recurrent nets, they have loops like L->L->L->L


My Keras bidirectional LSTM model is giving terrible predictions

I think you cannot use a bi-directional LSTM for prediction, because of the time dimension of the music. I mean the backwards layer has to predict the latest value first and only after predicting it sees the sequence which gives the context- This is like you watch a reversed movie and yo have to guess how the first frame looks like without knowing the rest of it.

Also, check out stylenet of the imanmalik model. The interpretation lstm which does the prediction is unidirectional. Only the genre net units are bidirectional, but they get the predicted value already as input.


The answer is it does not consider the sub folders. It assumes all the images available in the sub folders are belong to the same category. In this case all the non ship images are assumed they belong to category B.


Dynamic Regression Prediction with Flask(or sth. similar)

This seems more like an ETL job than an API one. Of course you can have both, but in your specific case I would design it as a series of tasks where each task you extract, transform and load data.
Airflow is an amazing tool for that: https://github.com/apache/airflow
In airflow, each ETL is defined as a DAG (directed acyclic graph) with a series of tasks that are customized by you.
So, for your specific case, you would create a DAG with following tasks:

Retrieve data, prep if necessary.
Make predictions
Save new dataset predictions.csv.

You can schedule it like a cron job to run at each 2 hours. I highly recommend you to look at.


For simplicity, I will talk about the classification problem, but regression follows the same rules.
You are given some data: part of it with known labels and others with unknown labels. The "classical" goal is to determine the missing labels based on known labels. The data is naturally split into two chunks:
(a) data without labels -- we call it a test set.
(b) data with labels -- in some textbooks, it is called the train set, but the train set is used for another thing as well (see below), so I would call it labeled data to avoid confusion.
You want to train a selected model on some data and apply the model to the test set. The data which is used for training is called the training set. So, a naive scheme is

Train the model on the train set
Predict the labels on the test set.

However, before making predictions on the test set, you need to measure how good the model is. We modify our approach to

Train the model on the train set
Verify that the model performs well
Predict the labels on the test set

To verify the model performance, we do the following: given a set called a validation set, we predict the labels on the validation set, and then check the accuracy. Note that the validation set must contain labeled data only. Why not to use the train set for validation? This approach is prone to overfitting:

deep enough random forest will predict each entry in the train set accurately
polynomial with a high enough degree will approximate all the points with zero error
KNN with k=1 will have each node guessed accurately on the train set because the closest node is itself.
etc.

To fix this issue, we make a train set and a validation set disjoint. Usually, we randomly split the labeled data into two chunks. Note that the test set contains another data.
In your question, you refer to the validation set as a test set. Some authors do this, but it is very misleading.
For increasing K-value in K-fold, we manipulate the bias-variance tradeoff. It is another concept, and you can read about it, for example, here: https://medium.com/@karenovna.ak/part-ii-evaluating-a-predictive-model-cross-validation-and-bias-and-variance-tradeoff-9874b836cd2e