Why aren't languages like C, C++ used for data analytics instead of R, Python?

Yes, you're correct -- it's that C and C++ are harder to use and are more burdened with boilerplate code that obfuscates your model building logic. When you build models, you have to iterate rapidly and frequently, often throwing away a lot of your code. Having to write boilerplate code each time substantially slows you down over the long run.  Using R's caret package or Python's scikit-learn library, I can train a model in just 5-10 lines of code.

Ecosystem also plays a big role. For example, Ruby is easy to use, but the community has never really seen a need for machine learning libraries to the extent that Python's community has. R is more widely used than Python (for stats and machine learning only) because of the strength of its ecosystem and its long history catering to that need.

It's worth pointing out that most of these R and Python libraries are written in low-level languages like C or Fortran for their speed. For example, I believe Google's TensorFlow is built with C, but to make things easier for end users, its API is in Python.


'group' in strictly lower case in the FOREACH is the thing you are looping/grouping over.

http://squarecog.wordpress.com/2010/05/11/group-operator-in-apache-pig/ says:

  When you group a relation, the result is a new relation with two
  columns: “group” and the name of the original relation.

Column names are case sensitive, so you have to use lower-case 'group' in your FOREACH.

'GROUP' in upper case is the grouping operator. You can't mix them. So don't do that.


Split a string and create a news columns with parents

I think you can achive the expected result by applying the following method. I've just create a fake dataframe to test.

import pandas as pd

df = pd.DataFrame(columns=["section"], data=[["level1.level2.level3.level4.level5"],["level1.level2.level3.level4.level5.level6"]])

columns = ['section', 'level', 'parent']

def split_parent(row):
    levels = row["section"].split(".")
    count_level = len(levels)
    sub_levels = '.'.join(levels[:(count_level-1)])

    #  create a new row for each level of section
    new_df = pd.DataFrame(columns=columns, data=[[row["section"], count_level, sub_levels]])
    for num_level in range(count_level-1, 0, -1):
        sec = ".".join(levels[:num_level])
        par = "" if num_level == 1 else ".".join(levels[:num_level-1])
        new_df = new_df.append({'section': sec, 'level': num_level, 'parent': par}, ignore_index=True)

    return new_df

result = pd.DataFrame(columns=columns)
for index, row in df.iterrows():
    result = result.append(split_parent(row), ignore_index=True)

print(result)

As result, you'll get:

                                      section level  \
0          level1.level2.level3.level4.level5     5
1                 level1.level2.level3.level4     4
2                        level1.level2.level3     3
3                               level1.level2     2
4                                      level1     1
5   level1.level2.level3.level4.level5.level6     6
6          level1.level2.level3.level4.level5     5
7                 level1.level2.level3.level4     4
8                        level1.level2.level3     3
9                               level1.level2     2
10                                     level1     1

                                parent
0          level1.level2.level3.level4
1                 level1.level2.level3
2                        level1.level2
3                               level1
4
5   level1.level2.level3.level4.level5
6          level1.level2.level3.level4
7                 level1.level2.level3
8                        level1.level2
9                               level1
10

If you need any further explanation, just write to me.

I hope I helped you.


What happens if at leaf node both classes have same number of samples?

This is an implementation detail, and I wouldn't necessarily rely on this behavior, but presently in sklearn, it will choose the "first" class.
The predict method calls for the probability prediction, then takes the argmax, which in case of ties takes the first one:
https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/tree/_classes.py#L403
https://numpy.org/doc/stable/reference/generated/numpy.argmax.html


Why is data science not yet widely applied to Law?

Welcome to the site and thanks for the great question! I recently led an NLP project that dealt with a lot of laws. While I have to obfuscate my actual work, here's a general view:

The laws themselves may not be the best source data. It would take a massively transformed recordset in order to make most laws actionable for modeling. I'm talking about big rooms, full of lawyers providing an annotated version of laws in order to create a recordset that can actually be useful
The above assumes that the laws have been digitized in some easy to digest format. That may not always be the case. In a lot of instances, you are referring back to classic OCR approaches as part of  your data prep and I don't know anyone that likes working with OCR :-)
The human-in-the-loop requirements are very high. So you have an algorithm, now what? That's not something you can just put out on Mechanical Turk for the layman to verify. You need more lawyers to help with the verification of your approach and correct mistakes that are happening
Finally, you must get very sophisticated with your embedding layers in how you create and apply them. That's not an easy thing to do and very processor intensive - a GPU is highly recommended and not a lot of grassroot efforts are going to have this processing power

Good luck!


How to improve accuracy of deep neural networks

You need much more data.

Deep NN shines when you have excessive amounts of data.

With only a little bit if data it can easily overfit. The big difference between training and test performance shows that your network is overfitting badly. This is likely also because your network model has too much capacity (variables, nodes) compared to the amount of training data. A smaller network (fewer nodes) may overfit less.


How does Implicit Quantile-Regression Network (IQN) differ from QR-DQN?

Would like to thank Will Dabney and Georg Ostrovski (two of the paper's several authors),  and Massimiliano Tomassoli (Simple-Machine-Learning blog owner) for helping me understand IQN.

To understand this algorithm, you will need to know:

TD Learning
Value Functions of Reinforcement Learning
What a DQN is.
What C51 is
What a Cummulative Distribution Function is (CDF)
What an Inverse CDF is.

Just like C51 or QR-DQN, IQN allows us to estimate an entire distribution of possible rewards if we take some action $a$.  However, IQN doesn't output a distribution like those did.  Instead, it outputs a single sample every time you ask it. It outputs that red dot you see on the right, of the bottom image.

The final output of IQN is 1 scalar value per action. In other words, just a single sample per action.  We re-run IQN several times to get more samples for the "CurrentState".

IQN takes input at 2 different stages: 1)  and 2)

First, the IQN takes a current state S (which is a vector), transforms it into another vector V (for example, of dimension 10).
now, we take any random scalar value called $\tau$ (drawn from a uniform [0,1] range), and feed that scalar value into function $\phi(\tau)$.  It gives us a vector H,  which has the same dimension as V
then, vectors V and H get combined via concatenation. In fact, we perform multiplication instead of concatenation if there won't be many forward layers further on.
At the end of this forward-pass, our IQN outputs an |A| dimensional vector,  containing samples of the action-distributions we are trying to estimate. |A| is the other way to say the "number of actions". Notice: because IQN outputs a vector, and not a matrix (like QR-DQN does), we only get 1 scalar sample per action when performing a forward pass from the current State.
To get more samples, we have to return back to step 2)  and re-run the forward pass with some another scalar $\tau$  once again sampled from [0,1] range. Or, of course, we can re-use our IQN, to run the needed number of such forward-passes in parallel, if hardware allows. No need to start all the way from step 1) because we only care about the end-layers where $\tau$ actually enters the system. "Any preceding layers" would be re-computed to the same values anyway.

When training, we evaluate "CurrentState+1" as well. We bootstrap our results from "CurrentState" towards "CurrentState+1" via usual TD-learning, just like a usual DQN would do.  Don't forget, we will have to re-run forward pass several times (as step 5. has mentioned) both for "CurrentState" and for "CurrentState+1" to get several samples of the current and target distributions. This gives us a fairly good idea about what those two distributions look like.

According to the paper, in most cases re-running the forward pass 8 times (thus getting 8 samples) suffices for approximating both the $Z_{\tau}$ and our target distribution $Z$ (the distribution of the best action for "CurrState+1").  So we perform 16 forward passes in total:  8 for  "CurrentState"  and 8 for "CurrentState+1" to get an idea about those two distributions.

Why does feeding a random $\tau$ into the network work?

You have to understand that a well-trained IQN network represents an Inverse CDF itself.  That is, given a request of some scalar 'amount' value (taken from a uniform range of 0 to 1), the Inverse CDF will output a value from the actual distribution, that sits at that amount.

Although distributions won't necessarily look like Bell-curves, have a look at this example:

Why do authors use cosine inside $\phi(\tau)$ ?  I still have no idea, but the Appendix section of the paper states it seems to work the best. Perhaps it works good with ReLU.

I also got confused by $w_{ij}$ or $b_j$. To me it seemed there is a matrix of weights and a vector of bias values. ...Do we input an entire vector of different $\tau$ values at once?

The answer is no. Notice, authors used $j$ only to remind that forward pass has to run several times.
In total, there is only i number of weights in $\phi(\tau)$ and 1 bias.

Therefore, we get some scalar $\tau$, and use it with all those weights.  Basically $\phi(\tau)$ resembles a simple fully-connected layer, which only accept 1 scalar input value at a time.  We will re-use this same layer for different $\tau$ scalar values.


Why do most GAN (Generative Adversarial Network) implementations have symmetric discriminator and generator architectures?

The main reason is that you want the Generator and the Discriminator to be equally powerful. The base intuition of GANs is that the two Networks can improve themselves through competition. If one is way better than the other, they will get stuck into some unwanted equilibrium in which one beats the other all, or almost all the times.

For this reason, they are usually chosen symmetric. It's the simplest and most effective way to make sure their competition will remain balanced.


How to build a symmetric similarity model on top of embeddings?

That's an interesting question ! Surprisingly, I never read specific scientific papers on this particular problem. Here you want to learn a symmetric function that transforms two input vectors into a scalar. In a more general setting, we can find in the literature some needs for permutation invariant functions (see the Set Transformer and the Aggregation Schemes for Graph Neural Network ). Basically, you have two options:

As you pointed out, you can take whatever non symmetric function, such as concatenation + feed forward network and train it with (x,y) and (y,x) pairs. But there might be some smarter way...
Design a permutation invariant architecture. Here your chance is that you only have 2 embeddings $e_i$ and $e_j$. Sadly, you say they can't be trusted for their mutual dot product. Then, I might suggest to apply some linear/nonlinear transformation to your embeddings i.e. define a trainable matrix $Q$, and compute the dot product in the new space $v_i = Qe_i$ and $v_j = Qe_j$. By training $Q$ with example pairs $(e_i,e_j)$, you'll find a suitable transformation of the embeddings such that the dot product $v_i \cdot v_j$ fit the true similarity. You can of course use a more complex transformation (multiple non linear transformations for example). The key points is that using the same parameters for transforming both embeddings ensure the symmetry.

Bonus points: I'm not sure of your question, but my proposition in (2) might naturally deal with transitivity and reflexivity.


Should I perform cross validation only on the training set?

Chapter 5 of "Introduction to Statistical Learning" covers CV and bootstrap. I strongly recommend to read this chapter, since sampling methods are extremely relevant in practice.

Cross validation (CV) usually means that you split some training dataset in k pieces in order to generate different train/validation sets. By doing so you can see how well a model learns (and is able to make predictions) on different samples of a training dataset.

During training and model tuning, your model should not see the test data! The idea is that you reserve a truly "exogenous" dataset (never used during training) to test how well your model does in the end. If you use your test set during training, information may leak from your test set to the model, and you cannot demonstrate exogenous validity of your model anymore (becasue information from the test set leaked to your model).

In the image below, you would make four different splits on your training set, where blue would represent training cases and orange the validation cases. You run the same regression/classification on each of the four blue parts, and obtain predictions on each of the four orange parts.

There are different strategies to deal with data characteristics. In order to balance groups etc, you can use a stratified sampling strategy. See an sklearn implementation here.

Since sampling strategies can be relevant, it is advisable to let some tool like sklearn do the CV splits. Most tools/methods come with some CV options if relevant.


Impose similar metric on segments to model

Given that in your data there is correlation between population type (good vs. bad) and target, your model may learn undesirable associations between both. Therefore, the population type is a confounding factor.

A natural tool to cope with scenarios with confounders it causal inference. You can find an overview of causal inference in Judea Pearl's work, either this article or his book. A terser introduction to causal inference can be found in Ferenc Huszár's blog including an entry for controlling confounders.

There are a few python packages providing causal inference functionality, such a Microsoft's dowhy or Causalinference.


How can the Adaboost technique be called an ensemble learning technique?

AdaBoost or Adaptive Boost is a boosting ensemble model which works by learning from it's previous mistakes, ie: misclassified data points.

We specify the number of decision trees to be generated while training and during each training step, it calculates the following :

The weighted error rate of the trained decision tree
The decision tree's weight in the ensemble = learning rate * log((1-e)/e)
Update weights of wrongly classified points

This process repeats until all the trees are trained. In the end, AdaBoost makes all of them "vote" by adding up the weight (of each tree) multiplying by the prediction (of each tree).

Therefore, this becomes an ensemble of multiple decision trees.


How to estimate real distance between two detected objects in an image?

Distance in an image is based on pixels. You will need to convert this distance to the actual real world metric like cm or inch. To make this happen, you will also need a reference object in your image whose real world dimensions are already known to you. Or you will need the distance between the object and camera. If you manage to get this distance using your camera lens, then you won't need any reference object. And then, it is very complex to get distances between objects placed in 3D space.
This is a great article that explains the aforementioned approaches in 2D space. It might be a good start for you.


DeepLearning: does it make sense to have more nodes in the initial layer than inputs--for tabular data

This might seem a little bit confusing, especially for newcomers. But the problem here is basically agreeing on some terminology.

Typical Neural Networks by definition have 3 types of layers, namely Input, Hidden and Output. Because the Input layer has no specific function other that passing the input values to the next layer, many times is not mentioned. This layer does not have any weights, biases or activations and in the vast majority of frameworks (such as Keras) you don't have to write it in you code. You only need to specify the size of that input layer with the argument input_shape=(x_size,) in the first hidden layer.

What you call in your example initial layer of 100 nodes is actually that first hidden layer. Which can be on any size. And your intuition is completely right: the number of nodes in the input layer of course has to match the size of the inputs.

Hope this clarifies.


What probability distribution would be more appropriate for monthly rate of going to the store?

I think you were right with your initial consideration of Poisson. I think that you can assume that the frequency with which people go shopping for a given good can be considered independent. Of course this assumption might not be perfect (like maybe you could argue that Person A buys item X every 3 days. However, if there is enough people I don't think this should matter too much).  Overall, sometimes I think if you need to use a distribution to approximate a process, you need to pick the best option and it seems like Poisson is best here.
Also, It can't be normal since its bounded (ie the lowest number of people that can go to a store is 0)


What methods are used to determine acceptable loss levels for autoencoders?

Acceptable loss levels depend on the specific domain, dataset, and model.

Loss levels can be compared within model (e.g., training loss by epoch) or across models (e.g., autoencoders with different levels of compression).


Calculating optimal number of topics for topic modeling (LDA)

LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing the number of topics, you can experiment by tuning hyper parameters like alpha and beta which will give you better distribution of topics.

The alpha controls the mixture of topics for any given document. Turn
it down and the documents will likely have less of a mixture of
topics. Turn it up and the documents will likely have more of a
mixture of topics.
The beta controls the distribution of words per topic. Turn it down and
the topics will likely have less words. Turn it up and the topics will
likely have more words.

The main purpose of lda is to find hidden meaning of corpus and find words which best describe that corpus.
To know more about coherence score you can refer this


Are there any interesting application of linear regression

The way that you have phrased this question makes it tough for people to answer without first offering you some background on linear regression (LR).  Its great that you are interested in learning some ML and LR is a great place to start.

Linear regression is really just finding a line (or plain or hyperplain) that maps a relationship between two or more variables or features.  The important requirement is that the target variable be a continuous numerical value.  So its less about finding a problem for which linear regression "works" and more about finding some data that interests you and playing with it using linear regression.

I suggest you download some open data in the realms that you have mentioned.  There is plenty of open data in every topic that you mentioned that contain continuous numerical values that you can predict using other features of the data.

stock trading
video games
astronomy
sports betting
flight time prediction

Also think about taking some sort of online MOOC as this will help you gain some footing in the subject.  Andrew Ng's Coursera on Machine Learning is highly recommended as a starting point and include some linear regression during the first portion and scikit-learn is a great Python based library.


'list' object has no attribute 'values' when we are using append in python

It is basically what the error message says. The problem is this:

y =y.values().astype(int)

y is a list and lists do not have a method values() (but dictionaries and DataFrames do).

If you would like to convert y to a list of integers you can use list comprehension:

y = [int(x) for x in y]

Or alternatively use map (but I'd prefer the list comprehension):

y = list(map(int, y))

Since this is actually a coding related question you might want to consider posting on https://stackoverflow.com next time.


As you mentioned, Orange is a data mining software developed by the University of Ljubljana. It can be used for developing and testing machine learning models as well as conducting exploratory data analysis and visualization.

One of the unique features that makes Orange "special" is its simplicity and ease of use. This is because components in Orange are drag and drop and widget based. This means instead of needing to write python code to import a data file and pass it through an algorithm, this can be done by connecting two widgets. Another feature is its specific widgets for things like Bioinformatics and Spectroscopy. It has built in modules for a wide range of fields which make it easy to work with various types of data from different applications.

Personally, I think some of the benefits of Orange include its rich visualizations, interactive models, and speed. I code mostly in Python but often use Orange to get a quick look at what the data looks like or cross validate a model's performance I am developing elsewhere very quickly. I also think its interactive KMeans and Gradient Descent widgets are great tools for understanding what is happening with these algorithms under the hood.

One of the drawbacks is that it is not enormously robust for working with large datasets. Datasets which may work well in Python may make Orange crash. This is why it is best suited for smaller projects, pedagogical purposes, or exploratory data analysis.

Below is an example snapshot of what an Orange workflow would look like:

Bottom line: I think every data scientist should have some familiarity with Orange to add more tools to their toolkit. Easy and quick to learn, worth the effort.


Neural Network Hidden Neuron Selection Strategy

A rule of thumb approach is:

start with a number of hidden neurons equal (or little higher) that the number of features.
In your case it would be 9. My suggestion is to start with 9*2 = 18 to cover a wider range of possibilities.
Be sure your test and validation sets are selected "fairly": a random selection and varying the seed some number of times to test different configurations would be ok.

In general, a number of neurons equal to the number of features will tend to make each hidden neuron try to learn that special thing that each feature is adding, so will could say it is "learning each feature" separately. Although this sounds good it might tend to overfitting.

Since your number of inputs and your dataset size is small its ok to start with a hidden layer size of the double (18) and start lowering down. When the training error and test error stabilize in a difference lower than a threshold then you could have found a better generalizing model.

Neural networks are very good at finding local optima by exploring deeply a solution from a starting point. However, the starting point it is also very important. If you are not getting a good generalization you might try to find good initial starting points with methods of Hybrid Neural Networks. A common one, for example, is using genetic algorithms to find an initial combination of weights and then start the neural from that point. Given that your search space would be better covered (in case your problem actually needs that).

As for every problem in machine learning is very important to clean your data before introducing it to the NN. Try to be very detailed in order to avoid the NN to learn things you already know. For example if you know how two features are correlated improve the input data by making this correlation explicit so less workload is given to the NN (that might actually get you in trouble).


The latent representation is the simplified model of your input data, for example, created by a neural network.
Considering an autoencoder, the central layer of this network (after training) will contain a simplified representation of the input data (i.e. summary of key features), which can be used to reconstruct the output.
If we take a dictionary definition of Latent: present and capable of emerging or developing but not now visible, obvious, active, or symptomatic, we can see how this describes the somewhat non-existence of the state, rather instead only a latent representation of the input data.
This image is a nice description. The latent representation is key features of the input data (here: the ears, nose, eyes of the animals.)

So yes, the latent representation is the sum of the latent features.


How to cluster texts by most relevant words

You can use these words with their weight as a vector representation of the document. The important point is to make all the documents vectors over the full vocabulary, so that any position $i$ in any vector always represents the same word $w_i$. This means that a vector should contain zeros in all the positions corresponding to a word which is not in the document.
Using these vectors you can indeed use k-means to cluster the documents. Of course the quality of the results depends on the data: if there are very few words in common, it cannot work very well.


Cannot underfit/overfit on the IRIS dataset

IRIS is a very small data set, with not that many data points. I have found that to play with a data set, MNIST does much better job, and you can easily do over fitting and under fitting there.


What are the reasons for drawing initial neural network weights from the Gaussian distribution?

The goal of initial weight distribution is get some amount of variance to allow learning to take place. Many distributions could work. Gaussian or uniform are the most commonly used.
Cauchy distribution would not be a useful choice for several reasons. The primary reason is the Cauchy distribution is not as common as Gaussian or uniform distribution, thus is not as well supported in programming environments. Another reason could be Cauchy distribution's variance is undefined.


Use Feature Constructor in Orange to extract number from string?

Just to highlight Adam's timesaving answer
int(''.join(filter(str.isdigit, "0"+str(inputColumn))))


Reformulating the maximal margin classifier optimization problem

If $(\beta, \beta_0)$ satisfies the inequality $(4.47)$, then for any positive $k$, $k>0$, $(k\beta, k\beta_0)$ would satisfies the inequality as well.

Also, $(\hat{\beta}, \hat{\beta}_0)=\left( \frac{\beta}{M\|\beta\|}, \frac{\beta_0}{M\|\beta\|} \right)$ satiesfies the inequlity as well since

$$y_i(x_i^T\hat{\beta}+\hat{\beta}_0)=\frac{y_i}{M\|\beta\|}(x_i^T\beta+\beta_0) \ge \frac{M\|\beta|}{M\|\beta\|}=1=M\|\hat{\beta}\|$$

Let's find $\hat{\beta}$ and $\hat{\beta}_0$ directly. Note that $\hat{\beta}$  satisfies the property that $\|\hat{\beta}\|=\frac1M.$

Hence we let $\|\hat{\beta}\|$ to be $\frac1M$, hence reducing the inequality to be

$$y_i(x_i^T\hat{\beta} + \hat{\beta}_0) \ge 1$$
$$\|\hat{\beta}\|=\frac1M$$

We want to maximize $M$, hence, we minimize $\frac1M$, which is equivalent to minimizing $\frac12\|\hat{\beta}\|^2$.


Difference between Data Engineer and Data Scientist

Data engineering is infrastructure work; maintaining "big data" pipelines from ingestion to output. Today you might be expected to know things like SQL, Hadoop, Spark, Docker, and AWS.

Data science is an umbrella term, so it can mean a lot of things, including data engineering. But it can also mean pure data analysis without any production work. It really depends on who's using the term; read the job description and ask the company for details.


How to create correlation matrix but with only part of the rows?

You can use a slice of the correlation matrix dataframe.

import pandas as pd
import seaborn as sns

df = pd.DataFrame({'A':[0,1,2,3,],'B':[4,5,6,7],'C':[8,9,10,11]})
sns.heatmap(df.iloc[2:3, 1:3], annot=True, cmap=sns.diverging_palette(220, 10, as_cmap=True),square=True, )


Why could my DDQN get significantly worse after beating the game repeatedly?

This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios.

If you trained a neural network to recognise cats and dogs and did the following:

Train it for many epochs on a full dataset until you got a high accuracy.
Continue to train it, but remove all the cat pictures.

Then in a relatively short space of time, the NN would start to lose accuracy. It would forget what a cat looks like. It would learn that its task was to switch the dog prediction as high as possible, just because on average everything in the training population was a dog.

Something very similar happens in your DQN experience replay memory. Once it gets good at a task, it may only experience success. Eventually, only successful examples are in its memory. The NN forgets what failure looks like (what the states are, and what it should predict for their values), and predicts high values for everything.

When something bad happens and the NNs high predicted value is completely wrong, the error can be high, and the NN may have incorrectly "linked" its state representations so that it cannot distinguish which parts of the feature space are the cause of this. This creates odd effects in terms of what it learns about values of all states. Often the NN will behave incorrectly for a few episodes but then re-learn optimal behaviour. But it is also possible that it completely breaks and never recovers.

There is lots of active research into catastrophic forgetting and I suggest you search that term to find out some of the many types of mitigation you could use.

For Cartpole, I found a very simple hack made the learning very stable. Simply keep aside some percentage of replay memory stocked with the initial poor performing random exploration. Reserving say 10% to this long term memory is enough to make learning in Cartpole rock solid, as the NN always has a few examples of what not to do. The idea unfortunately does not scale well to more complex environments, but it is a nice demonstration. For a more sophisticated look at similar solutions you could see the paper "The importance of experience replay database composition in deep reinforcement learning"


"Place coding" == "one-hot encoding"
This is the conventional way of encoding a category at the output of a classifier: for N categories, use an N-dimensional vector where the k-th component is "turned on" and all other turned off to encode the category k.
In general, the values for "on" and "off" do not need to be 1 and 0 (or +1 and -1). They could be anything.
The alternatives to "place code" are "coarse coding" and "distributed representation" of which "error-correcting codes" are a special case.
The classifier in the paper uses a form of distributed representation in which similar characters (like o, O, and 0) are represented by similar bit patterns.
That way, when the system is not sure if the character is o, O, or 0, it will produce an output bit pattern that still identifies the character as one of those 3 categories, and leave it up to the post-processing (e.g. a language model) to resolve the ambiguity.


I checked the exercise, there is an additional assumption that simplifies the question

  Exercise 3.2.3 : What is the largest number of k-shingles a document of n
  bytes can have? You may assume that the size of the alphabet is large enough
  that the number of possible strings of length k is at least as n.

Shingling is a common technique of representing documents as sets. k-shingle is said to be all the possible consecutive substring of length k found within a document. For example if the document is "The sky is blue and the sun is bright"
the k-shingles when k=3 is given below:

[1)"the sky is"    2)"sky is blue"     3)"is blue and"     4)"blue and the"
 5)"and the sun"   6)"the sun is"      7)"sun is bright"]

Now we can say that for a document of 9 words, the number of 3-shingles is 7. if you try multiple sentences you conclude that:

The number of k-shingles = the number of words - k+1

The question now is what is the largest number of possible words in an n-bytes document?

This depends on the smallest possible size of a word. In UTF-8 encoding each letter occupies 1 byte(8 bits). for example a word of 4 letter "Mars" occupies 4 bytes.

The smallest possible size of a word is 1 byte, that's when it consists of only 1 letter. Hence the largest number of words in n bytes document is  n . Now let's substitute it in the largest number of k-shingles

The number of k-shingles = n-k+1

At the end this answer is valid under the following assumptions:

UTF-8 encoding
Each word consists of 1 letter only
The size of the alphabet is large enough (>n) so that all sets are different


Discrete Values Matrix: Select Rows and Scattered Plot problem

I removed the paint widget. Works fine witout it. That was in the version i had the problem. The latest daily build i installed after worked fine with the color widget. Thanks.


How can I access dataset from Nasa website

It seems that there is no downloadable data (I clicked Show the complete Project library), but the project was completed in 2012, so you could ask the listed contact, Gary Jahns, explaining your motivation, to be sure: gary.c.jahns@nasa.gov


Any one read the over 100gb csv file and successfully concatenation?

Its too big file to handle by standard way. You could do it by chunk
for chunk in reader:
    chunk['col1']=chunk['col1']**2 #and so on

Or dump yours csv file to database.
number of rows
num=0
for chunk in reader:
    num+=1
num_of_rows = num*lines_of_chunk

#work around in bash and python
import subprocess
subprocess.check_output(["wc","-l", "file.csv"])


Why does TensorFlow convert my decoded image to float32 instead of uint8/16?

Types are changing to float due to tf.image.resize_images.

Convert them back to uint as follows:

train_images = tf.cast(train_images, dtype=tf.uint8)
train_labels = tf.cast(train_labels, dtype=tf.uint8)

Output:

(tf.uint8, tf.uint8)

Versions of my code:

tensorflow version: 1.14.1-dev20190330
numpy version: 1.16.2


How do two perceptrons produce different linear decision boundaries?

In the article which you linked, it seems that none of the two perceptrons are trained on data, you just have two perceptrons with different weights w0 and w1. If they have different weights, then they will have two different decision boundaries.
You can see that, looking at the code that plots the decision boundaries:

ax.plot(t0, decision_boundary(w0, t0), 'm', label='Perceptron #0 decision boundary')
ax.plot(t1, decision_boundary(w1, t1), 'g', label='Perceptron #1 decision boundary')

where decision boundaries is

def decision_boundary(weights, x0):
    return -1. * weights[0]/weights[1] * x0


I would try a semi-supervised learning technique where it passes you scraps and asks you to label them. What you're looking for will likely be kind of domain specific depending on the type of site. In the end you'll probably have a bunch of heuristics like:

If length < 50 and contains "LOGOUT", "REGISTER","SIGN IN","LOGIN"
If count of "|" > 1
If count of all upper case words > 1


best approach to embed random length sequences of words as a fixed size vector without having a maximum length?

May be Universal Sentence Encoder might work for you if it can encode words in language you want.
https://tfhub.dev/google/universal-sentence-encoder/4


This depends on what you want to show.
When working with metrics you shouldn't just take the value as is, but see what each metric are telling you. baseline_1 isn't better/worse than baseline_0 because it has a higher/lower value in metric X. Both baselines give an interesting perspective on a given dataset and if unsure I'd suggest keeping both.
A couple of notes:

when saying baseline, I will refer to the two baseline strategies that you said in your post
I will use the accuracy metric for examples but what I'm saying is true for any metric.

Why use baselines?
People usually tend to see accuracy (or other measures) as absolute values. E.g. accuracy=0.9? "very good", accuracy=0.3? "very bad". This isn't true however, as metrics are influenced by the number of classes and the proportion of samples between them.
However an accuracy of 0.3 in a classification task with 1000 classes is arguably much harder to achieve than an accuracy of 0.9 on a binary classification task (assuming class balance in both cases).
Here is were baselines come in. They can show how much better a model is than a dump classification strategy.
How baselines help?
Baselines help by putting a lower bound on your metrics. For example an accuracy of 0.55 on a binary classification task is slightly better than random, but the same accuracy on a 10-class setting is much better. Baselines help quantify that and tell you how much better you are than predicting random  or the most common values.
What effect do baselines have?
Now on to why keep both baselines:

The first baseline (i.e. random) helps show how metrics can be influenced by the number of classes has on the dataset.
The second baseline (i.e. most common) helps show how metrics can be influenced by class imbalance.

How baselines actually help?
Let's you have two models, one with an accuracy of 0.92 and another with an accuracy of 0.93. How much better is the second model to the first? This depends on the value of your baseline. If you have a baseline accuracy of 0.5 then both models are relatively strong and the difference is not that significant. If you have a baseline of 0.9 then the models aren't as strong and an improvement of that magnitude is more significant.


How to use Cross Entropy loss in pytorch for binary prediction?

In Pytorch you can use cross-entropy loss for a binary classification task. You need to make sure to have two neurons in the final layer of the model. Make sure that you do not add a softmax function.

Use the below for resources:

https://discuss.pytorch.org/t/do-i-need-to-use-softmax-before-nn-crossentropyloss/16739

https://discuss.pytorch.org/t/why-does-crossentropyloss-include-the-softmax-function/4420


Random forest implementation with probability of choosing column or guarantee of choosing set of columns

Ranger in R can do the first request. The always.split.variables argument defines which columns should always be included.


NGBoost and overfit - which model is used?

No, I'm afraid you won't get the best model if you don't ask for it specifically.
But don't despair - just set the fit parameter of early_stopping_rounds to a number - and it will stop after this number of rounds in which the validation loss got worse.
The number of rounds that would work best for you should be set experimentally (i.e., just fiddle around fit till you get best results, but don't forget hyperparameter overfit...)
PS.
The remarks in the code claim that this number of rounds is consecutive, but looking a the code it does not look like the count is reset each time validation loss improves (the counter val_loss_list is not reset).
You could change this manually be editing ngboost.py.


Does anybody know where this rule of thumb came from? Rule of them: embedding vector dimension should be the 4th root of the number of categories

google has published a word embedding that has embedding dimension of 300. Following the rule you have given it should have trained on $300^4 = 8.1*10^9$ words. If google is using ngrams instead of just words, then it seems plausible.


Reducing the GPU memory usage when the model is already small enough

I cannot answer regarding the gap. What I can help you is how to reduce the memory usage.

One thing you can try is you can convert your weights to fp16 and do inference with fp16. Don't worry you would not lose significant numerical precision especially since you are using that for deployment. In fact, this is how people do forward pass on mixed precision training. Theoretically this could increase speed and reduce memory usage. Hope it helps.


How to do feature selection after using pre-trained word embeddings?

When using a RNN, you don't feed all the data at once, you usually have a seq2seq model.
The models are created with an encoder-decoder architecture. The LSTM is used in the encoding phase.

So, let's say you have a text of 78 words. You will feed the embedding vector (size 300) of those 78 words, 1-by-1 into your LSTM and in the end you will get a hidden vector which represents your sentence.

Then, you can take this hidden vector and use it for classification (with a feedforward neural network, for example).

So, it doesn't matter that you have 4293 unique words in your data. You need to feed your LSTM a sequence of size [<number of words in sentence> × 300].


How to predict numerical value "y"? if I have datasets having different number of parameters

If you have to predict $y$ only given $a,b,c$, then as a first look just discard $d,e$ etc from other data sets*. Any kind of regression might be useful, you will have to explore that - you might as well start with linear regression as it is simplest.

In general, if you are not sure about which algorithm to apply for best results, you should set up an experiment to try multiple models:

Decide on what "best" means by choosing a test metric (mean square error is a common choice for regression problems).
Split your data into training, cross-validation and test sets.
Train different types of model on the training set:

Tune any hyper-parameters (of more complex models) by taking the best results from the cross-validation set.
Pick your best cross-validation result as your guess at the best model.

Finally get an unbiased estimate of its performance by using your test set.

* There might be ways to use $d,e$ values to refine some models, but if you are at the stage of asking whether or not to use linear regression, this is not worth exploring initially.


Non-Convex Constraints for Classification Problems

Notice that constraint  of the form of $g(x) \le c$ where $g$ is convex is convex.

To see this, consider $g(x_i) \le c$, then for $\lambda \in (0,1)$,

$$g(\lambda x_1 + (1-\lambda)x_2) \le \lambda g(x_1) + (1-\lambda) g(x_2) \le \lambda c + (1-\lambda) c = c$$

To make it non-convex, let $A$ be negative definite or indefinite.


Algorithm for user profiling without distinct profiles

It is not uncommon for machine learning models to employ thousands of features. I doubt your software / hardware repository contains significantly more than that so it is unlikely an issue.

Having said that, machine learning is useful if you need to generalize. You say that your input variables are going to be department, location and job_role. In such case you have three categorical and likely independent and always known features so I expect you can just look up someone with same department, role and location and create default hardware/software profile based on that. I do not see any value added by machine learning.


Synchronizing Multiple Cameras in Autonomous Driving

This paper from Texas Instruments pretty much outlines the entire problem and current state of the art.


I would first decide to bin the x-axis such that it can be plotted in groups. Thus if we want to for example group them into bins of width 5 then plot them next to each other we would do something like this

width_box = 5
low = -40
high = 40
width_graph = 1.5 # should be at most width_box/2 to not overlap

def sum_in_range(df, min_, max_):
    return df[(df['IC12']<max_) & (df['IC12']>=min_)].FRQ.sum()

indices = np.array(range(low, high, width_box))
sum1 = [sum_in_range(df, i, i+width_box) for i in indices]
sum2 = [sum_in_range(df_1, i, i+width_box) for i in indices]

fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(indices,sum1,width_graph,color='b',label='-Ymin')
ax.bar(indices+width_graph,sum2,width_graph,color='r',label='Ymax')
ax.set_xlabel('Test histogram')
plt.show()

Which gives us

You can play around with the width_graph parameter to change how wide the bars look, and you can change width_box to group up the x-axis in different ranges


how to evaluate feature quality for decision tree model

The main reasons for seeking an efficient feature selection are the machine learning algorithm get faster training, reduces the complexity of a model, facilitates interpretation and improves the accuracy of a model.

Look for Filter Methods , Wrapper Methods and Embedded Methods to learn more about your issue.

Filter methods are generally used as a preprocessing step. The selection of features is independent of any machine learning algorithms. Instead, features are selected on the basis of their scores in various statistical tests for their correlation with the outcome variable.  Here you have to look for Linear discriminant analysis, Pearson’s Correlation, Chi-Square.

Some common examples of wrapper methods are :

Forward Selection: Is an iterative method in which we start with having no feature in the model. In each iteration, we keep adding the feature which best improves our model till an addition of a new variable does not improve the performance of the model.

Backward Elimination: Here, we start with all the features and removes the least significant feature at each iteration which improves the performance of the model. We repeat this until no improvement is observed on removal of features.

Recursive Feature elimination: It is a greedy optimization algorithm which aims to find the best performing feature subset. It repeatedly creates models and keeps aside the best or the worst performing feature at each iteration. It constructs the next model with the left features until all the features are exhausted. It then ranks the features based on the order of their elimination.

Embedded methods combine the qualities’ of filter and wrapper methods. It’s implemented by algorithms that have their own built-in feature selection methods.

Some of the most popular examples of these methods are LASSO and RIDGE regression which have inbuilt penalization functions to reduce overfitting.

Other example of embedded methods that could fits you is Regularized trees.

follow the link with some of these algorithms in sklearn.

sklearn - Feature Selection

I hope this could help you to start.


Action Recognition for multiple objects and localization

So finding actions from videos happens to be a tricky task. I have no idea about temporal 3D convnet but in order to tackle a problem like this, I would couple the CNN layer on individual frames of video and then feed the frame timeline into another layer of LSTM in order to find the context of the video.

As the action being performed on the video covers anywhere from 10% to 90% of the frame, you can perform TestTimeAugmentation on the video in order to find the action with a higher confidence rate. Similar approach could be found in this video by Google.


Weighting of words in lexicon based sentiment analysis

Yes, you can. Not quite sure what else to add. Your formula can then look like:

$$ Score = \sum_{i}{f(salience_i, frequency_i, sentiment_i)}$$

Where $f$ is a function that weighs your sentiment score with the salience and frequency. Up to you to define how.

What if you don't know which $f$ to use?

Now, bear with me, this isn't something I've tried per se, but this could be an interesting approach. You could use a recurrent neural network and your input could be the salience, frequency, and sentiment score for each word. Not only will your RNN "create" (ideally) the best $f$ for your particular problem, but it will also use the sequential information of the words, which may even improve your results.


What possible data products can be built using this dataset

You might be able to find individuals that act as anchors.  That if Abe is here, then everybody else is more likely to stay longer. Interesting to know if it was always the same people who stayed or not.

There might also be some interesting patterns with groups that arrive, are present, or leave together.

As a business owner you could look for individuals who not just work shorter hours, but arrive at their own time, and leave with a group.


Distributional semantics models are based on representing an instance with the words that represent its meaning. Typically if one is interested in representing the meaning of a word $w_i$ then the words that represent its meaning are the ones which appear close to it, e.g. withing a window of N words preceding/following it.

Example where $w_i$ is the target word with a window +/- 2:

$w_1,w_2,...,w_{i-3},\mathbf{w_{i-2},w_{i-1}},w_{i},\mathbf{w_{i+1},w_{i+2}},w_{i+3},...,w_N$

So the "meaning" of $w_i$ in this instance would be represented by the four words in the context window: $w_{i-2},w_{i-1},w_{i+1},w_{i+2}$. By collecting occurrences of $w_i$ everywhere in the document one obtains a set of contexts that can be used in different ways. For example one could build a context vector $a_1,...a_{|V|}$ over the vocabulary $V$ where each cell $a_k$ contains the frequency of $w_k \in V$ in the context of the target word. By applying the same process with different words, each word can be represented with its context vector and then various operations can be made on these vectors: similarity measures, clustering, etc.

Generally clustering based on words semantics is done with specific methods such as Latent Semantics Analysis.

In theory the method can be applied to any unit (e.g. n-grams), but more complex units require more data, more memory and more computing power.


Explain to a layperson why text-to-speech (TTS) has made little progress?

Text-to-speech suffers from problems similar to the drivers of "uncanny valley" effects when viewing faces of humanoid robots and computer renderings of faces. Our ability to distinguish depth of meaning, emotional content and other subtle cues makes humans very sensitive to small details in audio containing spoken language. Note this is not quite the same as claiming text-to-speech actually has an "uncanny valley"; there doesn't appear to be much analysis of that claim either way around.

Existing text-to-speech systems have two basic generative processes: Concatenative models are essentially databases of phoneme samples that the system strings together. These suffer from lack of flexibility. Parametric models attempt to represent sound generation at a lower level to improve on this, but it is quite a challenge to create a model that includes all the fine detail that we notice. For instance, we notice many non-verbal cues including imperfections such as mouth noise, breathing etc.

There has been recent progress in this area by researchers running a very detailed generative model - one that literally generates sound, sample by sample, after consuming a large amount of training data.

This report on WaveNet by DeepMind team explains the different synthesis techniques used to date and shows off the capabilities of the new approach.

The process creates state-of-the-art results, and it is easy to hear the improvement from examples on the site. However, it is too computationally intensive to be used in real-time systems yet. Give it a few years to be refined, and it could be the basis of far better text-to-speech systems.

The above written in January 2017. In the intervening time (it is now October 2017) the DeepMind team have been working on the efficiency of their model, and it is now much faster and sounds even better. This is close to becoming a solved problem, albeit with some proprietary ownership. Give it a little while longer though and this breakthrough will allow real-time and natural-sounding parametric voice models in many applications.


The parse_date_time function from the lubridate package will do this. This function takes an argument orders which is like the tryFormats argument in 'as.Date', except the formats allowed are more generous and lubridate-like (without punctuation, delimiters, etc.). For example, you can set orders = c("dmy", "mdy").
Date <- c('22-04-2015', '4/8/2015')
parse_date_time(Date, orders = c("dmy", "mdy"))
"2015-04-22 UTC" "2015-08-04 UTC"


Why there is no exact picture of softmax activation function?

Softmax is a multivariable function, generally. You wouldn't take a softmax of a single variable just like you wouldn't take a maximum of a single variable. It's difficult to plot functions of more than 2 variables because our eyes see in 3 dimensions.

A sigmoid is a 1 dimensional function. In your picture, they are applying a sigmoid to every variable separately.


Making Inference from a Correlation Heatmap

Assuming you're using df.corr(), the results from a heatmap are Pearson correlation coefficients which can be thought of as "the explainability between two arrays."

An effect score closer to 0 translates to there being no relationship. A score closer to 1 or -1 is a positive or negative relationship. A perfect score of 1 is a direct correlation.

Additionally, I would caution against taking action using these values without testing the normalcy/distribution of your data.


How sensitive are feedforward Neural Network to weight changes?

you are asking for the condition I would say.
If you skim through the formulae, the german explanation is more
detailed.
In particular, the absolute condition at $x$ is defined as  $\kappa_{\text{abs}}:= \lim \sup _{\tilde{x} \rightarrow x} \frac{||f(x) - f(\tilde{x})||}{||x - \tilde{x}||}$, which means $\kappa_{\text{abs}} \geq 0 $ is the smallest number such that
there is a $\delta >0 $ so that all $\tilde{x}$ with $||\tilde{x}-x|| < \delta $ it holds that $|| f(\tilde{x})-f(x) || \leq \kappa_\text{abs} || \tilde{x} - x ||.$
The relative condition $\kappa_{\text{rel}} \geq 0 $ at $x$ is the smallest number such that there is a $\delta > 0 $ so that all $\tilde{x}$ with $||\tilde{x}-x|| < \delta $ satisfy:
$ \frac{|| f(\tilde x)-f(x) ||}{||f(x)||} \leq \kappa_\text{rel} \frac{|| \tilde{x} - x ||}{||x||}.$
The difference is thus, that the $\kappa_{\text{rel}}$ compares the relative change of the output with the relative change of the input.
In general, let $\tilde{f}$ denote an approximation of the real function $f$, $\tilde{x}$ the input with some noise, then there are 4 categories in numerics:

condition: Analyze $||f(x)-f(\tilde{x})||$
stability: Analyze $||\tilde{f}(x)-\tilde{f}(\tilde{x})||$
consistence: Analyze $||\tilde{f}(x)-f(x)||$
convergence: Analyze $||\tilde{f}(\tilde{x})-f(x)||$.

Your question is also related to adverserial attacks. You can have a look into the literature.
Note also that $\kappa_{\text{rel}} = \frac{||Df(x)||||x||}{||f(x)||}$, so that you could compute the condition number for any given neural network.


Can I fine-tune the BERT on a dissimilar/unrelated task?

The sentence "During pre-training, the model is trained on unlabeled data over different pre-training tasks." means that BERT was pre-trained on normal textual data on two tasks: masked language model (MLM) and next sentence prediction (NSP).  There were no other classification/tagging labels present in the data, as the MLM predicts the text itself and the NSP label is derived from the textual data itself. Both tasks were trained simultaneously from a single textual dataset that was prepared to feed the input text and the expected outputs for both tasks.
Therefore "different" here refers to the two pre-training tasks I mentioned: MLM and NSP.
When fine-tuning, you do not need to train again on the same sentence classification task, you just simply train it on the task you need. It is perfectly fine to fine-tune BERT on a sentence tagging task on your own dataset.


how to shuffle the data for model.fit with custom data generator?

The big issue here is that your generator yields after each file is loaded. This means that your batch size is always the number of examples stored in each file and that the training examples in each batch are always the same.

You can absolutely shuffle the order of the files, but that's just shuffling the batch order around and not shuffling data examples among batches, which is very important for the optimizer. (Each batch yields an estimate of the gradient of the loss function with respect to the model parameters given the particular batch data. The more combinations of examples you're able to give the network, the better you're able to estimate the gradient with respect to the training set.) There are lots of ways to try to fix this, the Tensorflow docs have tutorials for some of them. (You could load a random set of files and deliver batches that contain their data intermixed, then load more, etc)

Also, you have 75,000 files that appear to be (40, 40, 15). That means the whole dataset is (3 000 000, 40, 15) if all the files are the same size, which is ~14.4 GB if you're storing the data as np.float64.  That's big, but not ridiculous. You could probably get a significant chunk of that in memory at once, which is probably better.

Is this float data? If so, do you need that level of precision (np.float64, the default)? I'm guessing probably not.  If you can store it as np.float32 or even np.float16, you might be able to just get the whole thing in memory. If not, you could load it as bigger chunks (half the dataset at once, maybe) and intermix examples among them. Just something to think about.

EDIT:

Also, if you're going to concatenate them all, don't do it in a loop with a big array and iteratively concatenate the smaller arrays:

# NO

big_array = np.load("first_train_file.npy")

for train_file in rest_of_train_files:
    tmp_file = np.load(train_file)
    np.concat((big_array, tmp_file), axis=0)

# YES

list_of_little_arrays = []

for train_file in all_train_files:
    tmp_file = np.load(train_file)
    list_of_little_arrays.append(tmp_file)

all_data = np.concatenate(list_of_little_arrays)

You're going to have loads of memory problems using the first way. I don't know that that's what you tried or not, but just in case.


reduction of model accuracy while using PCA for a regression problem

When you do PCA to project your problem two a 'n' dimensional space, in your case 2 dimensions. So you are fitting a model with just two features.

If you print your variable 'explained_variance' you will see the percentage of variance respect to the total that you have.

Ideas to improve your model: search for hyperparameters, do target encoding instead of OHE, dont do PCA...

By the way Decision trees model don't need the data to be standarized.


Certainly. A part of topic modeling output is typically some vector for each document describing it as a distribution over the topics in your model. You can use these as continuous-valued predictors for a classifier trained with the labels you have. I've done this before, but, depending on the domain characteristics of your problem, it may not be the best approach, but it is certainly a testable one!


when x is a vector, derivative of vector diag(f'(x)) is formal notation?

You are partially right and partially wrong: $f'(\textbf{x})$ is a matrix, but $\text{diag}(f'(\textbf{x}))$ means taking the diagonal of that matrix and making a vector out of it.


Is it possible to compress a sequence of numbers through an autoencoder?

This is doable, but the caveat is that a neural network might not give you a better result than a lossless compression algorithm (depending on what you want to use the compressed version for).
The most sensible way to do it in my opinion is to have a function at the start that converts the coordinates into an image and a function at the end to convert the recovered image to coordinates again.
The way you would train it is to gather a large number of training examples, pass them as 16*16 tensors containing 1s and 0s to your autoencoder network, then use the reconstructed image to extract the coordinates where the value is 1.
There’s a good article/tutorial here to get you started.
If you’re determined not to explicitly give your network a 16*16 tensor you could pass your coordinates as an ordered set of two dimensional vectors and treat the problem as a sequence transduction task. The network would need to learn an embedding for each set of coordinates and then predict each embedding based on surrounding embeddings. I expect you won’t get much joy from a transformer model as they are notoriously hard to train, but a recurrent model with attention like the ones in the article I’ve linked to might give interesting results.


There is a very nice library of online machine learning algorithms from a group at NTU, called LIBOL. This would be a very good place to start experimenting with the algorithms.
The accompanying user guide, and associated JMLR publication are very nice introductions to the basic algorithms in this field.
Avrim Blum has an older and more technical survey paper on online learning algorithms.


We if you're using sklearn's LogisticRegression, then it's the same order as the column names appear in the training data. see below code.

#Train with Logistic regression
from sklearn.linear_model import LogisticRegression
from sklearn import metrics

model = LogisticRegression()
model.fit(X_train,Y_train)
#Print model parameters - the names and coefficients are in same order
print(model.coef_)
print(X_train.columns)

You may also verify using another library as below

import statsmodels.api as sm
logit_model=sm.Logit(Y_train,X_train)
result=logit_model.fit()
print(result.summary2())


Weights and bias' relative to preprocessed X

If $$y = w_1 + w_0 \left(\frac{x_{}-\mu_1}{\sigma_1} \right),$$

then we have $$y = \left(w_1-\frac{\mu_1}{\sigma_1}\right)+ \left(\frac{w_0}{\sigma_1}\right)x$$


Differences between big data, data warehousing, business intelligence and data science?

The simplest way to explain:

Big data is essential the data itself. It is "big" because of the size and scale of it. You can think of this as the datasets that are composed up of hundreds of thousands or millions of observations. With more people online, more data is generated and this is what people mean by big data.
Data warehousing is the storage of all of these massive amounts of data. You can think of this as the place where big data is stored. This can be on large servers or large databases or even in the cloud.
Business intelligence is the use of big data (or small data) to draw insights that could be useful for making business decisions or applications. This is essentially leveraging the data to help improve aspects of a company.
Data science is a very broad term that can encompass topics like ML, data analysis, Artificial Intelligence, statistics etc. There is no one specific definition but more generally can mean anything having to do with working with data these days.


Does Sklean's SGDClassifier automatically standardize the training data when regularization is turned on?

No, sklearn generally doesn't apply scaling inside of any of its models, instead relying on the user to do that.  This seems like the right way to do it, since you might want to try different scaling techniques depending on your data.

From the User Guide:

  Stochastic Gradient Descent is sensitive to feature scaling, so it is highly recommended to scale your data. For example, scale each attribute on the input vector X to [0,1] or [-1,+1], or standardize it to have mean 0 and variance 1...


How can I output tokens from MWE Tokenizer?

You can check the exact input and output parameters of the add_mwe method in NLTK's documentation for the class here.

This is the expected input:

>>> tokenizer.add_mwe(('in', 'spite', 'of'))

So, each phrase must simply be a tuple with the words in that phrase. If you provide that input, you should get the output you expect (in_spite_of). I've added a full snippet of working code below for convenience, there you can see how to use the class as intended.

Regarding the output of add_mwe, every time you call the method it adds a new word to the dictionary,  and all the words are stored in the class's _mwes attribute. So, given mwe = MWETokenizer(), you can then inspect the contents of mwe (with e.g. print mwe._mwes) to see what the class actually stores.

As stated in the documentation, it is actually a Trie with all the terms, so it won't look exactly as the words you added (it is a more efficient representation thereof). The link I mentioned earlier has more details on that.

Hope this helps!

import nltk

from nltk import (
    sent_tokenize as splitter,
    wordpunct_tokenize as tokenizer
)

from nltk.tokenize.mwe import MWETokenizer

test = """Anyone know how to output the tokens produced using MWE Tokenizer?

For a clearer explanation of what I am asking for those who did not understand my original brief question.

The multi-word expression tokenizer (MWETokenizer) provides a method/function (add_mwe()) that allows the user to enter multiple word expressions prior to using the tokenizer on text. Currently I have a file consisting of phrases / multi-word expression I want to use with the tokenizer. My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text. So this leads me to ask if anyone knows how to output the token generated by this method/function so that I can verify that I am correctly passing the phrase to the function (add_mwe()).?"""

mwe = MWETokenizer()

phrases = [
    ('multi', '-', 'word'),
    ('expression', 'tokenizer'),
    ('word', 'expressions'),
    ('multi', '-', 'word', 'expression')
]

for phrase in phrases:
    mwe.add_mwe(phrase)

for sent in splitter(test):
    tokens = tokenizer(sent)
    print ' '.join(tokens)
    print ' '.join(mwe.tokenize(tokens))
    print '---'

# Expected output:
#
# Anyone know how to output the tokens produced using MWE Tokenizer ?
# Anyone know how to output the tokens produced using MWE Tokenizer ?
# ---
# For a clearer explanation of what I am asking for those who did not understand my original brief question .
# For a clearer explanation of what I am asking for those who did not understand my original brief question .
# ---
# The multi - word expression tokenizer ( MWETokenizer ) provides a method / function ( add_mwe ()) that allows the user to enter multiple word expressions prior to using the tokenizer on text .
# The multi_-_word_expression tokenizer ( MWETokenizer ) provides a method / function ( add_mwe ()) that allows the user to enter multiple word_expressions prior to using the tokenizer on text .
# ---
# Currently I have a file consisting of phrases / multi - word expression I want to use with the tokenizer .
# Currently I have a file consisting of phrases / multi_-_word_expression I want to use with the tokenizer .
# ---
# ...


Is Minimax Linkage a Lance-Williams hierarchical clustering?

My take after a quick read of the references.
First of all, Lance and Williams's original paper mentions that their linear scheme works (and offers computational advantage) only for combinatorial strategies. Is minimax linkage such a combinatorial strategy? In other words, does it depend (linearly) on pair-wise distances? By the defintion of minimax distance it is clear that this is implausible.
It is like the difference between mean and median computation in statistics. Mean is linear while Median is non-linear. There is no linear combination that can compute median from mean (although in certain limited cases they can coincide).
Second, the authors do not mention the form (or values) of the $\alpha, \beta, \gamma$ parameters in a hypothetical Lance-Williams method for minimax linkage. But in any case they are constants and $\alpha(\cdot)$ can be constant or rational functions of the respective cluster sizes (per the original Lance-Williams reference).
$G_2$ might not be of the same cardinality in the two panels, but minimax linkage distance depends on radius of cluster not cardinality (unlike average or centroid linkage) and the two examples have same radius thus $\alpha(G_2)$ is same in both cases.
Another way to see that this is the case is a plausible variation of the proof where $G_2$ in both cases has both same radius and same cardinality but different configuration.

Maybe such a proof would make it more clear. But I will leave it at that, at this point.


How to fix inconsistent (variable spelling) categorical data and "fill in" missing data

I'm also new to Data Science, so my approach may be laughably naive.  But I work as a "terminology analyst" who prepares and maintains standardized vocabularies and code sets (which are meant to avoid the very problem you describe), so at least I'm describing things I've actually paid for.  In fact, when my program was first established, a slide presentation called "The State of the State File" was prepared describing a problem very similar to yours: The file used to populate the data entry pick list of states contained hundreds of different ways the 50 US states could be represented in text.

To answer your question: I would call the name of the problem "unstructured data" or "unstandardized data."  We call the process of fixing it "data standardization."  The outcome of such standardization is a "reference terminology," which is a list of standardized terms as well as synonyms (aka "search terms") for each standard term.  Sometimes alphanumeric codes are assigned to each standardized term, resulting in a "code set."

The approach my organization has taken to standardizing data is rudimentary and hardly qualifies as any sophisticated data science: we run a DISTINCT query to get all the different text strings used to represent concepts.  Then that list needs to be reviewed to see what different text strings represent the same thing.  If your list is too large to review each text string in the amount of time you have, then you need to get frequency data for the number of times each distinct text string occurs. Sort the list of terms from highest to lowest number of occurrences, and work your way down the list as far as you can in the time alotted.

Sounds simplistic, but sometimes you need to make judgement calls about equality or subsumption.  For example, if one of your text strings is "the continental united states" (excluding Alaska and Hawaii) do you lump that under "USA" or does it get its own category?

Your result would be a reference terminology of a finite number of standardized country names, and all the synonyms for each country.

One place to start is ISO-3166, a set of codes "for the names of countries, dependent territories, special areas of geographical interest, and their principal subdivisions (e.g., provinces or states)."  You may be able to match each of your text strings to one of these established codes.


k-Nearest Neighbours with time series data - how to obtain whole-time-period estimators

Do you have more numeric features? KNN works off distances (euclidean, L1, etc.) which requires numeric data. It doesn't take in categorical data. Your activity feature will not work with KNN and even converting the date to say an integer will just create a time series which will not perform well with KNN. I think your approaching this wrong. Try logistic regression. It will take in both categorical and numeric features.

  There is no particular value in keeping Team and Shift separate, so I should create a joined column which combines both.

I would build two separate models for team and shift. You'll get higher accuracy because your using the data to predict less classes and letting the models specialize on each task.

Overall I would use logistic regression not KNN. It's more appropriate for your problem. Hope this helps.


Accuracy is lower than f1-score for imbalanced data

I'll try to answer this with a couple examples:
Say we have 100 instances (55 negative, 45 positive). Let's say we predict 1/45 positives and 55/55 negatives correctly. Then our accuracy is 0.56 but our F1 score is 0.0435.
Now suppose we predict everything as positive: we get an accuracy of 0.45 and an F1 score of 0.6207.
Therefore, accuracy does not have to be greater than F1 score.
Because the F1 score is the harmonic mean of precision and recall, intuition can be somewhat difficult. I think it is much easier to grasp the equivalent Dice coefficient.
As a side-note, the F1 score is inherently skewed because it does not account for true negatives. It is also dependent on the high-level classification of "positive" and "negative", so it is also relatively arbitrary. That's why other metrics such as Matthew's Correlation Coefficient are better.


Relu does have 0 gradient by definition, then why gradient vanish is not a problem for x < 0?

You're mostly correct!  ReLU does have a problem with the gradient vanishing, but only on one side, so we call it something else: the 'dying ReLU problem'.  See this stack overflow response for more information: What is the "dying ReLU" problem in neural networks?

It's a small semantic difference.  Lots of functions (tanh and logistic/sigmoid) have derivatives very close to zero when you're outside the standard operating range.  This is the 'vanishing gradient' issue.  The worse you get, the harder it is to get back into the good zone.  ReLU doesn't get worse the farther you are in the positive direction, so no vanishing gradient problem (on that side).  This asymmetry might be enough to justify calling it something different, but the ideas are quite similar.


Is it necessary to perform rolling-window on LSTMs?

I cant either feed the network a sequence of the 7 first values with the 8th as the target:

For typical LSTM, if your input sequence is [a,b,c,d,e,f,g] then your target sequence will be [b, c,d,e,f, g, h]. You will train your network with a long sequence. And after training, you can simply pass in [h] as input to get the estimate for [i].

  Or I can do the classic rolling window, with a window size of, say, 2.

There is no point in using LSTM if your window size is 2. You will end with one input and one output. There is no longer term dependency to be learned in this setting.

You might use a fixed window approach if your individual sequence is very long. You can slice your series using the window approach. The benefit of doing this

Reduce the length of the sequence. LSTM will still have problem learning dependency over very long steps due to gradient vanishing at the forget gate.
Augment the dataset in the similar way you doing rotation and scaling for an image dataset.

Summary

Say you have a single sequence [a,b,c,d,e,f,g,h,i]. Using a sliding window of size 6, your single squence will be transformed into 4 sequences

[
    [a,b,c,d,e,f],
    [b,c,d,e,f,g],
    [c,d,e,f,g,h],
    [d,e,f,g,h,i],
]

And you input X will

[
    [a,b,c,d,e],
    [b,c,d,e,f],
    [c,d,e,f,g],
    [d,e,f,g,h],
]

And your corresponding target Y will look like

[
    [b,c,d,e,f],
    [c,d,e,f,g],
    [d,e,f,g,h],
    [e,f,g,h,i],
]


Is the GA R package the best Genetic Algorithm package?

For such questions, I like to go to the Task Views on CRAN, since the packages noted there are, to a degree, pre-vetted by the R community. I'd trust those a tiny bit more than just googling myself.

The Machine Learning Task View at CRAN says:

  Packages rgp
  and
  rgenoud
  offer optimization routines based on genetic algorithms. The package
  Rmalschains
  implements memetic algorithms with local search chains, which are a
  special type of evolutionary algorithms, combining a steady state
  genetic algorithm with local search for real-valued parameter
  optimization.


A Generative Adversarial Network (GAN) takes the idea of using a generator model to generate fake examples and discrimator model that tries to decide if the image it receives is a fake (i.e. from the generator) or a real sample. This was originally shown with relatively simple fully connected networks.

A Deep Convolution GAN (DCGAN) does something very similar, but specifically focusses on using Deep Convolutional networks in place of those fully-connected networks. Conv nets in general find areas of correlation within an image, that is, they look for spatial correlations. This means a DCGAN would likely be more fitting for image/video data, whereas the general idea of a GAN can be applied to wider domains, as the model specifics are left open to be addressed by individual model architectures.

The linked paper that proposed DCGANs specifically raises the topic of unsupervised-learning, and essentially wanted to marry the (at the time) recent success of conv nets with the new idea of GANs.

I also couldn't find any direct comparisons of when to use them, but there are plenty of articles that explain both models. This is a good place to start - after reading that, you could probably decide for yourself.

Regarding dimensions - I don't think the dimensions of your data would dictate which of the two variants to go for, other than of course influencing things that we always have to consider, such as training time, model complexity, capacity to learn and so on.


How to approach the numer.ai competition with anonymous scaled numerical predictors?

I've looked at the approach and I'd select K by trying a range, i.e. 5k, 10k, 15k etc and then exploring the range in which the best result falls, say the best is 15k then I might do 13, 14, 15, 16, 17 and so on.

So far I've not found any pre-processing to be effective.

Answering the comment:

I've tried using LogisticRegression, SVM, Neural Networks, RandomForests, Multinomial NB, Extra Trees. All except Neural Networks using the implementations in sklearn. PyBrain for the NN.


visually explication of dense layer of keras?

You can find more details about the arguments of Dense Layer in Keras Sequential model here.

Example

# as first layer in a sequential model:
model = Sequential()
model.add(Dense(32, input_shape=(16,)))
# now the model will take as input arrays of shape (*, 16)
# and output arrays of shape (*, 32)

# after the first layer, you don't need to specify
# the size of the input anymore:
model.add(Dense(32))

So basically 512 represents dimensionality of the output space of your model.


Here's one simple yet effective solution without using the deep learning algorithm.

Divide the problem into 2 parts:

Segmentation: You'll need to do some image processing like edge detection and segment all the objects in the given captcha and save the region of interest (OpenCV can help). Assign labels to the objects.
Similarity Measurement: Use some scale and rotational invariant feature descriptor algorithm to extract features from the objects, and a matcher to compare the objects for a similarity measure (check SIFT/SURF features).

Based on the similarity scores, make a decision.


What is the rationale for discretization of continuous features and when should it be done?

One reason to discretize continuous features is to improve signal-to-noise ratio.  Fitting a model to bins reduces the impact that small fluctuates in the data has on the model, often small fluctuates are just noise. Each bin "smooths" out the fluctuates/noises in sections of the data.


I think you made a mistake here:

i was able to achieve great results for my approach when i calculate the TFIDF and PCA first and then do the Test/Train split afterwards.

This evaluation was flawed due to data leakage: both the TFIDF and the PCA should be calculated using only the training set.
This is more or less the source of your issue with how to proceed in production: normally in production one should just apply the same process as with your test set, but here you were able to prepare your test set differently (and wrongly).

For the TFIDF step, the IDF should be computed only using the training set. Note that in the test set you might have to deal with out of vocabulary words.
For the PCA step, I'm not too sure how to proceed but I found this explanation.


Spam detection can be done with many different methods, the same goes for your task. They do share the similar idea of processing a given text and classifying it to be one of 2 classes (science/not-science or spam/not-spam).

What you first need to do is to turn the articles into a vector of constant size
(for example with Word2vec which takes as its input a text and produces a vector space).

Ones you have a vector representing each article, you can start training your classifier and feature extractor (these days they are trained together).

As for determining which machine learning approach to take, you can try first using an SVM, it will probably be good enough.

You can follow one of the following tutorials (there are many more), just replace their dataset with yours :

Email Spam Filtering: An Implementation with Python and Scikit-learn

Spam Classifier in Python from scratch


Handling categorical data with more over 100 unique classes

Even though you have 100 unique values - Do you see a pattern where perhaps 80 of those unique values appear only once or twice - and the remainder 20 occur much more often.
In that case - you may want to encode the 80 with a special "Other" value. By doing this you lose information - by not being able to distinguish between the 80 categories - but perhaps it doesn't matter to your problem.
The other option is - You can convert your categorical data into an integer (perhaps the ID approach you are talking about ?) and run a Decision Tree model (e.g. Random Forests) on it. My experience with Random Forests is that they work pretty well on such data.


What to use in setting up a Speech to Text engine in production?

This would be a gift to the world. I certainly hope someone is working on it even as we speak (ha ha). Let's both keep looking.


It is because of the kind of convolution you've used. It is a valid convolution. If you want the output to be $128$, set the convolution to be same. Consider that this is also applicable to the deep layers too. They also can have either of these convolutions.


It's very common to have some amount of errors or inconsistencies in a dataset. Sometimes these inconsistencies are not even errors, in some subjective tasks (e.g. translation), annotators may simply not agree on what is the best answer.

What to do with this kind of noise completely depends on the case at hand. If the noise caused by these errors represents a reasonably small proportion of the data, it can safely be ignored: in this case it's up to the learning algorithm to distinguish the relevant patterns from the noise. Otherwise there can be ad-hoc pre-processing implemented to clean up the data. In cases where the subjectivity of the annotator plays an important role, it's useful to have several annotators annotated the same data and check the inter-annotator agreement. This might in turn be used to filter out the least consensual instances, or aggregate the annotations in some way (e.g. majority voting).


Is there any implementation of Extended Isolation Forest algorithm in R/Python?

There is a package on Github called "Extended Isolation Forest for Anomaly Detection", I used it a couple months ago and it seemed to work. For how accurate or how buggy it is, I'm not sure but if anything seems off you can check the source code for errors in the implementation of the paper Extended Isolation Forest by Harir et.al.


When you import ResNet50 import in this way:

image_input=Input(shape=(512, 512, 3))
model = ResNet50(input_tensor=image_input, include_top=False)

After that, you can work with your desired image_size


In your case you have 3 options for training your model:

Make a class for each card number, so 13 classes.
Make a class for 1, 2, 3, king, queen, jack and one bucket class for "other", so 7 classes.
Make a class for 1, 2, 3, king, queen, and jack, so 6 classes. And whenever the model is not confident of any, assume it is "other".

Best would be option 1.
Option 1 would be the more difficult to get data for because you need labelled data for more classes, but that should not be a problem because it seems you already have it.
Option 1 allows your model to classify all classes, and let's you achieve the same as option 2 with post processing (i.e. if predicted_class in [4,5,6,7,8,9] then "other").
Option 2 could work, but the numbers in the "other" class are just as different from each other as to 1, 2 and 3. To be able to make this distinction, the first intuition is that a "larger" model would be needed in comparison to option 1. This approach would be handy if the "other" class is not similar to the other classes. For example, a model that classifies sports cars, family cars, clown cars and "other vehicles" (i.e. tractors, bikes and trucks).
Option 3 is best left for tasks that the "other" class is not defined, or too large to model. For example, a model that classifies if the playing cards are Uno, or Poker, or Blitz, or "any other game" card. In this case, there are too many "other" types of game cards for you to label and model.


Q learning for blackjack, reward function?

It depends on the goal of the project. Projects can be a spectrum from "pure" to "applied":

A completely pure project only give minimal information. In the case of the Blackjack reward signal, that would just provide the win, loss, or tie information to the agent at the end of each hand.

A more applied project could give more information to the agent before, during, or after each hand.

An applied project will most likely start training better, learn faster, and end training higher. However, an applied project is less likely to provide evidence about the specific capacities of the system.

For both pure and applied projects, the amount and type of reward signal is a hyperparameter that is best tuned through empirical experimentation.


Should the lambda for L1 norm regularizer inversely be proportional to the number of trainable weights?

Backpropagation doesn't handle Regularization like this i.e. if you are thinking "10 weights make penalty 100, so 100 weights will make penalty 1000.
So let's have a smaller $\lambda$

Backpropagation uses the partial differentiation of the Loss. Now the Loss has an extra $\Sigma$$w_i^2$ and so the derivative will have an extra piece which will be proportional to the weight(Derivative of $x^2$ = $2*x$).
It will be separate for every weight, so the number of weights will have no impact

The new equation will be -
$w_i$ = $w_i$ - (backprop stuff as before) - $\lambda$*$w_i$

$\lambda$ is just a knob. You also have any value proportional to $w_i$ e.g. 5% of $w$.


cannot use sklearn.naive_bayes MultinomialNB to predict from one attribute

If your problem is shaped as you stated, you might need to reconsider your Naive Bayes model. MultinomialNB does not seem to be appropriate for your case. You can rather use the Naive Bayes classifier that uses the gaussian distribution.

from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb.fit(X, y)
print(gnb.predict(X))
print(gnb.predict_proba(X))


Update machine learning model while retaining the prediction results exact for old data

The safest way to keep the same predictions for your older data is versioning your trained models. It is, keeping the generated model artifact (in python could be pickle file, h5 file, etc) to make sure that you can use it getting the same results as you say, and generating new models (so new artifacts) via retraining when new data come in.
The usual approach is deciding an evaluation metric for your ML model, and based on it, when new data come in, you retrain your models (the frequency for this is something to decide on your team) and when the metric value is better (enough) than the latest model version, you can promote this new model for use in production. Theere are quite goos frameworks for models versioning, as MLflow.
Keep in mind that transfer learning does not guarantee predictions on older data are going to be same, since a new model is trained based on another model, but a different model is eventually created.


Actually in these cases I usually plot the data based on features two by two. This approach is so much similar to watching covariance matrix. In this way you should see the new added features whether they have correlation, whether they change linearly with each of previous features. Suppose that you have already one feature for a classification task. Then you may want to add another feature. You have to plot data with each feature as one axis and investigate whether they have linear correlation or not. If they don't have any correlation or their correlation is near to zero, you should add the feature because that may help you, they provide a kind of knowledge that the previous features didn't provide. If a new feature has correlation with the previous feature, it means that adding that does not help you have a new knowledge or perception of that concept. Although there are debates here I prefer not to add correlated features because of computation complexity. That will be so time consuming.

For illustrating more, suppose you have a data-set A = {X1, X2, y} in which X1 and X2 are the feature and y is the label and all are binary values. Also suppose that the covariance matrix between these is as follows.

We may infer that we can ignore X1 in the process of classification without loss of accuracy, if e is equal to zero which means label y is independent of X1, since they are binary valued. We can ignore X1 in classification, regardless of other values.
We may infer that we can ignore either one of features in the process of classification without loss of accuracy, if |d| >> 0 which means there is high correlation between X1 and X2 and hence information redundancy. So we may ignore one of them without problem.
We may infer that we can not ignore neither X1 nor X2 in the process of classification, if d = 0 and none of e or f are zero. Then the feature are independent and both are influential on label. So we may not ignore any of them.


Use prediction as feature for a decision tree

Yes. Stacking is essentially feeding the predictions of the base learners to a meta learner. Sort of like a model of the models. Here's a good explanation of that.

Bagging,Boosting and Stacking


Predicting $0.5$ for all items in your case would also give you MSE of $0.25$. That is because independently of whether the true label is $0$ or $1$, the squared error for each example will be $0.5^2 = 0.25$

Your model is performing badly under an MSE measure, when such a simple model that does not take input data into account can get the same score.

However, it is debatable whether MSE gives you a useful metric here.


Should I create a separate column for each Id value in a feature column or can I use the feature column as it is?

You should not leave the column as it is since similar id values are not semantically related. Maybe a deep neural network could handle this, but linear models definitely do not.

There are two options, depending on the cardinality (the number of unique entries/categories) of the column.

If the cardinality is low you should one-hot-encode the series, one column for each series, like you propose. However, in your dataset, there are probably many series, and your data will end up having too many features. See this image (source), you need one column/feature for each category if you one-hot encode.

If the cardinality is high, you should find an encoding for the id column. One way would be using target statistics: Calculate the mean target (in your case revenue) a movie of a series makes for each series-category of the training set. Then replace the id column with a column of the target-means for the respective series. If there is a series in the test data, which is not in the training data, just use the overall target mean of the training set.
It is important for this approach that you never use target-statistics calculated from the test set, otherwise, your error estimate will be highly optimistic!
The rationale behind this approach is, that the model is allowed to learn things like "continuations of successful movies are often successful".


How can give weight to feature before PCA

PCA is unsupervised method for finding the most important components. I don't see a reason why you should want add a weight. If you know what features are important, why use PCA at all? Or perform PCA on the features where you are unsure about the importance.

Further, components are created in directions with highest variance and the importance is measured by eigenvalues. So I can imagine you can somehow increase the variance (like Jan van der Vegt proposed). But that's a sorcery with very questionable output.


Which is faster for imputing: R or Python?

I recommend you to use Python. With sklearn, you have many ways to perform algorithm multicore on your data.
Another reason to use Python: If you want to perform distributed computing you have many modules like DASK which are very easy to use and schedule automatically Tasks on different Machines without spending hours in coding multithreading algorithm.
40k rows, 40 columns are not a big problem for python, BUT I think "R" should be able to handle this, it's a very popular language for statistical algorithms. Maybe some settings are false...
Short:

Install Python (or better Anaconda)
on Sklearn-site you will find some demo code
Try it with your Dataset
Maybe you could benchmark it :)


How to select regression algorithm for noisy (scattered) data?

The model I would use is the one that minimizes the accumulated quadratic error. Both models you are using, linear and quadratic, looks good. You can compute which one has the lowest error.

If you want to use an advanced method you can use RANSAC. It is an iterative method for regression that assumes that there are outliers and remove them from the optimization. So your model should be more accurate that just using the first approach I told you.


You might benefit from random forests instead which aim to achieve the same objectives you are aiming for, i.e better generalization through pruning to remove overfitting.
scikit learn's random forest algorithm will let you specify how many or what proportion of variables you want to automatically drop across the many trees whose results will be averaged for even better generalization performance.


What to do if a specific label of a category appears only a few times?

Since a few cars models make your 20% data, you can create a similarity matrix of all the car classes with these 2-3 car models. So in this matrix, each car model has 2-3 values depicting its similarity with those 2-3 car models. You can now add these new features to your existing features to get some improvement over the classification of the underrepresented car models.

You may play with the number of cars models to compare each car model to. Here, these 2-3 car models cover 20% of your data. It might be that 30 car models would cover say 40% of the data which is still useful as there are 3000 car models.


I think first converting the data from a wide to long format should allow you to sort the variables and get the result you want. This would look something like this:
import pandas as pd

# create example df
df = pd.DataFrame({
    "age_bin": ["0-10", "10-20", "20-30"],
    "col1": [5, 4, 2],
    "col2": [15, 23, 34],
    "col3": [32, 12, 24]
})

age_bin
col1
col2
col3

0
0-10
5
15
32

1
10-20
4
23
12

2
20-30
2
34
24

(
    df
    # convert from wide to long format
    .melt(id_vars="age_bin")
    # sort dataframe to make sure highest means are first
    .sort_values(["age_bin", "value"], ascending=[True, False])
    # create groups based on "age_bin" column
    .groupby("age_bin")
    # select top N rows within the groups created, change to 5 to get the top 5 columns
    .head(1)
)

Result:

age_bin
variable
value

6
0-10
col3
32

4
10-20
col2
23

5
20-30
col2
34


You want to manually label some cases and then extend that "manual labeling" to the rest of the data.
This is a supervised learning excercise with prior manual labeling by you.
Let's suppose you have partitioned a random, suitably sized training data set. Now you need to model a classification algorithm via the classical modeling pipeline and use this model to predict the classifaction/label in the rest of the data.
So yes this is easily possible,however modeling a text classification model is non-trivial and you need to understand basic modeling.
Here are the basic steps, please read upon each step you do not know how to do:

Split your data into a training set (you will model on this data), a test set and a target set.

Manually label your training and test set.

Choose the kind of classification algorithm you want to use. You can use classical ML models but this involves heavily tokenizing and transforming your data to numerical components. You could also use more advanced deep learning techniques for text classification like BERT.

Create a suitable transformer to tidy and transform your data into the right format for your chosen algorithm.

Train the model on your manually labeled training data.

Evaluate and optimize your performance with the test data set.

Use the final model to predict labels in your target set.

Understand that the quality of the automatic labeling will only be as good as your manual labels.


Segmenting data in a Dataframe and assigning order numbers (Python using Pandas)

I think you are asking how to add a column "priority" to the end of a dataFrame that has been sorted.

Assuming the dataFrame is in the order you like, I would first get the length of the frame:

length = len(df)

Then apply that to a new column in the dataFrame:

df['**priority**'] = ["**"+str(x)+"**" for x in range(length)

I'm still new to posting on here so I don't know how to get the input/output examples working, but that should get you a new column on the end of your frame, in string format, like you pictured above.


How to predict product price range using machine learning algorithms

If in your dataset you have two columns for the minimum and maximum value as columns n-1 and n, you can train the dataset for n-1 columns, where n-1th column is your dependent value, then for predicting nth column, the nth column would be your dependent value and n-1 others independent.