Feasibility: train a model to learn how to extract data from documents

Both of the answers from FelixGK and Wargream have merit.  With the information you have posted though, I am not sure you are approaching the problem with the right objective, so the expected solution may seem further away than it probably is.

As FelixGK suggests, you should break the models down into a set of one-vs-the-rest binary classifiers, one for each label.

I am not sure what you mean when you state

  From my understanding this apporach would need like 20 labels for every training example. And also the prediction is also only "correct", if all labels are matching.

Do you expect one model to collect multiple data elements within a document and assign the correct label to the element?  This is probably difficult in a single model, but could be easier using some form of ensemble approach, like a set of binary classifiers.  If you want to try it all in one model, I would look at neural networks and specifically, a network that implements a SoftMax activation function in the final layer. This will allow you to produce a multi-class output with probabilities assigned to each class. When using softmax, you generally want to accept the result with the highest probability.


NLP CounterVectorizer (sklearn), not able to get it to fit my code

This is normal, the fit method of scikit-learn'sCountverctorizer object changes the object inplace, and returns the object itself as explained in the documentation. This holds true for any sklearn object implementing a fit method, by the way.
When you run:
cv.fit(messages)

The cv object learns the transformation (here it is simply about counting words) and stores all the necessary parameters in order to be able to later apply the transformation to any new data point.
If you want to transform the data, you need to use first the fit method (to learn the transformation as said before) and then the transform method.
In your case if you want to transform your list of sentences you should do as follows:
cv = CountVectorizer()

messages = ['This is a good product', 'It was a bit pricey for what it does', 'I found good value here']

cv.fit(messages)
vectorized_messages = cv.transform(messages)

vectorized_messages

This will output:
<3x14 sparse matrix of type '<class 'numpy.int64'>'
    with 15 stored elements in Compressed Sparse Row format>

So you will have successfully transformed your list of strings to a a matrix of token counts.
Note that sklearn often also implement a fit_transform method that does bothe at the same time:
cv = CountVectorizer()
cv.fit(messages)
vectorized_messages = cv.transform(messages)

#is the same as:
cv = CountVectorizer()
vectorized_messages = cv.transform(messages)


Pre-train using sigmoid and train using ReLU?

Since RBM has only one layer of weights, why you bother changing sigmoid to ReLU in 1-layer net? Gradient vanishing is very unlikely to happen in such a shallow net.

You can also train Gaussian-Bernoulli or Gaussian-Gaussian RBM (more here), which has identity activation function, which is closer to ReLU than sigmoid, and what is more justified if you have real-valued data, not binary. However, these types of networks are a bit less stable to train because of this unconstrained activation.


You should also check the correlation between the features. The problem you mentioned arises when features are highly correlated.

Also perform Kfold validation in your dataset.


How to train models using Google ML ( keras & tenserflow)

You should provide more information about the previous framework you used and your layer setup. I imagine you also want to "translate" your trained model into a proper weight format


Hypothesis vs Hyperplane in Machine Learning

Let's say you want to learn a specific mapping from $\mathbb{R}^{n}$ to $\mathbb{R}^{m}$.
The following elaboration assumes that you are referring to supervised learning.
Hyperplane
Hyperplanes play a key role in neural networks.
Consider the set $H_{v,d} := \{x \in \mathbb{R}^{n} \mid  \langle x,v \rangle = d \} $ for $v \in \mathbb{R}^{n}$ and $d \in \mathbb{R}$.
For $v = 0$, we have $H_{v,d} = \begin{cases} \emptyset & d \neq 0 \\ \mathbb{R}^{n} & d = 0 \end{cases}$
If $v \neq 0$, $\dim(H_{v,d}) = n-1$ and $H_{v,d}$ is a hyperplane.
If $d= 0 $, $H_{v,d}$ if a vector space (going through the origin), otherwise it is an affine space.
In general, a hyperplane is an affine subspace with co-dimension 1, which is of the form $H = v + U := \{v+u \mid u \in U \}$, where $U$ is a subspace of dimension $\dim(U) = n-1$.
For example in a binary classification task, a hyperplane can be used to seperate the two classes. A geometric explanation of the role of hyperplane in neural networks can be found here.
In short, a neural network uses for each neuron a hyperplane (in the hidden or output layer) to define the output value of the neuron. All points within the same side (with respect to the hyperplane) are either mapped the same value (using a heavy-side function), or it depends on the distance to the hyperplane (e.g. using the sigmoid function). This understanding can be used to interprete the mapping of an input vector to a layer $l>1$. Essentially a neural network learns an arrangement of hyperplanes, which define regions. Each region is mapped to the same value (in case of the heavy-side function) or again, it further depends on the position within such a region (see here)

Model Hypothesis
A model hypothesis usually refers to something else.
We consider a set of model hypothesis $\mathcal{H}$, and each model $h \in \mathcal{H}$ may be used as the desired mapping. For example, we can define the set $$P[n] := \left \{ \begin{align*}
f: \mathbb{R}^{n} &\rightarrow \mathbb{R} \\
 x &\mapsto \sum_{r = 0}^{n} \sum_{\substack{b_{1}+\ldots+b_{n} = r \\ b_{k} \in \mathbb{N}, \forall k} }  a_{b_{1},\ldots,b_{r}} \prod_{s = 1}^{r} x_{s}^{b_{s}}
\end{align*} \middle \rvert  a_{b_{1},\ldots,b_{r}} \in \mathbb{R}
 \right \}. $$
The set collects all multivariate polynomials of degree at most $n$.
Then we can consider the hypothesis set
$\mathcal{H} := \left \{
\begin{align*}
f:\mathbb{R}^{n} &\rightarrow \mathbb{R}^{m} \\ x  &\mapsto \begin{pmatrix} p_{1}(x) \\ \vdots \\ p_{m}(x) \end{pmatrix} \end{align*} \middle \lvert p_{1},\ldots,p_{m} \in P[n]
\right \}$.
Then, $h \in \mathcal{H}$ maps an input vector $x$ to $h(x) = y$, where each component $y_{u}$ is given by a multivariate polynomial function in $x$ of degree at most $n$.
The task of the training is then to find the best coefficients which results in an  optimal hypothesis $h^{*} \in \mathcal{H}$, which is the model that is used to perform inference. Here, optimal means that we find the best model within the set of all considered models in $\mathcal{H}$.
Likewise for neural networks, if the "architecture" of the neural network is fixed, this defines the hypothesis set $\mathcal{H}$, where each $h \in \mathcal{H}$ uses the same neural network architecture, but with a specific choise of weights.
Training a neural network then delivers a model $h \in \mathcal{H}$, with "optimal" weights (in practice the weights are often not optimal).
So in short, the set of model hypothesis $\mathcal{H}$ defines which mappings can be used. The training within supervised learning then selects the best hypothesis $h \in \mathcal{H}$.


What kind of analysis can I do in this dataset?

You may go through two different approaches:
Unsupervised Learning (Clustering)
You can choose attributes who make someone's profile and try to cluster it (e.g. using k-means). If you look at clusters according to different attributes it may give you some insight about the data. Do not forget to exclude names as they are meaningless for analysis.
Supervised Learning (Classification)
You can use relevant attributes which may affect an output e.g. age and city may affect gender or Nr of children so if u put these as targets you can classify different users according to their age and city.
The point is that your data is not suitable for more sophisticated analysis as your features are not many and too complicated (i.e. answering the questions like age distribution among a certain gender or the relation between cities, age and number of children or something like them are the only questions can be answered)
I hope it helped and if there are more questions please comment here.
UPDATE 1
You can look at your data from different point of view so first try to choose one. For example clients can be clustered via their transactions (when they bought, what they bought, how expensive they bought, etc) or by their personal info (gender, age, payment method, etc). It gives you a first impression of your client data. Then go through clustering using all those information i.e. put personal and transactional features together. It gives you an overview of customers in general and might result in some significant clusters (categories).
After that you can look at the problem from product point of view meaning you look at the transactional features and try to see different distributions and histograms to get an overview of what's happening to the product e.g. you can set time as the x axis and try to extract different time series like how many product are sold at this point of time?, what kind of people (age, gender, etc) bought at this point of time? and so on.
For general analysis a dimensionality reduction (e.g. PCA) might reveal some information and give you insight. Please note that for any kind of analysis ONLY use relevant features e.g. IDs are not informative but categories.
The most important point for recommending product to a customer (or predicting what they buy) is to use your input/output pair properly so the customer info (age, city, when does he/she usually buy something?, etc) are inputs and products (their categories, their type, whatever u know about them) are outputs.
Good Luck!
PS1: From experience of doing an industrial version of exactly the same project I would say payment method is not much informative.
PS2: StackExchange provides an upvote and an accept button for good answers!


Finding point of interest in time series data

Try derivative after either a low-pass filter or smoothing (probably exponential smoothing) to cut down on the noise. Big changes result in a big derivative (up or down).


Approach to classify spatial trajectories of vessels

Your data contains spatial and temporal data, which means that you can account for location and speed.

Not sure how global the coordinates are, but perhaps grouping them per region might be interesting. It would be like putting all coordinates that you're given and defining zones of interests (zone A, B, C, etc.). Perhaps, some boats only sail in specific regions. Indeed, yachts might be found more often in the Caribbeans than in the Arctic sea. Of course, the number of zones you want to define needs to be fine-tuned.

Assuming that your data accounts for an entire trip from port A to B, you could, of course, use the length of the journey as a feature, as I'm assuming that not all boats travel the same distances. You could also use the overall direction of the trip as the angle between port A and port B.

You've already accounted for speed by computing the mean, standard deviation, and other metrics. What you could also do is fit a polynomial of the speed, and use its coefficients as features. Perhaps certain boats have a more steady speed than other boats.

Following the same thought, you could perhaps fit a polynomial describing the course of each boat and use its coefficients as features. View the course here as the change of angle. This should capture how much "zig zagging" a boat does. Of course, you'll need to fine-tune to pick the right polynomial.

Good luck!


How to get a feature from sound/audio data learning using machine learning supervised classification?

Your pipeline is roughly right, but clarifying it a bit below:

1. Load the audio
2. Convert the audio to a `spectrogram` using STFT
3. Apply mel-scale filtering to get `mel-spectrogram`
4. Transform using DCT to get `MFCC`

You now have a time sequence of MFCC frames. It is typical for the classifier to operate on a time window of such frames. The window length can be relatively small (say 4-10 frames), or be the entire length of an audio clip. If they are smaller, the windows are often processed with some overlap.

Learning and outputting predictions for the entire file is the easiest, a standard classification problem.

If labels are only available on the entire file ("weak labeling") but one wants to output a prediction per window, this is usually done with Multi-Instance Learning.


What is the selection criteria to choose between XGBoost and Random Forest

Let´s say that the best way to choose is empirical. You run both algorithms in the dataset and check which one has better performance.

It's true that you can do a lot of theoretical analysis but at the end you have to try no matter what. They both use decision trees ensemble so the results should not be too different. By experience gradient boosting tends to achieve better results. Also, it is more mathematically complicated to understand.

Normally decision trees don´t require too much parameter turning, or at least less than other models.

There is no classical statistic test that will tell you will perform better. There are some heuristics but I find them overly complicated.


The approach seems correct, don't worry. But I'd suggest to use Naive Bayes Classifier instead of SVM - it perfectly works for this task (and actually popular email services use this algorithm for spam detection).


correcting conditional and marginal distribution in transfer learning

Bringing the distributions closer means that we are trying to modify the source data usually by performing weights in the instances or in the features (or in both in hybrid algorithms) in order to make the weighted data more similar to the targer data. If we achieve that then we will be able to train a model considering also the source data, which usually has a larger size or it is well annotated in comparison with the target.


Getting the earliest date (duplicates due to several call ids and agents)

This solved the problem:

select distinct on (customer)
customer, agent, min(call)
from call_data;

I am using PostgreSQL.


How to handle images of different sizes that are smaller than the input layer of a deep learning model?

The issue was fixed by resizing the Tensor to fit into the model. I was reshaping them instead of resizing them.


Which NN architecture solves my problem?

Welcome to SE:DataScience.

What you are looking for is called image captioning. A common approach for this is called an encoder-decoder model, where the encoder is a CNN-based NN to encode the content of an image into a vector representation, and a RNN-based decoder generates its caption conditioned on that vector.

This is a relatively well-studied area, with a lot of well-formatted dataset and open-source implementations, for example in TensorFlow. The link above can also serve as a good tutorial for this topic.

More recent models can use the attention mechanism to focus on specific part (object) in an image, do dense captioning, or answer questions about images, etc.


Anomaly detection on a time-series data in a CSV format using python

With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. Visually inspect the data for outliers. People are pretty good at this. I would then try to describe a mathematical model to fit the data with the outliers removed. Finally you can identify the outliers as those points greater than some statistical measure away from the difference of the model and the data.


Yes, and it would be $124 \times 124 \times40$.

To calculate the output dimensions:

$(dim_x - filterdim_x + 1) \times (dim_y - filterdim_y + 1) \times filters$

in your case:

$(128 - 5 + 1) \times (128 - 5 + 1) \times 40 = 124 \times 124 \times40$


How can I perform categorical encoding when the dataset is too large for memory?

You can read the file line by line (or block by block), assuming it is in a format where that can be done, and keep track of the unique values of the category.

Can handle multiple fields in 1 pass of the data instead of the 1 field like the below code.

Of course the easiest way is to get more memory or move to a cluster. The line-by-line solution will not be the fastest.

In pseudocode

unique_values_in_field_1 = set()

file = open(input)
while not EOF
    line = readline(file)
    # can also read a block of lines and adjust the code accordingly
    field_1 = line.field_1
    # make a function to handle multiple fields during the same pass of reading the file
    if field_1 not in unique_values_in_field_1:
       unique_values_in_field_1.add(field_1)

close(file)

# now have all of the unique values in this field. Do whatever is needed to transform them.


Building predictive model with low correlated data

really hard answer for u question cause there is to little information. Try to make EDA and attach it to question. Cause EDA define the model

Anywhere, for low corellated data, try to use k-NN. If u use Python: scikit-learn have implementation. Also u can try decision trees.

Sorry, but i'm also new in DS and can be wrong:)


Masked Language Modeling on Domain-specific Data

First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. Datasets (benchmarks) they used to train and evaluate this model from scratch were WMT 2014 Engligh-to-German, WMT 2014 English-to-French (section 5.1 of the paper). The conclusion is that you cannot train transformers from scratch unless you have your sentence pairs in 2 languages.
MLM on the other hand is something that BERT used for training. So if you want to go in this direction, you can use the pre-trained BERT and fine-tune it with Masked language model head on top of your BERT model using your own dataset. You need to stick to the tokenizer that BERT was originally trained on but at least you are taking advantage of some general context that was learned during training BERT from scratch. If you want to train BERT either from scratch or fine-tune with MLM head you can follow this tutorial from hugging face


Handling data imbalance and class number for classification

People talk a lot about data imbalance, but in general I think you don't need to worry about it unless your data is really imbalanced (like <1% of one label).  50/200 is fine. If you build a logistic regression model on that dataset, the model will be biased towards the majority class - but if you gave me no information about an input to classify, the prior probability is that the new input is a member of the majority class anyway.

The question you want to be able to answer is whether you are differentiating classes fine - so if you do have a minority class, do NOT use 'accuracy' as a metric. Use something like area under the ROC curve (commonly called AUC) instead.

If your data is really super imbalanced, you can either over-sample the minority class or use something called 'SMOTE', for "Synthetic Minority Over-Sampling Technique", which is a more advanced version of the same thing.  Some algorithms also let you set higher weights on minority classes, which essentially incentivizes the model to pay attention to the minority class by making minority-class errors cost more.

To learn to differentiate between lots of classes, I think (a) you will need to have a ton of examples to learn from and (b) a model that's expressive enough to capture class differences (like deep neural network, or boosted decision tree), and (c) use softmax output. If those still don't work, you might try a 'model-free' approach like K-nearest-neighbors, which matches each input to the most similar labeled data.  For kNN to work however, you need to have a very reasonable distance metric.


Time Series Generation - Multi Dimensional Time Series Data

Time series data must contain all your observations with some standard effect of time (Bit obvious here). If i wants to test theory on some model & need some dataset then the parameters would be like, Timestamps * no. of features (includes player's position with respect to a source). A 2d tensor would suffice. My reason for not making a 3d tensor is that it would lead to more complex scenario where i would have to co-relate the 3rd dimension (No of players [N]) with first 2 dimensions to predict. Better to simplify the positions & in a single row i can have multiple labels as every position will matter to my model.

Time series generation generally falls under continuous prediction with previous data predicted taken to be an observation. I would rather make it fall under Reinforcement learning. Yes you can work with supervised learning in mind but try RL approach as well.

As per the missing values i would say that remove them if they don't make more than 10-15% of data. There are no fix bars on the mentioned percentage. If it is more than that please fill in with interpolation or rolling average (both benefited me). Rest pre-processing depends upon type of data, normalise data, remove outlier etc.

Yes it will affect the data, but you can generate data using sign wave's different fluctuation for your testing or use any other functions to generate a signal(scipy preferred).

Currently i am testing bidirectional lstm - CNN combination for my time series, yes ARIMA is good but doing a little of this expirmenet won't hurt. I would say go for CNN-Any of RNN combination for time series.

Hope this helps.


SGDClassifier fit and partial_fit functions

fit(), always initializes the parameters like a new object, and trains the model with the dataset passed in fit() method.

Whereas partial_fit(), works on top of the initialize parameter and tries to  improve the existing weights with the new dataset passed in partial_fit().

It is always good to save the model in persistent storage (say pickle file), for later use or for further training.


Why does reducing polynomial regression to linear regression work?

I don't really understand your problem statement. You can simply add squared terms to any linear model. Say you have $y$ and $x$ and you want to model a polynomial function, you can write a model like:

$y=\beta_0 + \beta_1 x_1 + \beta_2 x_1^2 + u$.

In matrix form this would look like

$y=X\beta + u$.

An example with some numbers would write:

$\left( \begin{array}{rrrr}
5 \\
8 \\
7 \\
6 \\
\end{array}\right) $ =
$\left( \begin{array}{rrrr}
1 & 8& 8^2  \\
1 & 4 & 4^2  \\
1 & 3 & 3^2  \\
1 & 6 & 6^2  \\
\end{array}\right) \beta + u.$

You can solve this like $\hat{\beta} = (X'X)^{-1} X'y$, where $\hat{\beta}$ are the coefficients of the linear regression model.


You need to make a dictionary of words. It means you have to make a dictionary which you assign each word a unique value. then you can use one-hot-encoding to represent each word uniquely. If this is what you need it will do what you want. But this has a big problem. When you think about cats and dogs, you may find similarities and differences between them. This is because you have more knowledge than the only representation of words in your brain. Consequently, you should use approaches to assign a unique number to each word, and put near concepts as neighbors. For the first part take a look at here and the second part, take a look at here.


How to aggregate features to a group level as a feature in machine learning model?

Ages
I guess adding up the ages won't makes sense, since the number 30 means that there is one person of 20 years and one of 10, or a bunch of 5 years old?
What you could do is have number_of_adults_in_household and number_of_children_in_household. This way you still aggregate the ages but keep more information.
Education
Education might be a more difficult feature to aggregate. My educated suggestion would be to have highest_education_degree_in_household.


Stanford parser Python : Combine NER and POS tags

This question has already been answered. See Alexis' answer at https://stackoverflow.com/questions/30664677/extract-list-of-persons-and-organizations-using-stanford-ner-tagger-in-nltk

If you want to use chunking NER without the Stanford library see alvas' answer at https://stackoverflow.com/questions/31836058/nltk-named-entity-recognition-to-a-python-list


R: Rates of change from an initial value

You can reproduce both of your desired outputs using dplyr. I don't know how you were trying to use lag, but one solution is to have mutate create dummy columns that you will remove at the end.

df %>%                                            # Set up the pipe
arrange(Group, Year) %>%                         # Ensures the data are in ascending order
group_by(Group) %>%                              # Group by 'Group'
mutate(lag = lag(Data),                          # Create a dummy column 'lag'
     RateOfChange = (Data - lag) * 100 / lag,    # Calculate RateOfChange
     First = head(Data, 1),                      # Dummy column for the frist yearly value by group
     BaselineChange =                            # Calculate BaselineChange
       case_when(Data != First ~ (Data - First) * 100 / First,
                 TRUE ~ 1 * NA)) %>%             # This line provides NA value for first year
select(Year, Group, Data,                        # Select only columns you want
       RateOfChange, BaselineChange) %>%
head(5)                                          # Print the first five rows to validate

Here is the output:

# A tibble: 5 x 5
# Groups:   Group [1]
  Year Group  Data RateOfChange BaselineChange
  <int> <fct> <dbl>        <dbl>          <dbl>
1  2010 1a      3.5        NA             NA
2  2011 1a      4          14.3           14.3
3  2012 1a      3.8        -5.             8.57
4  2013 1a      4.1         7.89          17.1
5  2014 1a      4          -2.44          14.3


Support Vector Machine (SVM) for classification problem based on Earth Mover's Distance (EMD)

pyemd has a different API than scikit-learn. Per the scikit-learn docs:

"Your kernel must take as arguments two matrices of shape (n_samples_1, n_features), (n_samples_2, n_features) and return a kernel matrix of shape (n_samples_1, n_samples_2)."

As you rightly point out, the EMD implemented by pyemd takes three arguments, but the distance_metric is not the distance matrix between the distributions. It defines the distance between the histogram bin edges, which is independent of the particular distribution.

If you want to use the EMD implemented by pyemd, I'd suggest wrapping it in another function so that you match the function signature expected by scikit-learn. That will require you to compute distance_metric in the wrapper and pass it as an argument to pyemd.emd(). Again, the distance_metric argument only depends on how you have chosen the bin edges for the discretized distribution.


Replacing the non-zero values is meant to simplify the matrix, i.e. we only care whether the value is zero or not.

Then you're supposed to calculate the Jaccard distance between pairs of columns. For instance between the first two columns you have $|a\cap b|=1$, $|a\cup b|=2$ so the distance is 1/2.

After that you're supposed to apply hierarchical clustering using the distance. This means progressively grouping elements (here columns) which are close to each other according to the distance.


Do the benefits of ridge regression diminish with larger datasets?

One thing that you might be overlooking with your reasoning is the fact that increased predictors does not necessarily lead to a better model. In this case, the more predictors you have the higher the risk of collinearity that there is between them - thus increasing the utility of ridge regression.
In fact, a second point:
The larger the number of predictors that you have in your model the more useful techniques like ridge regression are. This is because with so many predictors is is very difficult to determine if collinearity or other relationships exist between predictors. When you have a smaller model with say 5 predictors, you could verify this fairly easily. With 1,000 predictors this would be much more difficult.
In the case of the Lasso regression the benefits are even more obvious. Since predictor coefficients values can be shrunk fully to zero this acts as a form of feature selection. Here, you are potentially removing predictors that capture redundant information.
Also - in answer to the second part of your question - OLS does not perform any form of feature selection. Therefore, it will not be able to pick out which parameters are the most important by adding more. The more parameters you add to your model the higher the variance gets. Why is this? You have more estimations to make and therefore collectively your model is less reliable. Consequently, your model will likely inflate the coefficients on the predictors giving undue "weight" or importance to certain predictors.


How to speedup message passing between computing nodes

Firstly, I would generally agree with everything that AirThomas suggested. Caching things is generally good if you can, but I find it slightly brittle since that's very dependent on exactly what your application is. Data compression is another very solid suggestion, but my impression on both of these is that the speedups you're looking at are going to be relatively marginal. Maybe as high as 2-5x, but I would be very surprised if they were any faster than that.

Under the assumption that pure I/O (writing to/reading from memory) is not your limiting factor (if it is, you're probably not going to get a lot faster), I would make a strong plug for zeromq. In the words of the creators:

  We took a normal TCP socket, injected it with a mix of radioactive
  isotopes stolen from a secret Soviet atomic research project,
  bombarded it with 1950-era cosmic rays, and put it into the hands of a
  drug-addled comic book author with a badly-disguised fetish for
  bulging muscles clad in spandex. Yes, ØMQ sockets are the world-saving
  superheroes of the networking world.

While that may be a little dramatic, zeromq sockets in my opinion are one of the most amazing pieces of software that the world of computer networks has put together in several years. I'm not sure what you're using for your message-passing layer right now, but if you're using something traditional like rabbitmq, you're liable to see speedups of multiple orders of magnitude (personally noticed about 500x, but depends a lot of architecture)

Check out some basic benchmarks here.


What's the difference between data products and intelligent systems?

This is a very vague question. However, I will try to make sense of it. Considering rules of logic as well as your statement that both entities are "software systems that are based on data and algorithms", it appears that data products are intelligent systems and intelligent systems are, to some degree, data products. Therefore, it can be argued that the difference between the terms "data products" and "intelligent systems" is purely in the focus (source of information or purpose of system dimensions) of each type of systems (data vs. intelligence/algorithms).


Why the number of neurons or convolutions chosen equal powers of two?

Deep Neural Networks are usually trained on GPUs to speed up training time. Using power of two for the network topology follows the same logic as using power of two for image textures in computer games.

The GPU can take advantage of optimizations related to efficiencies in working with powers of two. (see https://gamedev.stackexchange.com/questions/26187/why-are-textures-always-square-powers-of-two-what-if-they-arent)


classification imbalance data - bias and class weight

Normally the bias is "part" of the weights, and for this particular algorithm or model, the bias is a way to use to help solve unbalanced data issues. For fraud detection, the rate of event data is really rare, here event is the transaction is a fraud.

In this so-called machine learning model, the whole tutorial was doing logistic regression, a century-old statistical learning. We mostly using it for classification.

keras.layers.Dense(1, activation='sigmoid',
                         bias_initializer=output_bias),

At the last layer of the model, it has a sigmoid activation, here it's a logistic function (https://en.wikipedia.org/wiki/Sigmoid_function).

For fraud detection, the data in a tabular format, aka, the data is one dimension, not like the image. For one dimension data, statistics will shine over deep learning. Since the target event is rare, a representative sample is unlikely to have enough target events to build a good predictive model. Fortunately, the amount of information in a data set with a categorical outcome  (such as a response to a marketing campaign and fraud detection) is determined not by the total number of cases in the data set, but by the number of cases in the rarest outcome category.

One way the tutorial borrowed from the statistics is oversampling. While oversampling improve the model performance with less time, it also introduces some biases. Yo need to correct these biases so that the results or the model learned are applicable to the population. For example, one widespread strategy for predicting rare events is to build a model on a sample that disproportionally over-represents the event cases. Here you choose a sampling of data that contains all of the events (fraud) and only a subset of the nonevents (normal transaction).

Again, it will introduce biases, and later you need to correct them, this is why you need to calculate the correct bias.

Don't confuse with another bias from the coefficient, it is from the same rationale, but for now and here, you should think the bias we are talking now is not $\beta_0$ or the intercept in the linear equation but the bias of the model.
$$y=\beta_0+\beta_1 x$$
"The linear equation assigns one scale factor to each input value called a coefficient, and it is commonly represented by the Greek letter Beta (β). One more coefficient is added, giving the line an additional degree of freedom (moving the line upwards or downwards) and is called intercept or the bias coefficient."

The effect of the oversampling is like the figure showing below.

This graph of the response surface for a logistic regression model, which is the machine learning model in the tutorial. As you can see, oversampling does not affect the slops, but it does make the intercept too high, which is why we call it bias also. The difference between the intercepts is called offset, whose equation is $ln(\dfrac{\pi_{0}\rho_{1}}{\pi_{1}\rho_{0}})$.

$\pi_0$ is the proportion of non-events in the population
$\pi_1$ is the proportion of events in the population
$\rho_1$ is the proportion of events in the sample
$\pi_0$ is the proportion of non-events in the sample

Another method for adjusting for oversampling is to incorporate sampling weights, by correcting via sample weights. These weights are not the $\beta_1$ from the linear regression function, nor from the machine learning weight, it is a ration for the event and non-event classes.

$ weights_{i} = \dfrac{\pi_{1}}{\rho_{1}}$ if $y_i = 1$
$ weights_{i} = \dfrac{\pi_{0}}{\rho_{0}}$ if $y_i = 0$

The offset method and the weighted method are not statistically equivalent. So the parameter estimates are not exactly the same, but they have the same large-sample statistical properties. So here the offset, using bias approach is better and considered superior.

The bottom line is there are two sets of biases and weights:
- model parameter: you can think bias and weight as $\beta_0$ and $\beta_1$.
- oversampling: bias is talking the sampling method is non-traditional, and weights are the "proportional rate" between classes.

Hope this helps.


Choosing best model produced from different algorithms. Metric produced by cross-validation on the train set or metric produced on the test set?

Your main question seems to be what about different algorithms?, and the answer to that is it makes no difference.
While hard to argue with the "test set" answer logic it is also easy to find a counter-example. If your training set has 100K examples, and you used, say 5-fold CV, and then you also have a test set with only 200 examples... the test set is 100x smaller and likely to be much less reliable.
Normally you would use cross-validation, or a test set. And if your training data is large, I would generally recommend just using a test set, because you probably have enough data and because CV takes longer to train. When training data is scarce, cross-validation makes more sense: training time probably matters less, you want to train the final model on everything you have, and also you will value the N different opinions that N-fold CV gives you, because your sample size is small.
If you have used both cross-validation and a test set for multiple models (whether same algorithm or different algorithms), then you should expect them both to agree on your choice of best model. If model A does better than model B on cross-validation, but model B does better on the test data, then you should go looking for a problem.
If that does happen, I'd first be suspicious that the test data is not representative. E.g. to exaggerate, your training data covers the period 1950 to 1970, and your test data is for the year 2020. At the other extreme, I'd check that the test data is not actually in your training data (implying that model B does a better job of memorizing).


When you look at a linear regression $y_i = \beta_0 + \beta_1 x_i + u_i$, you can estimate the (ex ante unknown) coefficients $\beta$ using matrix algebra $(X'X)^{-1} X'y = \hat{\beta}$.
A point estimate would be the "best guess" $\hat{y}=\hat{\beta} X$.
Each $\hat{\beta}$ is associated with uncertainty about the estimate, expressed by the standard error of the coefficient (see this post). The confidence interval intuitively says that you want to find the range in which the true $\hat{\beta}$ is with 95% likelyhood (assuming a normal distribution). In this case, you can calculate a confidence interval (for $\hat{\beta}$) by:
$$ CI_{0.95}^{\beta} = [\hat{\beta_i} - 1.96*SE(\hat{\beta_i}), \hat{\beta_i} + 1.96*SE(\hat{\beta_i})].$$
So you can say (under some assumptions) that the true value of the estimated $\hat{\beta}$ is within the confidence band with 95% probability. Note that in linear regression, when you say a coefficient is "statistically significant", this coincides with "the confidence band is strictly positive or negative" (does not cross zero).
Example in R:
Linear regression:
library("ISLR")
auto = ISLR::Auto
ols = lm(mpg~horsepower,data=auto)
summary(ols)

Result:
Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 39.935861   0.717499   55.66   <2e-16 ***
horsepower  -0.157845   0.006446  -24.49   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.906 on 390 degrees of freedom
Multiple R-squared:  0.6059,    Adjusted R-squared:  0.6049
F-statistic: 599.7 on 1 and 390 DF,  p-value: < 2.2e-16

This tells us that when horsepower goes up by one unit, mpg goes down by -0.158 (point estimate). Now when we ask, what is the true effect (under quite some assumtions) with 95% probability, we look at the CI.
# Confodence interval
confint(ols)

Which gives:
                2.5 %     97.5 %
(Intercept) 38.525212 41.3465103
horsepower  -0.170517 -0.1451725

We can do this "manually" using:
# Get standard errors
sqrt(diag(vcov(ols)))

(Intercept)  horsepower
0.717498656 0.006445501

And we can calculate the CIs:
# Lower CI
-0.157845 - 1.96*0.006445501

# Upper CI
-0.157845 + 1.96*0.006445501

Which yields:
[1] -0.1704782
[1] -0.1452118

So we could say that the true effect of horsepower on mpg is between -0.17 and -0.15 (and since the CI does not "cross" zero, the effect is statistically significant, meaning p-value < 0.05).


What is the definition of precision for no positive classifications?

I just found that sklearn.metrics.precision_score handles it like this:

>>> sklearn.metrics.precision_score(y_true, y_pred)
/home/moose/.local/lib/python2.7/site-packages/sklearn/metrics/classification.py:1074:
  UndefinedMetricWarning: Precision is ill-defined and being
    set to 0.0 due to no predicted samples.
    'precision', 'predicted', average, warn_for)
0.0

So they give a warning and set it to 0.


How to normalize data for Neural Network and Decision Forest

I disagree with the other comments.

First of all, I see no need to normalize data for decision trees. Decision trees work by calculating a score (usually entropy) for each different division of the data $(X\leq x_i,X>x_i)$. Applying a transformation to the data that does not change the order of the data makes no difference.

Random forests are just a bunch of decision trees, so it doesn't change this rationale.

Neural networks are a different story. First of all, in terms of prediction, it makes no difference. The neural network can easily counter your normalization since it just scales the weights and changes the bias. The big problem is in the training.

If you use an algorithm like resilient backpropagation to estimate the weights of the neural network, then it makes no difference. The reason is because it uses the sign of the gradient, not its magnitude, when changing the weights in the direction of whatever minimizes your error. This is the default algorithm for the neuralnet package in R, by the way.

When does it make a difference? When you are using traditional backpropagation with sigmoid activation functions, it can saturate the sigmoid derivative.

Consider the sigmoid function (green) and its derivative (blue):

What happens if you do not normalize your data is that your data is multiplied by the random weights and you get things like $s'(9999)=0$. The derivative of the sigmoid is (approximately) zero and the training process does not move along. The neural network that you end up with is just a neural network with random weights (there is no training).

Does this help us to know what the best normalization function is? But of course! First of all, it is crucial to use a normalization that centers your data because most implementation initialize bias at zero. I would normalize between -0.5 and 0.5, $\frac{X-\min{X}}{\max{X}-\min{X}}-0.5$. But standard score is also good.

The actual normalization is not very crucial because it only influences the initial iterations of the optimization process. As long as it is centered and most of your data is below 1, then it might mean you have to use slightly less or more iterations to get the same result. But the result will be the same, as long as you avoid the saturation problem I mentioned.

There is something not here discussed which is regularization. If you use regularization in your objective function, the way you normalize your data will affect the resulting model. I'm assuming your are already familiar with this. If you know that one variable is more prone to cause overfitting, your normalization of the data should take this into account. This is of course completely independent of neural networks being used.


What does it exactly mean when we say that PCA and LDA are linear methods of learning data representation?

LDA is linear classifier since the classification boundary in LDA has the following form :

As you can see, the above equation is linear in $x$. In above equation, ${\Pi_i}$ is the prior probability of class $i$, which is estimated from the training data and ${\mu_i}$ is the mean of class $i$ also estimated from the training data. ${\Sigma}$ is the common covariance matrix of all classes (it is assumed to be same in all classes in LDA, and that's why we get linear boundary, if you don't assume the same covariance matrix, the classification boundary no longer remains the linear in $x$).
PCA on the other hand is not a regression/classification algorithm. It is rather a feature extraction/ dimensionality reduction method, which helps you to represent your data in lower dimensions. It generally extracts the most significant $k$ features of your data. The value of $k$ is decided by you based on how many features you want to keep in your data. Or in what dimension you want to represent your data. PCA is the best representation of your data in to a lower dimensional space. This is linear transformation because of following equation :
$$ y \space = \space  W^Tx $$
As you can see, this equation is also linear in x. In this equation, the matrix W is a matrix obtained from the covariance matrix of your data. The first row of matrix $W$ corresponds to the eigen vector of covariance matrix of $x$ corresponding to highest eigen value. This is because that eigen vector gives the maximum variation in your data. (This is proved in the optimization of PCA). The second row corresponds to second max eigen value because it represents direction with second highest variation in your data. You can refer to the notes of Prof. Ali Ghodsi or Prof. Andrew NG for the proof of why we select the eigen vectors in this order in PCA.
Now coming back to your question about reproducing the data only linearly. Yes, you can only reproduce back your data by linear transformation in PCA. Why do you want want to reproduce your data non linearly when your transformation was linear? Even if you want to reconstruct your data non linearly from transformed data $y$, do you have any enough information to do so? You just have a matrix $W$ using which you linearly transformed your data $x$ into a lower dimensional space. A non linear reconstruction might look like this :
$$ x \space = \space y^TW_1^Ty \space+\space W^{-T}y $$
This is a non linear reconstruction. But do know what is $W_1$ here? Even if you try to estimate $W_1$, you'll endup getting it as a zero matrix since $y = W^Tx$ and so $W^{-T}y = x$. So, any other factor in the estimation will give you nothing but the zeros as a coeffocients of the other non linear factors (factors like $y^Ty$).


A timestep's prediction depends on future data

I think you have assumed that future timesteps also are used for learning by a simple LSTM. But, that actually is Bidirectional LSTM which learns from the past as well as future timesteps. Simple Recurrent Neural Networks don't have the access to future input information. You might want to refer to a book on recurrent neural networks. You can also look this question on cross validated for training an LSTM.


How the combination of cross entropy loss and gradient descent penalizes and rewards

The value of the loss function depends upon the prediction (which is a function of the input data and the model parameters) and the ground truth. Gradient descent works like this:

Initialize the model parameters in some manner.
Using the input data and current model parameters, figure out the loss value of the current network weights and biases.
Figure out how to update the weights and biases such that the loss value improves.
Update the weights and biases a certain amount based on the current learning rate.

This process is repeated until convergence (as measured in a few different ways, say improvement against a validation set).

If you want to dig in to how this all works, start with 3Blue1Brown's Neural Network series on YouTube, it's not even that long. It's a good start to get a feel for the concepts before you dig in to the math.


Here is a good example of how to implement forward and backward propagation, in numpy, but there should be similar functions in tf.


How to retrieve column names from applying a wrapper method in feature selection?

Based on this scikit-learn documentation, you can get a boolean mask (in the same order) of the input features, via the get_support method:


Correlation among variables (categorical, binary and numerical)

There are some distance/dissimilarity functions for two binary/boolean vectors such as Jaccard distance and  Hamming distance. You can check them from scipy's documentation. You can calculate similarities between two boolean columns by using these distance functions.
from scipy.spatial import distance
distance.hamming([1, 0, 0], [0, 1, 0])
0.66666666666666663

from scipy.spatial import distance
distance.jaccard([1, 0, 0], [0, 1, 0])
1.0


No.  I don't think so. And I don't think that is what "multivariate" means.

I rather think all of Weka's APIs and classifiers assume that there is exactly one attribute which gets assigned the "class" attribute label (that's the column to be predicted/classified).

Although there is a MultiClassClassifier meta-classifier, this is designed to improve
algorithms that can only handle binary class-attribute prediction problems (e.g. Weka's Glass.arff dataset with Logistic regression)


Why is it good news that the expected gradient of the loss for a mini batch is equal to the gradient for the whole set?

It is usually good news when an approximating function you use to take an action to optimise something is unbiased compared to the thing being approximated. It means your actions in aggregate will head towards the same optimisation point.

If the gradient was biased in the long term, then (depending on nature of the bias) an optimisation routine will converge incorrectly by following them in gradient descent, and give an incorrectly optimised answer.

It is sometimes OK if a value is initially biased, but the bias reduces over time towards zero. Not that this is the case here, but you will see that in some machine learning algorithms. For instance, in a neural network using momentum, typically the initial momentum values are biased, but the bias decays exponentially (it is actually possible to make an unbiased momentum estimate, but most implementations of SGD with momentum or Nesterov momentum do not do that - Adam implementations typically do).


Intuitive explanation of Noise Contrastive Estimation (NCE) loss?

Taken from this post:https://stats.stackexchange.com/a/245452/154812

The issue

There are some issues with learning the word vectors using an "standard" neural network. In this way, the word vectors are learned while the network learns to predict the next word given a window of words (the input of the network).

Predicting the next word is like predicting the class. That is, such a network is just a "standard" multinomial (multi-class) classifier. And this network must have as many output neurons as classes there are. When classes are actual words, the number of neurons is, well, huge.

A "standard" neural network is usually trained with a cross-entropy cost function which requires the values of the output neurons to represent probabilities - which means that the output "scores" computed by the network for each class have to be normalized, converted into actual probabilities for each class. This normalization step is achieved by means of the softmax function. Softmax is very costly when applied to a huge output layer.

The (a) solution

In order to deal with this issue, that is, the expensive computation of the softmax, Word2Vec uses a technique called noise-contrastive estimation. This technique was introduced by [A] (reformulated by [B]) then used in [C], [D], [E] to learn word embeddings from unlabelled natural language text.

The basic idea is to convert a multinomial classification problem (as it is the problem of predicting the next word) to a binary classification problem. That is, instead of using softmax to estimate a true probability distribution of the output word, a binary logistic regression (binary classification) is used instead.

For each training sample, the enhanced (optimized) classifier is fed a true pair (a center word and another word that appears in its context) and a number of kk randomly corrupted pairs (consisting of the center word and a randomly chosen word from the vocabulary). By learning to distinguish the true pairs from corrupted ones, the classifier will ultimately learn the word vectors.

This is important: instead of predicting the next word (the "standard" training technique), the optimized classifier simply predicts whether a pair of words is good or bad.

Word2Vec slightly customizes the process and calls it negative sampling. In Word2Vec, the words for the negative samples (used for the corrupted pairs) are drawn from a specially designed distribution, which favours less frequent words to be drawn more often.

References

[A] (2005) - Contrastive estimation: Training log-linear models on unlabeled data

[B] (2010) - Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

[C] (2008) - A unified architecture for natural language processing: Deep neural networks with multitask learning

[D] (2012) - A fast and simple algorithm for training neural probabilistic language models.

[E] (2013) - Learning word embeddings efficiently with noise-contrastive estimation.


Which method should be considered to evaluate the imbalanced multi-class classification?

This answer explains how micro and macro are different and which one you should use.


What is the difference between the value -99 and NaN in a data column?

No, it is not the same. It may have that meaning in that particular dataframe, but don't take that as a rule.

I'd recommend you replace that magic number with actual NaNs, and then try to find the best possible way of filling up the missing values.


I solved the problem this way:

I realized that I needed to find the HCF (highest common factor) of both the length of x_train and x_test. For this I wrote a simple HCF python code the output of it is the batch_size.

Hence data set of different size will have different batch sizes.

Following is the code I used to find the HCF:

def computeHCF(x, y):
    if x > y:
        smaller = y
    else:
        smaller = x
    for i in range(1, smaller+1):
        if((x % i == 0) and (y % i == 0)):
            hcf = i

    return hcf

batch_size= computeHCF(x_train.shape[0], x_test.shape[0])

And use this batch_size  both while defining the model, fitting the model(model.fit(..)) & predicting(model.predict(...)).

NOTE: We need to specify batch size while predicting the model. As:

model.predict(x_test, batch_size=batch_size)

Hope this helps someone!


Is it possible to use deep learning on small number of samples but each sample has a large amount data?

You have a database of 50x6 currently.

When you run a exercise of deep learning on images, you have n (images) x (width(px) x height(px) x 3) variables.

You have similarly characterized dataset when you use the wavelet transforms, so it is possible. You have to be very cautious because you will have more variables than n, so the danger of overfitting increases a lot.


Theoretical background for model arhitecture choosing

The theoretical estimation of the error depends on the data, and the fact that the labels can be predicted by the variable X. There's no way to a priori know which will be the best option regarding the two models, so just try.

However, I would bet that the second option will work better if y2 is correlated in some way to y1, and I would bet for the first option if they are independent.

However, when you train your model, I highly recommend sampling the data in a more balanced way, so that the model not learns to predict (0, 0) always regardless of the values of X (as a model that does these dummy predictions would have high accuracy). Another way to work around the imbalanced classes is to use AUC in ROC as a metric to choose among different models.


What is the Q function and what is the V function in reinforcement learning?

$V^\pi(s)$ is the state-value function of MDP (Markov Decision Process). It's the expected return starting from state $s$ following policy $\pi$.
In the expression
$$V^\pi(s) = E_{\pi} \{G_t \vert s_t = s\} $$
$G_t$ is the total DISCOUNTED reward from time step $t$, as opposed to $R_t$ which is an immediate return. Here you are taking the expectation of ALL actions according to the policy $\pi$.
$Q^\pi(s, a)$ is the action-value function. It is the expected return starting from state $s$, following policy $\pi$, taking action $a$. It's focusing on the particular action at the particular state.
$$Q^\pi(s, a) = E_\pi \{G_t | s_t = s, a_t = a\}$$
The relationship between $Q^\pi$ and $V^\pi$ (the value of being in that state) is
$$V^\pi(s) = \sum_{a ∈ A} \pi (a|s) * Q^\pi(s,a)$$
You sum every action-value multiplied by the probability to take that action (the policy $\pi(a|s)$).
If you think of the grid world example, you multiply the probability of (up/down/right/left) with the one step ahead state value of (up/down/right/left).


Can gradient boosted trees fit any function?

Yes - create a region for each data point (i.e., memorize the training data).

Thus it is possible for gradient boosted trees fit any training data, but it would have limited generalization to new data.


What does xgb's scale_pos_weight parameter do for regression?

There are a few unused or deprecated parameters in both XGBClassifier and XGBRegressor, so it might just be a matter of sloppy inheritance/c+p.

A couple of possibilities:

They copied over the params from Sklearn's GradientBoostedClassifier
They copied over the params from XGBClassifier
They inherited the properties from some class that already had those attributes.

There have been some consistency issues for a while now (I faintly recall nthreads versus n_job issue.)


How do I plot the predictions made by a LinearRegression model?

You cannot visualise the dimensions of an n-dimensional model all at once if n > 3. However, what you can do is plot the pairwise relationships between dimensions. You might want to look into the Seaborn library for visualisation tools.


What are the cases where it is fine to initialize all weights to zero

Whenever you have a convex cost function you are allowed to initialize your weights to zeros. The cost function of logistic regression and linear regression have convex cost function if you use MSE for, also RSS, linear regression and cross-entropy for logistic regression. The main idea is that for convex cost function you'll have just a single optimal point and it does not matter where you start, the starting point just changes the number of epochs to reach to that optimal point whilst for neural networks the cost function does not have just one optimal point. Take a look at here. About random initialization, you have to consider that you are not allowed to choose random weights which are too small or too big although the former was a more significant problem. If you choose random small weights you may have vanishing gradient problem which may lead to a network that does not learn. Consequently, you have to use standard initialization methods like He or Glorot, take a look at here and Understanding the difficulty of training deep feedforward neural networks.

Also, take a look at the following question.

How should the bias be initialized and regularized
When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization


Text Classification : Classifying N classes vs rest of the classes

This is called an open-class text classification problem, it's used in particular for some author identification problems.
I don't have any recent pointers but from a quick search I found this article: https://www.aclweb.org/anthology/N16-1061.pdf
In the field of author classification there is a similar problem called author verification, which can be treated as a one-class classification problem. You could consider using it in this way:

one-class classification between "known classes" vs. others
regular classification between the known classes


Using Majority Class to Predict Minority Class

Yes that's correct, but assuming that you follow the exact same methodology you will obtain exactly the same performance at the end, so there's no advantage.

Keep in mind that the problem with class imbalance is not that one class is harder to identify than the other, but rather that it's harder to properly separate the two classes.

[edit] It would be a different story when using one-class classification. I'm not sure if it makes sense in this case but maybe it could be something worth trying.


Linear vs Non linear regression (Basic Beginner)

Whether you use a linear model or a non-linear model depends completely on whether or not the data is linearly separable, not the number of features. If you have a dataset that is linearly separable, i.e a linear curve can determine the dependent variable, you would use linear regression irrespective of the number of features. Since real-world data is rarely linearly separable and linear regression does not provide accurate results on such data, non-linear regression is used.

And to answer your second question, yes you are correct. Overfitting occurs when your model has a very high accuracy for training data but a considerably low accuracy for validation/test data. It happens because the parameters are adjusted completely based on your training set, so there is not a generalised curve. Hence on newer data, the model would perform badly.


How many channels do ultrasound images have?

What is happening has nothing to do with the type of image (ultrasound in this case). It has to do with the type of file (JPEG in this case). By default, cv.imread assumes it is a 3-channel color image. You can specify grayscale when calling cv.imread if appropriate. The last dimension (the color dimension) in the im object could be empty.

There is also depth info (8 bits per channel). The default is 8 bit, you could specify 16 bit or 32 bit if apporiate.

Do not rely on function or software defaults. It is better to work from first principles of the raw data.


Is label-encoded data quantitative or qualitative?

Mapping to numbers in a random fashion e.g. Label encoder doesn't make it quantitative. It's still qualitative.
It just that most of the library implementation of ML Models only expect numeric data.
So, Label encoding will work fine with models that don't do any quantitative operations on any two encoded values i.e. Tree-based models that simply use the label to divide the space
If you use it with models that treat the mapping as quantitative e.g. Neural Networks then the result might be error-prone e.g. 10 will become 2 times 5 although no such relationship was intended.


Approximating density of test set in train

The approach that comes to mind, is to calculate the kullback-leibler divergence between the kernel density estimations of your train dataset and of your test dataset.

The kernel density estimation of each of your datasets will give you an approximation to the pdf's of your datasets. The kullback-leibler divergence will give you a number that will represent the divergence in bits from one distribution to another (if you use base 2 for your logarithm). Below are some references I think you would fine useful.

https://en.wikipedia.org/wiki/Kernel_density_estimation
https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/

If you would like me to show the math behind this method. Feel free to ask.

EDIT: Added math as asked by author of question

Let $\hat x_1, \hat x_2,\hat x_3...,\hat x_n$ be your training dataset while $x_1,x_2,x_3...,x_n$ is your testing dataset, where both $\hat x_i$ and $x_i$ belong to $\mathbb{R}^d$.
$$\hat f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-\hat x_i;H)$$
$$f(x;H)=\frac{1}{n} \sum_{i=1}^{n}K(x-x_i;H)$$
$\hat f$and $f$ represent the kernel density estimation for training set and testing set respectively. The parameter $H$ represents the bandwidth parameter and is a symmetric positive definite $d \times d$ matrix. $K(u;H)$ can be rewritten as $$|H|^{-\frac{1}{2}} K(H^{\frac{1}{2}} u)$$ where K can be any kernel function. I would recommend for simplicity purposes the standard multivariate normal kernel. Okay so now that we have the kernel density estimations of both our training and our testing dataset, we can use the kullback-leibler divergence in order to estimate the difference between the two.
Optimally we would like to calculate the kullback-leibler divergence with respect to every point in our space. Mathematically speaking.

$$\int_X f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})dx$$

But this is computationally unpractical to compute. We can approximate this integral by sampling a set of points from the $f(x;H)$ and then computing the discrete sum.
$$\sum_{x \in X} f(x;H) \ log_2(\frac{f(x;H)}{\hat f(x;H)})$$

Quick Note: To sample from a kernel density estimator, uniformly randomly select a point from the respective dataset. Then sample from the kernel of choice centered around the point chosen.


Replacing column values in pandas with specific column with multiple database operation?

You can use the python ternary operator with list comprehension:

df['Buckets(HRS)'] =
 ['<2hrs' if x<=120 else '2-4' if 120>x<=240 else '4-6' if 240>x<=360 else '>6' for x in db['Stoppage time in minutes']]


The type of your problem is time series regression. The common way of dealing with such problems is using ARMA models and Kalman filters with input (control vector). Look at this links:

https://www.sas.upenn.edu/~fdiebold/Teaching104/Ch14_slides.pdf

https://en.m.wikipedia.org/wiki/Kalman_filter

For time series regression using LSTM, check
Multivariate LSTM Forecast Model part from this link:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras


Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

The comments about iteration number are spot on. The default SGDClassifier n_iter is 5 meaning you do 5 * num_rows steps in weight space. The sklearn rule of thumb is ~ 1 million steps for typical data. For your example, just set it to 1000 and it might reach tolerance first. Your accuracy is lower with SGDClassifier because it's hitting iteration limit before tolerance so you are "early stopping"

Modifying your code quick and dirty I get:

# Added n_iter here
params = [{}, {"loss": "log", "penalty": "l2", 'n_iter':1000}]

for param, Model in zip(params, Models):
    total = 0
    for train_indices, test_indices in kf:
        train_X = X[train_indices, :]; train_Y = Y[train_indices]
        test_X = X[test_indices, :]; test_Y = Y[test_indices]
        reg = Model(**param)
        reg.fit(train_X, train_Y)
        predictions = reg.predict(test_X)
        total += accuracy_score(test_Y, predictions)

    accuracy = total / numFolds
    print "Accuracy score of {0}: {1}".format(Model.__name__, accuracy)

Accuracy score of LogisticRegression: 0.96
Accuracy score of SGDClassifier: 0.96


location of the resampled data from SMOTE

SMOTE is an algorithm used to generate " synthesize" new samples from the real samples.
It selects randomly one of the k-nearest neighbors, find the distance between these two pints , synthesize new point by modifying the sample considering the distance and a random number between 0 and 1.
SMOTE algorithm does not use samples from majority class only samples from minority.
It synthesizes new samples
It expects a high density minor class with small variation within the class
Anything else is related to the implementation, many implementation return only the synthesized samples


Are the image data augmentation generators in Keras randomly applied

I suggest having a look at the relevant documentation.
There, it states:

rotation_range: Int. Degree range for random rotations.
...
horizontal_flip: Boolean. Randomly flip inputs horizontally.

Saying each of the operations is applied randomly, I would say, means your images will be generated sometime with and sometimes without the augmentation steps, independently from one another.
If that doesn't convince you, here is the relevant snippet from the source code:
if self.horizontal_flip:
    if np.random.random() < 0.5:
        x = flip_axis(x, img_col_axis)

The code for the rotation step is a little more involved, but contained within the same class that I liked above.


Determining which categorical data is beneficial in predictive modelling

You can try two things -

Try finding the correlation between the Categories and the Target.
Since, It's between Categorial features and a Continuous Feature, you should -
Get the r-square Or Adjusted R-square score of Regression, see which one is best and drop the lowest few and try.
Read more - Kaggle

Calculate Feature Importance using random Forest.
 Read here - MachineLearningMastery


Find average sequence from a set of sequences

One way would be not to approach this as a calculation per session. Most data science solutions like to end up with a number, probability or classification. I suggest you structure your data differently so that you try to answer the question - what next action is likely given the last action.

In order to do this you would have to restructure your session data and use information from across all your sessions. For example, if you compare in how many sessions a player 'buys a gun', and if so record over all those sessions what their next action is, e.g. in 60% they 'play a mission' next. You will then have a probability of their next action based on the number of choices players made in all those sessions.

Once you have those probabilities, you will be able to answer the question, 'What comes next?'. This will in turn enable you to build that most average session that you are after by stepping through a session and building it by the most probable next step.


What's a good book (or other resource) to learn Imitation Learning?

I found a great book, it's just what I was looking for. In case someone finds it useful:

An Algorithmic Perspective on Imitation Learning: T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbel, & J Peters.


Absolutely, you could use bert pre trained model to suggest corrected sentences without annotated data as long as you are using a pre trained model. Here is a good repo that shows it
https://github.com/sunilchomal/GECwBERT
Hope it helps


Importing .ipnyb file from Kaggle into local Jupyter

Thats because "ipnyb" is not proper format. Try downloading/changing it properly to .ipynb extension.


Counting non-overlapping objects in a semantic segmentation prediction mask

This could be one of the approaches (if you are looking for the exact solution):

I assume that the image (your prediction) consists of only two values: 0 - black - roof, and 255 - white - other.

Then, this becomes an easy problem of counting the components of a graph:

Start a DFS from some black node which is not visited yet. Traverse all neighboring black nodes which are not visited yet. While moving, keep count of the number of black nodes you visited, and mark them as visited (so that you don't go back ...). Repeat the whole process if there are black nodes to start from (they are not already visited). This way you get the total number of roofs (this is the number of times you started a DFS from non-visited black node) and the total area of roofs (the number of black pixels you visited).

That is classical DFS problem. Here is an example in Python: https://eddmann.com/posts/depth-first-search-and-breadth-first-search-in-python/

This will work well, if you predictions are good. Actually, it will do what I have described, even if your predictions are bad, because it is an exact algorithm. But you have to check if your predictions are fine some other way.

If you are looking for different type of solution (i.e. not exact solution), the interesting approach would be to teach a model to count the number of roofs and estimate their area from an image. In order to do that, you would have to create a dataset with labeled images: for each image the label could be a total number of roofs on the image and the area of all roofs. So, that would be a regression problem.


Random-Forest-based Similarity Matrix for clustering: how does it behave?

I recently presented a poster at a conference where we used the same approach you describe for clustering. Generally, I think it's a great approach for clustering, as you can get clusters which have a distance that is relative to an outcome or variable that you are interested in.

For some insight, I have a few pointers:
1) When getting the co-occurence in trees, depending on how many subjects you have, this can end up being a very sparse matrix. To make the matrix less sparse you can increase the minimum number of samples required in terminal nodes.
2) After you get the similarity matrix, do a PCA on this, ending with individuals in rows and PCs in columns. Get the distance between individuals in this PCA space, restricting to some top number of components that you find acceptable. I recommend this because the similarity matrices can be huge if you have a lot of cases.


You can find a python script over here: https://github.com/gwding/draw_convnet

I hope it's helpful


How to model this "un predicatability" problem?

tl;dr: This is not a great candidate for machine learning solutions.

It is not exactly true that "the strength of an encryption scheme is measured by the randomness (unpredictability, or entropy) of the output".  Instead, strength of an encryption scheme is how resistant it is to cryptanalysis. Yes, input $\rightarrow$ output maps that show patterns are relatively easy to break. But the input $\rightarrow$ output map might be very random (high entropy) from many perspectives but still might be defeated through feasible cryptanalysis.

You offer another definition: "informally, the strength of an encryption scheme is determined by the number of such input-output pairs needed beyond which it becomes predictable."  This is usefully different from your first definition because it focuses on the information that might be gleaned about the encryption scheme from a stream of input $\rightarrow$ output pairs.  This is different and easier than the usual cryptanalysis problem where all you have is a data set of (encrypted) outputs.

Framed this way, I think you are heading toward a model selection problem.  While this may be relatively straight forward with weak encryption schemes, it becomes very hard, very quickly with strong schemes.  In fact, I believe you walk right in to the No Free Lunch theorem:

  "In formal terms, there is no free lunch when the probability distribution on problem instances is such that all problem solvers have identically distributed results. In the case of search, a problem instance is an objective function, and a result is a sequence of values obtained in evaluation of candidate solutions in the domain of the function. For typical interpretations of results, search is an optimization process. There is no free lunch in search if and only if the distribution on objective functions is invariant under permutation of the space of candidate solutions."

Translated: if you find yourself needing to search over all possible cryptanalysis methods, there exist input $\rightarrow$ output streams where no method is probabilistically better than any other. This will defeat any model selection method based on optimization of some objective function.


What algorithms will stuck in the local minimum?

Any greedy algorithm -- which try to maximize your objetive by choose whichever is locally best -- may get stuck in local minima.

These include, but are not limited to:

gradient descent optimization like neural networks, which try to gradually minimize the loss function by making small changes to the weights in the direction that minimize the loss. This also includes algorithms like gradient boosting. You can mitigate such problems by using second-order derivatives (Hessian) or smart heuristics.
one look-ahead algorithms like decision trees, which locally choose the feature $x$ and threshold $t$ whose split $x<t$ maximizes/minimizes a given metric like gini or entropy (and, as a subsequent, random forest) -- if a decision tree was able to see the final impact of all possible combinations, then it might make different choices, but that would be infeasible.
expectation-maximization (as used by e.g. k-means) is also heavily influenced by the initialization.

I am being a little imprecise in my listing. What defines if an optimization problem may get stuck in a local minima is the function that you are trying to maximize. If you use gradient descent to solve the SVM optimization problem, then you'll always converge to the global minimum. More important than the algorithm is the surface of the optimization problem (in SVM, the objective is a nice quadratic surface).

Also notice that algorithms like Random Forests (when the number of trees is very, very high) become almost deterministic, but they still suffer from getting stuck in local minima. I am point this out because some people confuse optimizations being stuck in local minima with the algorithms being stochastic -- they reason that they can mitigate local minima by making multiple runs of the algorithm. But the algorithm being vulunerable to local minima $\neq$ the algorithm being stochastic. You'll never e.g. fix the one look-ahead problem by running a decision tree or a random forest several times. For algorithms like k-means, it is indeed a good idea to run it several times, as most implementations do behind-the-scenes.


SVM is not fitted when tried to fit it into a model

15 attributes with 30161 instances are not much to cause this delay, I've trained with much more. Your system config also plays a major role in this. For instance if you are using anaconda on chrome and you have multiple processes going on then it would cause delay as distances are calculated on the fly which will be heavy on memory.

You can also try SGD as you are going for linear. It is faster than SVM.


Time Series Classification with multiple rows per date

This problem can be modeled with a standard binary classification since you labeled data. Time can be considered a feature which can be encoded relative to outcome date. Random forest would be a good algorithm to try.
You need to rearrange the data to be tidy data where each LEAD_ID is in a single with row with all features for that id in the same row.


Addition and subtraction of values of two columns when using Orange GUI

Use Feature Constructor like so:

and X1 will be the name of your new feature (you can obviously change it)


Is Java or Python a better choice for an application involving data intensive algorithms employing natural language processing?

I work with python and java in big-data settings every day.

python is definitely my language of choice when researching, experimenting and fitting models.
python has a ton of very useful libraries such as spacy, nltk and sklearn that makes exploration very easy, especially in within a jupyter notebook.

When running the code in production, If performance is a consideration, we implement only the prediction phase in java.

java is more suited for production, has great support for packaging, tests, deployment.


online detection of plateaus in time series

Why don't you use a Shewhart Control Chart (more modernly referred to as 6-sigma) with a moving variable length window to understand the process that you have.

As you find your process variability decreases markedly, by having too many results within $±1σ$ then you have a "process change" and that might be defined as a plateau. You will need to decide how many observations you require to establish a plateau.

The as you find your variability increases and your $±3σ$, you know you again have a process change and need to recalculate your limits.


Image feature extraction Python skimage blob_dog

Please read through the scikit documentation that is found here and I am assuming that you have gone through the method through which it calculates blobs in the images. If not the link is here. It return the 2D array of arrays with 3 values in each array, giving coordinates and std.deviation of Gaussian of the blob found. The length of the array returned is the number of blobs it has found in the image. So eliminating elements from that array by resizing it, is a wrong way of approach and may eliminate the features which you might find important.


What are the possible ways to handle class unbalance in a large scale image recognition problem with Deep Neural Nets?

You can also modify your cost function so that instances in the smaller class have more weight. This makes training computationally less demanding, when compared to your option 1.


You Can do one thing as per my knowledge take the difference Between two response times and plot them in a line graph which ever have the more deviation from the 0 line you can say they are taking more time


How many video streams can single GPU handle for object detection

Since batch training is nothing else than computing multiple samples at once, you could test what the maximum batch size your GPU can handle for the model and that is how much you could calculate at once.
With regards to FPS it really depends on what GPU you have and how big the model is. Just try it out.


Extracting structured data from semi structured data

I am making the following assumptions:

The first column of numbers is the first few digits/numbers in each line
The last column of numbers is the last few digits/numbers in each line
So, in the second column, texts like MALL1ETAGE ZARA1st FLOOR should not have numbers in the front or end. So, 20215652671MALL1ETAGE ZARA1st FLOOR 2345561 will be interpreted as 20215652671, MALL1ETAGE ZARA1st FLOOR, 2345561 and NOT 2021565267, 1MALL1ETAGE ZARA1st FLOOR, 2345561

To this end, you can use the following that makes use of regular expression to capture the first and last group of numbers:
import re

re.findall("^([0-9]+)(.*?)([0-9]+)$", "2021565267MALL1ETAGE ZARA1st FLOOR 2345561")
# [('2021565267', 'MALL1ETAGE ZARA1st FLOOR ', '2345561')]

Applying strip() to each of the result removes the trailing whitespace to get the output you desire.


Since you're only interested in the gender affinity of businesses in your dataset, no model is needed.  You can compute gender affinity directly:  For each business, find the total number of customers, the total number of male customers, and the total number of female customers.  Then $Pr(Male) = \frac{males}{total}$ and $Pr(Female) = \frac{females}{total}$


How to Build Mobile Application for Image Recognition?

I would advise you to use kivy for Python. It has an active community and there is also a book on this topic Practical Computer Vision Applications Using Deep Learning with CNNs With Detailed Examples in Python Using TensorFlow and Kivy. Kivy is very easy for newbies and you can develop multiplatform applications (Windows, iOs, Android).


Do the labels have to start from 0?

No it doesn't matter where they start as long as they have distinct values.

Do the labels have to be sequential?

Well it depends from the feature. For example if you have features that are showing order of magnitude, like small<big<vast, then yes the order matters and they are called ordinal features, but if the feature's values represent for example countries then there is no such thing as order, so probably one should use OneHotEncoder, in order to be equally distanced in space. (see here)

What happens if I replace all label 0s with 3 so that my labels are 1-2-3 instead of 0-1-2 (This is done before training)

Except the previous bullet, one should consider the type of model that will use. For example tree based model like RandomForest work very well with categorical data, and the numerical value of a category could be arbitrary. But this is not the case for the linear models.

Closing if you want to convert a categorical feature to numerical values, you should consider two things the features values (ordinal?) and the type of model.
P.S. To improve the performance of the model there are many way to convert categorical to numerical features, like Target encoding techniques, that have been showing to improve also tree based classifiers, but perhaps this is a conversation for another time :)


XML is a markup language similar to html. One uses tags with attributes to build data structures. For example,

<sampleXML>
  <Menu>
    <Food>
      <item1>Spaghetti Bolognese</item1>
      <item2>Spaghetti Carbonara</item2>
    </Food>
    <Drinks>
      <item1 class = 'drinks'>Sprite</item1>
    </Drinks>
  </Menu>
</sampleXML>

As you can see XML employs tags such as <Food></Food> and attributes such as class = 'drinks' which is exactly what HTML has. To access the XML data in java you got couple choices. You can read it in as a string and parse it using built in DOM parser. Or you can use JAXB to map XML directly to Java objects.

Surely you can convert XML to a csv file. There are free websites online for that. Just google "XML to csv converter."

Binary files are not XML, but can be. This needs little explanation. The letters and words you see here are ASCII characters. This is human readable text. Each ASCII character has a binary representation. For example, j in binary is 1101010. A binary file is any file in computer language (0 an 1). A binary file can also be a combination of text and binary. You can convert binary to ASCII and those files you download may indeed be XML I described above. To convert binary to ASCII just google it.


What happens to a machine learning technique (specifically Decision Tress and Logistic Regression) if the validation dataset has a new category?

There are no categorical value support in the decision trees used in scikit-learn. Either the values are just numbers, or they have been one-hot-encoded.

One-hot encoding with an unseen input value will change the width of the feature array, which will give an error when running decision_tree.predict(X).

Without one-hot encoding the output will most likely be the same as for the value closest to it. Because the tree learns rules on form feature[2] > 3 to distinguish say a value of 3 from a value of 4 (during training). So a value of 4,5,6... etc will be treated the same way.
There are no feature[2] > 4 or feature[2] < 5 rules, since there was no data for that to cause splits that could be evaluate/optimize during training.

So if you want an error also when not using one-hot-encoding, you will have to check the range of features before you pass it to the Decision Tree.


Gas consumption outliers detection - Neural network project. Bad results

Just an idea - your data is highly seasonal: daily and weekly cycles are quite perceptible. So first of all, try to decompose your variables (gas and electricity consumption, temperature, and solar radiation). Here is a nice tutorial on time series decomposition for R.

After obtaining trend and seasonal components, the most interesting part begins. It's just an assumption, but I think, gas and electricity consumption variables would be quite predictable by means of time series analysis (e.g., ARIMA model). From my point of view, the most exiting part here is to try to predict residuals after decomposition, using available data (temperature anomalies, solar radiation, wind speed). I suppose, these residuals would be outliers, you are looking for. Hope, you will find this useful.


Use Python to Scape Data on website ends with aspx

You can use selenium for this crawl. When you hit that the link first time, it will ask to accept the terms and conditions then you need to click on the checkbox. After that, it will redirect to page, just pass the input to your current URL and read the HTML element using XPath or regex.


What is the difference between upsampling and bi-linear upsampling in a CNN?

In the context of image processing, upsampling is a technique for increasing the size of an image.

For example, say you have an image with a height and width of $64$ pixels each (totaling $64 \times 64 = 4096$ pixels). You want to resize this image to a height and width of 256 pixels (totaling $256 \times 256 = 65536$ pixels). In the new, larger image you only know the value of $1$ out of every $16$ pixels. How are you going to calculate the values for the rest?

Well, the methods that do that for you are called upsampling techniques. The most common are:

Nearest-Neighbor: Copies the value from the nearest pixel.
Bilinear: Uses all nearby pixels to calculate the pixel's value, using linear interpolations.
Bicubic: Again uses all nearby pixels to calculate the pixel's values, through polynomial interpolations. Usually produces a smoother surface than the previous techniques, but its harder to compute.
Other more complex resampling algorithms, e.g. Lanczos.

An article explaining the differences among image resampling techniques can be found here.


MXNet Julia Package - flexible and efficient deep learning in Julia

https://github.com/dmlc/MXNet.jl

Pros

Fast
Scales up to multi GPUs and distributed setting with auto parallelism.
Lightweight, memory efficient and portable to smart devices.
Automatic Differentiation

Cons

Doesn't have yet low level operations for algorithm implementation. But they are working on this issue (https://github.com/dmlc/mxnet/issues/586)


Cross Correlation Between Input-Output Sine Waves

These formulas are discrete fourier transform.

It's very standard procedure when you analyze wave-alike processes. What it does it transforms the signal from the time-space to the frequencies-space. The idea is that you take sin-wave with specific frequency, multiply it by signal and if the value is big then you have something similar to the wave of this frequency.

$\frac{1}{NT}$ term is just overall time of the sequence (N samples with length T)

The $I_{s}$ and $I_{c}$ is amplitude(intensity) for the sin and cos, respectively. You get them for the specific frequency $\omega$ and it's a single number for each. From this you can get full amplitude (G) and phase $\phi$. Why do you need phase? Because it's important - if you have two waves with similar frequency and $\phi = \pi$, then they would cancel each other, and if they have same phase, they would double.


Intuitively, the correlation matrix is symmetric because every variable pair has to have the same relationship (correlation) whether their correlation is in the upper right or lower left triangle. It doesn’t make sense to say the correlation between variables $X_1$ and $X_2$ is $\rho$, but the correlation between $X_2$ and $X_1$ is $\rho’\neq \rho$ if calculating a Pearson correlation (so correlation is symmetric).

Mathematically, correlation between two variables, $X$ and $Y$, is commutative: $Corr(X,Y)=Corr(Y,X)$.

In OP’s case, the correlation between Q1 and Q2 is the same calculation and therefore the same result as the correlation between Q2 and Q1. Therefore the correlation matrix will be symmetric.

There are more mathematical reasons and proofs why a correlation matrix of real valued variables has to be symmetric and positive semi-definite, but I’ve excluded them from this answer.


How to do spelling correction for a language but also correct some words in other language

There are many ways to build a spell corrector. One of the simplest is:

Detect an incorrect word
Generate candidate suggestions
Score and rank the candidate replacements

For detecting an incorrect word, a simplifying assumption is that any word not a dictionary is a spelling error. Otherwise, you have to build a separate model to detect if a word is a potential spelling mistake based the current context.

To generate candidate words, you need to find dictionary words that are similar to the incorrect word. That requires defining "similar", typically similar is measured by edit distance (e.g., deletes, transposes, replaces, or inserts of individual characters).

For scoring and ranking, probabilities of candidate words are weighted by a language model and channel model. A language model weights how likely a word will appear in the current context. A channel model reflects if an error happens depending on how the word is transmitted (e.g., full computer keyboard errors are different from mobile phone errors).

For your example, you need to construct a custom dictionary of all possible words and a corpus reflecting the frequency of occurrences for the custom dictionary words. If possible, construct an error model of common mistakes.

Peter Norvig goes into greater detail here.

Later you can add Deep Learning methods, but it is better to start with a simpler approach that requires less data.


How to select algorithms for ensemble methods?

In general in an ensemble you try to combine the opinions of multiple classifiers. The idea is like asking a bunch of experts on the same thing. You get multiple opinions and you later have to combine their answers (e.g. by a voting scheme). For this trick to work you want the classifiers to be different from each other, that is you don't want to ask the same "expert" twice for the same thing.

In practice, the classifiers do not have to be different in the sense of a different algorithm. What you can do is train the same algorithm with different subset of the data or a different subset of features (or both). If you use different training sets you end up with different models and different "independent" classifiers.

There is no golden rule on what works best in general. You have to try to see if there is an improvement for your specific problem.


Classification for High dimensional data

Before answering your question I would ask if you actually need so many variables or you can apply some dimensionality reduction technique (i.e. PCA)? It could boost your training speed and maybe result in a better model efficiency.

Lets suppose for some reason (i.e. model explainability) you need the exact variables. You can still remove some of them without using feature importance (i.e. removing highly correlated ones, but there are other alternatives too). Indeed using only feature importance to remove variables can be dangerous. Imagine there is an important factor that highly influence your model but it shows up in several variables. In this case it can happen that the importance of this factor will be distributed among those variables. You will experience maybe that those variables are less important and remove them all thereby also eliminating an otherwise important factor.

To answer your question. I would before step 1 take away my test set and only use the training set for any kind of modeling activity. It's a good practice to guarantee that your test set has never been seen by the model and it did not influence it in any way.

In this case you can just remove less important features and measure the final results on the test set, it's not necessary to do CV.

On the other hand you will probably face with the challenge how to define your threshold for eliminating variables based on feature importance.You can try to do some gridsearch on it. In this case you should avoid measuring the results of different models on the test set or you risk over-fitting on it. In this case using CV can be a good solution to define your threshold.


Python function returning a 4x4 matrix instead of a floating number like in an equivalent Octave function in a Linear Regression problem

Change the compute_cost function to the following to match dimensions for matrix multiplications:
def compute_cost(X, y, theta):
    m = len(y)
    ans = np.dot(X.T, theta).T - y
    product = np.dot((1/(2*m)), np.dot(ans, ans.T))
    return product[0, 0]

Result for the data given:
32.0727338775
54.242455082

PS: one can simplify the transpose operations further..


How will a rotation matrix affect contestants in machine learning contests?

Kaggle competitions with clean, anonymised and opaque numerical features are often popular. My opinion is they are popular because they are more universally accessible - all you need is to have studied at least one ML supervised learning approach, and maybe have a starter script that loads the data, and it is very easy to make a submission. The competitions become very focused on optimising parameters, picking best model implementations and ensembling techniques. The more advanced competitors will also refine and check their CV approaches very carefully, trying to squeeze the last iota of confidence out of them in order to beat the crowd climbing the public leaderboard.

Examples of historic Kaggle competitions with obfuscated data might be Otto Group Product Classification or BNP Paribas Cardif CLaims Management. For some of these competitions the data is adjusted for anonymity of the users who might otherwise be identified from the records. In other cases it is less clear what the sponsor's motivation is.

However, there are negative consequences (you will find these complained about in the same competitions):

Use of insight from domain knowledge, or exploration/study of the underlying principles from the subject being predicted are effectively blocked. It is hard to assess the impact of this, but it is possible that the sponsors miss out on potentially better models.
Doing "just" the machine learning side can be a bit too mechanical and boring for some competitors, who may not not try as hard.

  How much do contestants reply on statistical analysis on the (raw, i.e. non-PCA-ed) input features?

There are always data explorations and views of data published in forums (and Kaggle's scripts - called kernels), and many people view, upvote and presumably use the insights from them. I recall at least one competition forum thread where there was a lot of discussion about weird patterns appearing in data, which were probably an artefact of obfuscation (sorry I cannot find the thread now).

With obfuscated data, there can be attempts to de-obfuscate, and they have sometimes been partially successful.


Python sklearn PCA transform function output does not match

If you look at the source code, the PCA is calculated through the SVD. I believe it iterates until "good enough."
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/decomposition/pca.py


Extract person's academic degree from text

I would have started with a regexp to find all degrees in the email and then compare them to take the highest one (you can order them) ex. Masters > Bachelor. This might be a little bit too easy but worth try.