sklearn.cross_validation.cross_val_score "cv" parameter question

From the sklearn docs for cross_val_score's cv argument :

For int/None inputs, if the estimator is a classifier and y is either
binary or multiclass, StratifiedKFold is used. In all other cases,
KFold is used.

I believe that in the first case, StratifiedKFold is being used as the default. In the second case, you are explicitly passing a KFold generator.
The difference between the two is also documented in the docs.

KFold divides all the samples in $k$ groups of samples, called folds (if
$k = n$, this is equivalent to the Leave One Out strategy), of equal
sizes (if possible).
[...]
StratifiedKFold is a variation of k-fold which returns
stratified folds: each set contains approximately the same
percentage of samples of each target class as the complete set.

This difference in folds is what is causing the difference in scores.
As a side note, I noticed that you are passing a random_state argument to the KFold object. However, you should note that this seed is only used if you also set KFold's shuffle parameter to True, which by default is False.


What does updated alpha mean in LDA model?

The alpha is a hyperparameter which controls the mixture of topics for any given document. Turn it down and the documents will likely have less of a mixture of topics. Turn it up and the documents will likely have more of a mixture of topics.

Can I use the updated alpha to determine whether the topic number is
appropriate?

Tuning it depends upon your problem statement which could be solved with help of domain knowledge.


You made some mistakes on MinMaxScaler.

MinMaxScaler shouldn't be fitted twice(as internal parameters inside MinMaxScaler will be changed), and dataX & dataY should have their own scaler(as they have different minimum and maximum values.

Try to do something like this:

self.x_scaler = MinMaxScaler()
self.y_scaler = MinMaxScaler()

dataX = self.x_scaler.fit_transform(dataX)
dataY = self.y_scaler.fit_transform(dataY)

self.model.fit(dataX, dataY, nb_epoch=1000)

data = numpy.array([[val1, val2, val3]])
data = self.x_scaler.transform(data)
prediction = self.model.predict(data)
real_prediction = self.y_scaler.inverse_transform(prediction)


Sklearn ValueError: X has 2 features per sample; expecting 11

X = dataset.iloc[:, :11].values

It is asking for 2 features per sample. But you are passing the 11. Do it like this

X = dataset.iloc[:, :2].values


Can we call a CMS like Drupal Machine Learning?

No: a CMS, like many other software tools, processes data. But this process is predefined, it's not "learnt" from the data.
Machine Learning is supposed to involve at least one step where the system "learns" from the data, i.e. uses some algorithmic process to determine the characteristics (patterns) inside the data. In supervised learning there is a specific training stage where the model "learns", which is followed by a testing stage where the model is applied to some new data. In unsupervised learning, learning and applying happen in a single stage: the model discovers the patterns and directly produces an output based on these pattern.


How to create column for my csv file in python

You can use Pandas for this, your file format isn't exactly comma separated values file. But still you can us pandas read_csv() method.
Suppose you're file name is test_file

import pandas as pd
df = pd.read_csv('test_file', sep=':', header=None)
>>> df
   0      1
0  I    30n
1  J     0n
2  J     0n
3  U  1000n
4  C     0n
5  I    12n
6  I    10n
7  I    10n
8  I    10n
9  I    10n

Then you can use the pivot() function:

>>> df.pivot(columns=0)
\     1
0    C    I    J      U
0  NaN  30n  NaN    NaN
1  NaN  NaN   0n    NaN
2  NaN  NaN   0n    NaN
3  NaN  NaN  NaN  1000n
4   0n  NaN  NaN    NaN
5  NaN  12n  NaN    NaN
6  NaN  10n  NaN    NaN
7  NaN  10n  NaN    NaN
8  NaN  10n  NaN    NaN
9  NaN  10n  NaN    NaN

If your intention is to write it back to a file you can use to_csv() method.

# this row eliminates the level headers of the columns at level 0
>>> df.columns=df.columns.get_level_values(1)
>>> df
0   C    I   J      U
0      30n
1           0n
2           0n
3               1000n
4  0n
5      12n
6      10n
7      10n
8      10n
9      10n
>>> df.to_csv('new_test_file', index=False)

OR

If you wish to make it less sparse, you can first turn it into a dict and then back to DataFrame:

>>> _dict = df.groupby(0)[1].apply(list).to_dict()
>>> _dict
{'C': ['0n'], 'I': ['30n', '12n', '10n', '10n', '10n', '10n'], 'J': ['0n', '0n'], 'U': ['1000n']}
>>> pd.DataFrame.from_dict(_dict, orient='index')
       0     1     2     3     4     5
C     0n  None  None  None  None  None
I    30n   12n   10n   10n   10n   10n
J     0n    0n  None  None  None  None
U  1000n  None  None  None  None  None

>>> pd.DataFrame.from_dict(_dict, orient='index').T
      C    I     J      U
0    0n  30n    0n  1000n
1  None  12n    0n   None
2  None  10n  None   None
3  None  10n  None   None
4  None  10n  None   None
5  None  10n  None   None

pd.Series.to_dict()
pd.DataFrame.from_dict()
pd.DataFrame.T


How can I load file from my local drive in keras using get_file?

get_file() is used for loading files from a URL,hence it can not load local files. If you have mounted you gdrive and can access you files stored in drive through colab, you can access the files using the path '/gdrive/My Drive/your_file'.
for eg:

base_dir ='/gdrive/My Drive/weld_data'

train_generator = datagen.flow_from_directory(
base_dir,
target_size=(IMAGE_SIZE, IMAGE_SIZE),
batch_size=BATCH_SIZE,
subset='training')

If you need to unzip the file, you can use
    import zipfile
    with zipfile.ZipFile(your_file, 'r') as zip_ref:
        zip_ref.extractall(directory_to_extract_to)

I used '/content' as the directory_to_extract_to.

Then you can access the data the usual way.

base_dir = '/content/my_folder'

train_generator = datagen.flow_from_directory(
base_dir,
target_size=(IMAGE_SIZE, IMAGE_SIZE),
batch_size=BATCH_SIZE,
subset='training')


Your code looks decent, although this is not really the place to ask these types of questions. You're looking to write a wrapper around SK-Learn,

What I would try to do is to write a method to handle different types of datasets(json or csv) and then call it on a filepath and then have a function which reads in a set of models and applies each of them to the dataset in question, and finally uses metrics present in sklearn to get some results.

There is auto-sklearn which does this for you and they have a pretty solid codebase that is well documented. Here is a link for your convenience (https://automl.github.io/auto-sklearn/stable/)


How can I imporve accuracy for text classification and mapping using SVM?

A general good approach for high dimensional text data would be to use a word embedding and neural networks. If you want to stick to the models and approach you are using, I suggest you don't over-engineer your dataset and first try with simple bag of word approaches and while cross-validating try different n-grams, pos-tagging but I suspect the very high dimensional feature space is bringing your accuracy score down.


can machine learning/Deep learning used to minimize an objective function?

We actually wrote this for an engineering company. Basically we modeled the project business processes, then decomposed them to metrics for analysis. I can high level three cost savings areas we found.

Purchasing: use receipts, billing, and time metrics to identify outliers and reconciliation failures (receipts != payments). Outliers/missing may indicate mistakes or fraud/theft/overcharging in billing (ie:$200 hammer). Used filter functions to find.

Contract and Permit analysis: make sure all required permits and contract provisions are in place. Used text classification and boolean (exception reporting, really).

Project length prediction: if a project is shorter or longer than others in it's class, investigate. Used statistical analysis (the client already classified: public/private, plumbing/sewage, location, etc).

This was before TensorFlow, which likely could improve on our results. You will find that there are plenty of features (metrics) to model in these processes.

So hopefully this can help on model definition and  consequently the objective functions.


assignment of observations to known size classes

You can potentially model this Information in form of a Prior, depending how you proceede with modelling. Check out bayesian statistics


Why are there over 600 open pull-requests in Scikit-learn github repo?

mean, reviewing a pull-request is not that big of a deal compared to writing commits and also this normally has higher priority than writing code, so why I see new commits in the project, but PRs are still not reviewed?

I assume that it's not that simple if they want to maintain the stability and the homogeneity of their code base. If they were just quickly accepting PRs the whole software would become an unmanageable mess in no time, causing problems for a large community of users. It's almost unavoidable for any significant software project to become harder and slower to maintain and develop when it reaches a certain size, especially if many users depend on it.

Also it's maintained by a community of volunteers so there might not be that many contributors who are both knowledgeable enough about the code and have enough free time to study all the PRs coming regularly.


Simply counting the frequency of characters ought to easily distinguish between English language and ciphertext, because they're so obviously different.

You can just count the frequency of characters in a big corpus of English, and a big corpus of ciphertext, and apply a chi-squared test to each to figure out which one matches the counts in a new chunk of text.

Or if you can assume ciphertext has a roughly uniform distribution over characters, that alone lets you construct a good test for whether new text is unlikely to be ciphertext.

I did a short blog post on something similar. https://blog.cloudera.com/blog/2016/09/solving-real-life-mysteries-with-big-data-and-apache-spark/


Trained BERT models perform unpredictably on test set

BERT-style finetuning is known for its instability. Some aspects to take into account when having this kind of issues are:

The number of epochs typically used to finetune BERT models is normally around 3.
The main source of instability is that the authors of the original BERT article suggested using the Adam optimizer but disabling the bias compensation (such a variant became known as "BertAdam").
Currently, practitioners have shifted from Adam to AdamW as optimizer.
It is typical to do multiple "restarts", that is, train the model multiple times and choose the best performing one on the validation data.
Model checkpoints are normally saved after each epoch. The model we chose is the checkpoint with best validation loss among all epoch of every restart we tried.

There are two main articles that study BERT-like finetuning instabilities that may be of use to you. They describe in detail most of the aspects I mentioned before:

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
Revisiting Few-sample BERT Fine-tuning


Autoencoder for cleaning outliers in a surface

A possible approach would be a denoising autoencoder. It is like a normal autoencoder but instead of training it using the same input and output, you inject noise on the input while keeping the expected output clean. Hence, the autoencoder learns to remove it. This kind of autoencoders are also described in the blog post you linked to.

In your case, you could just train your denoising autoencoder injecting to the inputs spikes of the height you expect to be removed.

About what kind of architecture (e.g. fully connected, convolutional), only actual tests can tell you what is appropriate and what is not.


What if My Word is not in Bert model vocabulary?

The problem is that you are not using BERT's tokenizer properly.
Instead of using BERT's tokenizer to actually tokenize the input text, you are splitting the text in tokens yourself, in your token_list and then requesting the tokenizer to give you the IDs of those tokens. However, if you provide tokens that are not part of the BERT subword vocabulary, it will not be able to handle them.
You must not do this.
Instead, you should let the tokenizer tokenize the text and then ask for the token IDs, e.g.:
tokens_list = tokenizer.tokenize('Where are you going?')

Remember, nevertheless, that BERT uses subword tokenization, so it will split the input text so that it can be represented with the subwords in its vocabulary.


This is a learning to rank problem, in particular feature selection for learning to rank.

There are many ways to solve the problem. One common method is maximizing feature importance while minimizing similarity. In other words, rank documents based on the most valuable and unique features. This is a variation of Minimum-redundancy-maximum-relevance (mRMR) feature selection.

To maximize feature importance, first rank all documents using all features, evaluate performance with evaluation measures, and show documents with the highest importance scores.

To minimize similarity, find features that have the least redundancy. Redundancy could be measured as the average value of all mutual information values between each feature pair.

"Feature Selection for Ranking" goes into greater detail.

Once the features are ordered with mRMR, the user can select specific features. Then re-rank documents based on the user-selected features.


Theano/Lasagne/Nolearn Neural Network Image Input

Just out of curiousity: why use 3-channel images? I work in CV as well and from what I've seen, it's standard to grayscale the images. Especially with MNIST, which is not colored to begin with, it doesn't seem that using color has any benefit.

I also believe that there's an intuitive reason to use grayscale - often times color is a confounding variable. If you were to look at a red "1" and and blue "1", you'd say "Hey! Those are both ones! They're just different colors". However, computers are much stupider than you or I. Even with intelligent algorithms, a computer may still end up in the situation where it's looking at a blue "1" and say to itself, "Oh crap, what is that?!? Why is it blue? I've never seen anything like this before!" Keep in mind how large of a difference there is from the color (255, 0, 0) and (0, 255, 255) and how slight the differences in images can be (especially outside the comfortable playground of MNIST).

At any rate, most of the NoLearn examples I've seen use data shapes like:

X = X.reshape(-1, 1, size, size)

If you grayscale your images you can coerce it into this shape. Unfortunately, I'm not sure of how you'd cram your colored data into noLearn and get the results you're looking for.


You can run DBSCAN without storing the distances in a matrix. This has the drawback that each time you visit a point, you have to recalculate all the relevant distances, which requires more time. However, the space complexity stays $O(n)$, since the only things you have in memory at any single time are the positions of the n points, their various labels, the neighbors of the current point and the neighbors of a particular neighbour in the case that the point turns out to be a core point.


Naive about which Naive Bayes in article

First off, yes there are different Naive Bayes algorithms. But they are all based on the same principle, namely Bayes Theorem where the features are assumed to be independent.

Here is a short guide on when to use which for spam detection (or document classification in general for that matter):

Bernoulli Naive Bayes

  Each e-mail is represented as a binary vector, and in this case, whether a word is present in the e-mail matters as much as if it isn't. For instance, the fact that "viagra" is in the e-mail might mean it's "spam" and the fact it isn't might mean it's "not spam"

Multinomial Naive Bayes with boolean features

  Each e-mail is represented as a binary vector, and in this case, whether a word is present in the e-mail matters more than if it isn't. For instance, if "grandma" is in the e-mail, it might mean that it's "not spam", but if it isn't in the e-mail, it doesn't necessarily mean that it's "spam".

Multinomial Naive Bayes with term frequency

  Each e-mail is represented by how many times each word occurs.

Gaussian Naive Bayes

  This deals with continuous values, so this isn't applicable here.


Examples on kohonen self organizing maps

There are alternative solutions for self organizing maps.

Best of them I found pymvpa where the example is easy to read and understand. It is also maintained quite activately as you can see from their Github.

I tried to run the kohonen 1.1.2 test file, but it did not run after two days of trying. So, let's have a try for the another solution.

To run pyMVPA example som.py, you have to do the following (at least):

1 install some pre-requisites: at least numpy, scipy, nibabel and swig to be able to run setup.py
2 run the commands:
   python setup.py build
   python setup.py install
3 modify example under doc/examples (som.py) to have
 import matplotlib.pyplot as pl
4 run command:
   python som.py

Note that this works only in python 2, not python3 can be used as command.


What is "posterior collapse" phenomenon?

With the help of better explanations provided in Z-Forcing: Training Stochastic Recurrent Networks:

When posterior is not collapsed, $z_d$ (d-th dimension of latent variable $z$) is sampled from $q_{\phi}(z_d|x)=\mathcal{N}(\mu_d, \sigma^2_d)$, where  $\mu_d$ and $\sigma_d$ are stable functions of input $x$. In other words, encoder distills useful information from $x$ into $\mu_d$ and $\sigma_d$.

We say a posterior is collapsing, when signal from input $x$ to posterior parameters is either too weak or too noisy, and as a result, decoder starts ignoring $z$ samples drawn from the posterior $q_{\phi}(z|x)$.

The too noisy signal means $\mu_d$ and $\sigma_d$ are unstable and thus sampled $z$'s are also unstable, which forces the decoder to ignore them. By "ignore" I mean: output of decoder $\hat{x}$ becomes almost independent of $z$, which in practice translates to producing some generic outputs $\hat{x}$ that are crude representatives of all seen $x$'s.

The too weak signal translates to
$$q_{\phi}(z|x)\simeq q_{\phi}(z)=\mathcal{N}(a,b)$$
which means $\mu$ and $\sigma$ of posterior become almost disconnected from input $x$. In other words, $\mu$ and $\sigma$ collapse to constant values $a$, and $b$ channeling a weak (constant) signal from different inputs to decoder. As a result, decoder tries to reconstruct $x$ by ignoring useless $z$'s which are sampled from $\mathcal{N}(a,b)$.

Here are some explanations from Z-Forcing: Training Stochastic Recurrent Networks:

  In these cases, the posterior approximation tends to provide a too
  weak or noisy signal, due to the variance induced by the stochastic
  gradient approximation. As a result, the decoder may learn to ignore z
  and instead to rely solely on the autoregressive properties of x,
  causing x and z to be independent, i.e. the KL term in Eq. 2 vanishes.

and

  In various domains, such as text and images, it has been empirically
  observed that it is difficult to make use of latent variables when
  coupled with a strong autoregressive decoder.

where the simplest form of KL term, for the sake of clarity, is
$$D_{KL}(q_{\phi}(z|x) \parallel p(z|x)) = D_{KL}(q_{\phi}(z|x) \parallel \mathcal{N}(0,1))$$
The paper uses a more complicated Gaussian prior for $p(z|x)$.


Could someone please offer me some guidance on some kind of particular, SPECIFIC project that I could attemp, to "get my feet wet, so to speak"

I would suggest Kaggle learning projects - http://www.kaggle.com/competitions

Look for the ones in the 101 section that offer knowledge. There's many pre-made solutions ready, which you can ingest and try variations of.

Also, I have bookmarked a Comprehensive learning path – Data Science in Python, which among other things gives a few answers to your specific question.


I recently had a similar problem (removing abnormal peaks from a time series). That's what I suggest you:

Get the smoothed trend. There are several techniques you can employ, such as various forms of exponential smoothing.
Find the difference between actual trend observations and smoothed ones.
Normalize this distribution of distances (using Z-score, i.e. sklearn's StandardScaler)
Substitute the observations that lie k standard deviations away from the mean (that is 0). The choice of x can be arbitrary or data-driven; in my case I chose k = 3 (i.e. a very conservative anomaly removal). You can use the smoothed values as a substitute. In your case, the interpolation could be perhaps a good choice (that's up to your preference).

This will automatically remove abnormal peaks like the one you displayed.


Time Series Classification for 1 hour blocks

Correct me if I am wrong, but the problem you describe here sounds like a classification problem, not a times series forecasting. You, just want to know to what class each 1 hour of data belongs to. If this is the case, you can try using a CNN with 1 dimensional convolutions and 3 channels.


Is converting a categorical value into numerical needed to find a correlation?

Your categorical feature is an ordinal one meaning it conveys an order between the levels that represent it. So you can convert it to a numerical feature, however the difference between 2 consecutive levels is unknown and cannot be translated into numbers.

Example : High - Medium - Low --> 3 - 2 - 1 or 4 - 2 - 1 or 5 - 2 - 1.
See? The order is preserved but the difference between the levels is arbitrary.

Also you oughta now that picking values for your feature affect correlation due to the fact that you're free to pick any values you want for your categorical feature.


What does it mean when people say a cost function is something you want to minimize?

No, it means you are trying to find the inputs that make the output of the cost function the smallest. It doesn't mean that you should "minimize" use of it.


Difference: Replicator Neural Network vs. Autoencoder

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output is measured - outliers are expected to have a higher reconstruction error.

The main difference seems to be the way how the input is compressed:

Plain autoencoders squeeze the input through a hidden layer that has fewer neurons than the input/output layers.. that way the network has to learn a compressed representation of the data.

Replicator neural networks squeeze the data through a hidden layer that uses a staircase-like activation function. The staircase-like activation function makes the network compress the data by assigning it to a certain number of clusters (depending on the number of neurons and number of steps).

From Replicator Neural Networks for Outlier Modeling in
Segmental Speech Recognition:

  RNNs were originally introduced in the field of data compression [5].
  Hawkins et al. proposed it for outlier modeling [4]. In both papers a
  5-layer structure is recommended, with a linear output layer and a
  special staircase-like activation function in the middle layer (see
  Fig. 2). The role of this activation function is to quantize the
  vector of middle hidden layer outputs into grid points and so arrange
  the data points into a number of clusters.


Can i use chi square test to remove a particular variable from the model?

What you described is the Filter Method. One of the three methods of Feature Selection. The most common mistake that one could do is to make the feature selection as an independent step on your whole process and then decide which model he will use. The most appropriate way to do it is to include your Feature Selection process in your accuracy test and try different kind of feature selections. The best way for the feature selection is the domain knowledge. If you don't have it, then you start working with one of the three methods, like the Filter Method.

What you need to do is to test the chisquare or mutual information on your features with your label column. This will give you which of the features have low effect on your prediction and you can remove it.

You can find a briefly detailed article about Feature Selection here and here.


Check out Applied Data Science with Python Specialization from Coursera. It is a series of 5 courses from the University of Michigan.


Photorealistic synthetic data for object segmentation

I used such an approach at the Computer Vision Lab at TU Delft for my Bachelor thesis.
The goal was to analyze different aspects of the same problem: identifying LEGO bricks in a photograph of an unsorted pile of LEGO bricks. We built a small real dataset as well as a large (relatively) photorealistic synthetic dataset using Blender.
Personally, I looked at it from the explainable artificial intelligence (XAI) point of view, but some of my team mates did have objectives closer to yours (image segmentation, object detection, classification).
You can find a description which is a bit more detailed here, in section 4.1.
I can edit this answer with more details if you're interested.


Encoding of high cardinality multi-label categorical feature?

When you have high cardinality I suggest you two options (one doesn't exclude the other):

Aggregate least frequent categories into one called other
Use Neural Network with an embedding layer for high cardinality categories. In case you don't know what an embedding is, it is just a table that map each category into a vector of k features (k your choice)


Ridge regression - varying alpha and observing the residual

I still haven't figured out what the previously posted code is doing wrong. However, manually populating the alphas array gives me results that are close to the original figure.

import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error
np.random.seed(12344)
x_1000 = np.linspace(0, 1, 1000)
x_train = np.linspace(0, 1, 10)
noise_10 = np.random.normal(0, 0.3, 10)
y_train = np.sin(2*np.pi*x_train) + noise_10
x_train = x_train[:, np.newaxis]
x_test = np.linspace(0, 1, 100)
noise_100 = np.random.normal(0, 0.3, 100)
y_test = np.sin(2*np.pi*x_test) + noise_100
x_test = x_test[:, np.newaxis]
low_alpha = make_pipeline(PolynomialFeatures(degree = 9), Ridge(alpha=np.exp(-18)))
low_alpha.fit(x_test, y_test)
plt.figure(1)
plt.plot(x_train, low_alpha.predict(x_train), label = 'alpha = ln(-18)')
plt.plot(x_1000, np.sin(2*np.pi*x_1000))
plt.legend()
plt.show
high_alpha = make_pipeline(PolynomialFeatures(degree = 9),
              Ridge(alpha=np.exp(0)))
high_alpha.fit(x_test, y_test)
plt.figure(2)
plt.plot(x_train, high_alpha.predict(x_train), label = 'alpha = 1')
plt.plot(x_1000, np.sin(2*np.pi*x_1000))
plt.legend()
plt.show
alphas = np.array([np.exp(-30), np.exp(-29), np.exp(-28), np.exp(-27), np.exp(-26), np.exp(-25), np.exp(-24), np.exp(-23), np.exp(-22), np.exp(-21), np.exp(-20), np.exp(-19), np.exp(-18), np.exp(-17), np.exp(-16), np.exp(-15), np.exp(-14), np.exp(-13), np.exp(-12), np.exp(-11), np.exp(-10), np.exp(-9), np.exp(-8), np.exp(-7), np.exp(-6), np.exp(-5), np.exp(-4), np.exp(-3), np.exp(-2), np.exp(-1), np.exp(-1), np.exp(0), np.exp(1)])

test_errors = []
train_errors = []

for a in np.nditer(alphas):
    ridge = make_pipeline(PolynomialFeatures(degree = 9),
              Ridge(alpha=a))
    ridge.fit(x_train, y_train)
    mse_train = mean_squared_error(y_train, ridge.predict(x_train))
    mse_test = mean_squared_error(y_test, ridge.predict(x_test))
    train_errors.append(np.sqrt(mse_train))
    test_errors.append(np.sqrt(mse_test))
plt.figure(3)
plt.plot(alphas, test_errors, 'g^', label = 'Test Error')
plt.plot(alphas, train_errors, 'bs', label = 'Train Error')
plt.xscale('log')
plt.xlabel('Regression coefficient Lambda')
plt.ylabel('Residuals')
plt.legend()
plt.show()

The output (only for the third figure) is shown below:


what are the next step after ML prediction and how to proceed?

If one understands the business problem you are trying to solve, then this shouldn't be that hard.

You can present the results of your model. Understand your data first, is it skewed, balanced? Understand your model, check the feature importance. Do they make sense with the domain knowledge you have? For example, if your data is highly imbalanced and you have accuracy of 0.99, then it's highly likely that your model is classifying almost everything as one class. Try and make sense of the model, may be try running some shallow models to establish a baseline.
There's no free lunch. You can't have frameworks for interpreting results, you can have a good set of metrics that you can use but metrics change based on how your data is.
Model is learning continuously?? You mean re-training the model. You need to establish a pipeline for it. You can have a python project for this or use DAG mechanisms like AirFlow.
There's no definite guide, a few text books might help but this comes with experience and understanding the concepts of machine learning thouroughly.


I resolved the issue by making use of the Natural Language Generation, Python Library , nlglib

The solution I came up with is depicted through the Python Code below.

import nlglib
realise_en = Realiser(host='nlg.kutlak.info', port=40000)

p = Clause(NP('jackets'), VP('achieve','highest sales'))
p['TENSE'] = 'PAST'
q = Clause(NP('sweaters'), VP('exhibit','lowest sales'))
q['TENSE'] = 'PAST'
print(realise_en(p))
print(realise_en(q))
r=Clause(realise_en(p)[:-1],'while',realise_en(q))
print(r)

This prints the statements

Jackets achieved highest sales.
Sweaters exhibited lowest sales.
Jackets achieved highest sales while Sweaters exhibited lowest sales.

This is the best solution I found so far.


How to handle columns with categorical data and many unique values

For categorical columns, you have two options :

Entity Embeddings
One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects.  So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

An Overview of Categorical Input Handling for Neural Networks
On learning embeddings for categorical data using Keras
Google Developers > Machine Learning > Embeddings: Categorical Input Data
Exploring Embeddings for Categorical Variables with Keras by Florian Teschner


How do I test a difference between two proportions representing fatality rate for Covid 19 in Philippines and World (except Philippines)?

It is not a case of paired nominal data. Hence, Mc Nemar's test can not be applied to check whether there is a higher fatality rate   in Philippines ?. THE fatality rate is given for Philippines and world ( excluding Philippines ). As defined,  it is expressed as proportion. Therefore, t-test/z-test shall be appropriate given that you meet other conditions such as sample-size.


Deep Learning with Keras. Classification problem with large data

There are actually a lot of examples out there which show how to apply Keras to different types of data. A really good source is F. Chollet's book "Deep Learning with Python". The code is online. You could start with the tutorial on predicting house prices. It is a regression problem, but you can easily change it to a classification problem.

A general remark: Neural nets (such as Keras) usually work well with very complex data (images, sound, text). For "normal" data, other approaches are often better suited (e.g. boosted trees). However, with the large number of features you have, NN is worth a try.


Vertical and horizontal lines appearing on large confusion matrix?

If the $x$-axis is actual and the $y$-axis is predicted, then vertical lines means a given input of the class $x$ is not being discriminated sufficiently and as a result is randomly mapped to a number of classes. Whereas a horizontal line signifies that many classes are mapped as belonging to a particular class.

Class imbalance

Many reasons can lead to these lines in your confusion matrix. The most common is class imbalance. If you train your model using data that is not well balanced across the different classes this will result in both horizontal and vertical lines. Consider a model trained to distinguish cats, dogs and foxes with 1000, 1000, and 10 instances each. Evidently, you can see how many future instances of foxes may be classified as dogs or cats.

You can remedy this by obtaining more data. This is often not possible, so you can try and weight the model to account for the class imbalance. You can do this by weighting the loss function. Or by oversampling the underrepresented class. Furthermore, you can try and synthesize new instances of the underrepresented class. This can be done with a GAN (hard, requires lots of data, unstable), or you can use k-NN type techniques to find out where the data is concentrated and add novel points to add density to this set. The new technique SMOTE is also highly recommended for its simplicity.

Overlapping distributions

Another big reason is a distribution that overlaps. This is when a class' distribution encompasses one or more class distributions. For example, if you are classifying between cats, dogs, shiba inu, corgi. You can see that any amount of training will always cause conflicts between the class dog and shiba inu, corgi. Since these are dogs.

This problem is easy to distinguish for human perceptive classification, like images, but often times we are working with data where the span of these distributions is not as obvious. You can use clustering methods to see if certain classes fall within the distribution cloud of others. I use percent overlap as a metric for this task.


Why an increasing validation loss and validation accuracy signifies overfitting?

Overfitting is when your model is memorizing the training data, and therefore it is not able to generalize to unseen data, like the validation set.
This way, you see overfitting when the training loss decreases but the validation loss increases.
The fact that, while the validation loss increases, the validation accuracy also increases is a separate effect. To understand it, we should take into account that the accuracy is computed comparing the highest output of the softmax with the correct label, and it does not vary depending on the actual value of the softmax output, which is what the validation loss takes into account.


What are the true error and the sample error?

The true error represents the probability that a randomly drawn instance from the entire distribution is misclassified while the sample error is the fraction of sample which is misclassified.
As true error represents entire population it becomes difficult to calculate hence we use sample to check our hypothesis and use evaluation methods to check it's confidence level. A sample might not be a true representation of population so the difference in results are sample error. We try different sampling methods so that there is no bias in choosing a sample like randomised and stratified sampling.
For more you can refer this


Why does my Keras model learn to recognize the background?

Your model is overfitting. Each epoch only has 5 images. The model is "memorizing" the answer for each image.

In order to minimize the chance of overfitting, increase the number of images. There should be several thousand example images for each category of object.


How to use reduced dimensions (of a PCA) for detection purposes?

Well, mathematically speaking, applying PCA on a bunch of data points usually means (there are some variants) removing the mean and rotate to get uncorrelated features. Then you reduce dimensionality by discarding some of the uncorrelated features.

You may get the corresponding points in the original feature space by doing the inverse rotation and adding the mean back again. Of course, this points now will be contained in a shape of less dimension than before. For example, if you went from 3 original features to 2 uncorrelated features, then perhaps your original data points had some 3D shape, say a football. Then your new points (transformed-inverse transformed) will have the shape of an ellipse.


GridSearchCV with MLPRegressor with Scikit learn

It would be helpful to get the ouput of the program (or at least the error thrown)

However, MLPRegressor hidden_layer_sizes is a tuple, please change it to:

param_list = {"hidden_layer_sizes": [(1,),(50,)], "activation": ["identity", "logistic", "tanh", "relu"], "solver": ["lbfgs", "sgd", "adam"], "alpha": [0.00005,0.0005]}

https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html


Two-class model with predicted scores needed - classification or regression approach

This is a classification problem, not a regression problem, so your first instinct was correct. For most models, you can easily get the probability estimate for each prediction. If you did tree based methods or a linear model, you would have probabilities easily. However, because of how an SVM works, it does not automatically output a probability estimate. It makes sense that Caret has to do some extra computations to get your probabilities.

For the specifics, see: https://stats.stackexchange.com/questions/335527/what-are-the-predicted-probabilities-from-an-svm


Question on ANOVA and Correlation/Association

About what ANOVA is used for: it can answer whether the difference between the mean values for the data samples I have is due to randomness or is it statistcally significant. Then it is a significance-test that gives you an idea about whether your mean values are (statistically significantly) the same or not. A drawback is that it does not tell you which data sample/s differ from the rest or by how much (useful source).
You can think of the process as follows (as described in Practical Statistics for data scientists):

Combine all the data together in a single box
Shuffle and draw out n resamples of m values each (where n is the number of data samples and m the number of data points in each sample)
Record the mean of each of the n groups
Record the variance among the n group means
Repeat steps 2–4 many times (say 1,000)
What proportion of the time did the resampled variance exceed the observed variance? This is the pvalue.

On the other hand, the direct measure of correlation gives you a number, it tells you by how much two data samples vary linearly along with each other


Predicting next adverse event based on previous events

What you're describing is known as a multilabel classification. You want to predict some output labels (adverse effects) in a given set of possible output labels  (all possible adverse effects), using a given a set of input indicators (previous adverse effects, drugs prescribed, dosage levels).

For those final two features, I'd combine them into one feature where if the drug is not prescribed obviously dosage is 0, and higher otherwise.

In R, you can use the mlR package for this prediction stage. You may also be able to use the generateFilterValues() function in this package to extract feature importances, however I'm not sure if this will work with multilabel and additionally this will only tell you how much your classifier believes your input features contribute to the output labels relative to all the other input features. It won't give you other valuable information for example at which dosage levels of a particular drug does the probability of negative symptoms increase most.

Again, I'm unsure how this would work with multilabel classifications as I don't have any experience with them, but take a look at plotting partial dependency plots to get this last bit of information on how each input feature contributes to your output labels.


Why is there a $2$ at the denominator of the mean squared error function?

This is just for mathematical convenience. When you differentiate $C(w,b)$, you will get an extra $2$. To eliminate that, $2$ is kept beforehand in denominator.

You can also watch this video on SVM lecture by Patrick Winston, where he uses a similar formula and then tells that he is using $2$ in denominator just for mathematical convenience.


YOLO annotation guildelines: overlapping and partially visible objects

There is only 1 restriction, your truth bboxes with the same class_id shouldn't be overlapped more than IoU > 45%, because Yolo uses nms_threshold = 0.45.

In general, you should mark your objects in such a way as you want to detect these objects.


Pytorch - Gradient distribution between functions

Recall that you passed net.parameters() to the optimizer, so it has access to the "Tensor" objects, as well as their associated data. One of the associated data fields associated to each learnable tensor parameter is a gradient buffer. Hence, backward() not only computes the gradients, but stores them in each parameter tensor, so that the gradient vector per parameter is stored along with that parameter. In other words, for some parameter $\theta_i$, backward() stores $ \partial \mathcal{L}(\Theta)/\partial \theta_i$ along with that parameter. The optimizer.step() call then simply updates each parameter via the gradient stored along with it.


same or different number of epochs for dataset of different sizes?

More data could potentially lead to more epochs required to find the best model. The number of epochs to reach the minimum loss will vary depending on your hyperparameters, dataset, and initial weights. I would get out of the mindset of trying to find the exact number of epochs required. Most popular deep learning libraries will allow you to stop training if the loss converges after a certain length of epochs. Additionally, if you overfit, you can create a callback function to save the best model only.


FuzzyWuzzy would be a good python library for this purpose. A code example from the GitHub readme:

>>> choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
>>> process.extract("new york jets", choices, limit=2)
    [('New York Jets', 100), ('New York Giants', 78)]
>>> process.extractOne("cowboys", choices)
    ("Dallas Cowboys", 90)

I imagine you would like to use the extract function with limit 5 with choices being set equal to all the 'official names' from your table2 csv file.

Details on installation and use are at the link above.


Predicting average temperature is a regression task, not classification. You should be using DecisionTreeRegressor instead. Temperature is a continuous value and you are treating it as a category by using a classifier.

Tinkering with the hyperparameters (maximum number of leaves, minimum samples per leaf, and maximum depth, etc) is still important since decision trees always are prone to overfitting. If you struggle to find good parameters yourself then you can try some automated methods such as GridSearchCV or RandomizedSearchCV in sklearn.


Visualizing many one-to-many relationships

Without seeing the data itself, two visualisations come to mind. One option is to run hierarchical clustering on the repositories, and plot the results as a dendrogram with heatmap (or clustermap). The second would be to generate a force-directed graph of language co-occurrence across the repos.

Dendrogram with Heatmap

This plots all language co-occurrences, similar to an adjacency matrix; this clustermap produced with seaborn is an example of a dendrogram with heatmap:

It might be possible to gain insight on predominance/hierarchy from the dendrogram by considering which languages are seen more frequently with others, or not at all. The general co-occurrence of languages should be evident from the heatmap.

The advantage of this visualisation is that it should show relationships between languages more clearly than the chord. In particular, some insight might be gained from the 'hierarchy' of languages within each cluster generated. However, a drawback is that the number of repos is not easily represented.

Force-directed Graph

A visualisation that addresses the drawback of the previous method is the force-directed graph, where structure is derived by using "repulsive forces between nodes and attractive forces between adjacent nodes". This might show the numbers of repos involved more effectively, as nodes appearing more frequently with other nodes are weighted with greater 'mass':

The graph above by Mike Bostock is a great example with code and data, but for your use case it may be improved if language counts were represented by the size of node, similar to this visualisation by Christopher Manning. As an afterthought, this bubble chart might also serve if it could be reworked with edges connecting nodes to show where languages were used together in a repo.

The caveat is that without seeing the data I'm not exactly certain what is possible with this dataset. If you can wrangle the data into counts of which languages appear in each repo, and assume that all languages appearing in the same repo are co-occurrences, then these visualisations should be an option.


Build train data set for natural language text classification?

It would help to do some analysis of the scripts to identify aspects that distinguish the various categories. Once you do this manually for some examples, you could consider writing some rules based on the observations. The rest of the examples can be labeled using the rules. For a model-based approach, if you label a small set of examples (~50), then a simple model (Naive Bayes, etc.) can potentially be trained on these.


How to replace levels with certain value?

I could be able to achieve this with the following code:

testing$var1 <- as.character(testing$var1)
a <- data.frame(a)
testing$var1 [testing$var1 %in% a[1,] <- "Others"
testing$var1 <- as.factor(testing$var1)

In case if there is any other better/effective solution/function to achieve this, please let me know.

Thanks all.


Finding a pattern in reponses in R programming

Generally I would suggest to look for differences with the general pattern, the general pattern being the answers from most users. A very basic way to do that:

Calculate for every question the proportion of text 1, store it as a vector. This distribution is the "mean vector".
For every player represent the vector of their answers: for every question the value is 1 if they chose text 1, 0 if they chose text 2.
For every player compare the vector of their answer with the "mean vector", for instance with cosine similarity (or any other distance/similarity measure).

Then look at the distribution of the similarity scores: normally most players have a high similarity scores, so if some of them have very low similarity it's likely that they didn't do the test seriously.
If you want a more advanced method, you could cluster the players' vectors or do anomaly detection.

[edit based on comment]

Take the proportion of "text 1" for every participant and compare to the mean proportion across all participants. This would catch the most obvious outliers, but probably not any subtle pattern.
Measure how much correlation there is between the question number and the answer: normally the two are independent so the correlation should be very low, if not there is pattern. Note that I think there would be a better statistical test than Pearson correlation, but I don't remember which one would suit this case (Spearman correlation might be relevant, I'm not sure).


Is it possible to use a generative model to "share" private data?

Unfortunately I don't think a generative model could prevent from leaking private information from the original dataset.

Like any other kind of model, the generative model is based on the values obtained from the training data. The idea of using such a model in "generation mode" is indeed interesting since it would make it difficult to reverse-engineer the instances it generates back to the real individuals. It would be difficult sure, but not impossible: by re-connecting pieces of information together or exploiting rare (distinctive) cases, somebody could acquire at least partial personal information from the instances.

Additionally the design of the generative model itself introduces a huge bias in the data: the distribution of the instances would be modeled after this design, which might or might not accurately represent the real distribution. This issue significantly lowers the interest in exploiting the generated instances, since it's essentially artificial data.

For the record I think I've seen research about using distributed ML methods in order to overcome the privacy issue. As far as I understand the idea is to keep every individual/institution in control of their data, allowing specific automatized methods to read it in some kind of safe way.


What model do I use to predict a regression problem with timeseries data

[Updated]

If you have measured Reagent B for same number of time steps for each experiment, then:

Using these time steps as feature values, you can build a regression model using, Multi Layer Perceptions, XGBoost/LGBM/RandomForests etc.
Even Recurrent Neural Nets could work if enough data is available. Treat your time step values of Reagent B as single dimention embedding values and use BiLSTM architecture with Linear activation function in the last layer. Loss could be mean squared error. BiLSTM could work in your case as it makes sense to read from last time step to first time step also

If reagent B is measured for different time steps, in different experiments:

You can still use Ensemble Trees/MLP/RNN based methods, but you will have to consider adding paddings(imputing) or limiting max time steps to make number of time steps uniform across all experiments.

XGB/LGBM can take missing values as inputs and handle it well, so you dont need to do any imputation.


How can I make a face verification system?

I am assuming Facenet is an image classifier and it will give embedding for a face similar to other CNNs. If that is the case you don't need to train different classifiers, you can just remove the head of Facenet and initialize a small network on top of it and train an end to end network for better accuracies.


It's common to perform clustering on certain features, and then use characteristics such as mean differences between other features (not used in clustering) to show how clusters differ.  It's not always meaningful to say that clusters differ merely by lower distances between cluster centers and objects assigned to them (as in k-means).

As an example, you might acquire other features (variables) for the states and then average each feature among objects assigned to each cluster.  Then show bar charts of averages for those variables among all the clusters to assist in explaining how the clusters differ.  You might also test for significantly different means of the supplemental features across clusters.


Can AlphaGo Zero adapt to oponents skills/profile?

It's not important to adjust to the opponent, as Go is only about winning or losing. There is no bigger reward the faster / the more obvious it wins.

Or to put it different: only the current board situation is important in a min-Max setting (although the value approximation of a state admittedly depends a bit on the opponent)


In Tensorflow, What kind of neural network should I use?

Sure, you can use an RNN. I would create two features for the past $k$ run lengths, as well as the length of the current run; e.g., just before t=150, the current run would be length 2 (red), and previous three runs would be (1,1,1) for red and (1,1,5) for blue. The intuition here is that the run lengths seem to follow some sort of exponential distribution, and you want to help the model estimate the scale parameter by feeding it samples of the length. You could additionally encode the past k events as a bit string, with 1 representing red and 0 representing blue. You have a classification problem, so you should use classification loss like the cross-entropy, and a softmax output layer to get your probabilities.


What kind of classification should I use?

Using your input data as-is (many categorical attributes), you should start with random forests.  The levels of your features probably aren't ordinal in any sense, which would explain why multinom (a linear model) didn't work.  A decision tree (component of a random forest) won't have trouble with categorical features that aren't ordinal.

If your features are correlated (I'm guessing they are highly correlated), you might also try first reducing the dimensionality of your input dataset.  A nice way to do this is to view your categorical features as "words" and map each input sample (a word in all its forms) to a sparse vector (training a TF-IDF model).  Then, reduce the dimensionality of your sparse vectors using LSI or LDA (dense vectors).  Train a classifier (again, random forest is a good starting point) using these dense vectors as input data.


Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I have found the answer to my own question, went back to the Python script and in the command that fits the model i.e.

   LR = LogisticRegression (C=0.1, solver = "sag",max_iter=1000).fit (X_train, y_train)

The parameter C was set to 0.001 which is a very small value (meaning lambda is very high as C=1/lambda) (C is the regularization strength and smaller values indicate stronger regularization). more on that matter can be found here and here


Selection of Features and Data in Random Forest

First, I am confused whether at each node in all the trees, do we
  randomly pick features from the lot to be pitted for best split or
  does each tree get a random subset of feature and then all the nodes
  in that tree have to work with those features?

m-features are chosen randomly at each split of the tree, not at the creation of the tree itself.

  Now, regardless of the case whether we pick random features at tree or node level, why do choose random subset of data for each tree? Because if the features are different at tree/node level, the trees will be different. Why do we need to add another level and make the distribution of data also random to the trees and not just feed the complete original dataset to the trees?

This is exactly what makes random forests unique compared to other algorithms. Indeed, we sample with replacement (bootstrap) from the training dataset and each bootstrap sample is unique to each tree (which you appear to understand).

However, we need to consider how the final predictions come out of a random forest in the first place. Recall that the final prediction for an observation $y_i$ will be the average predicted probability among all of the trees in the forest. This is supposed to work because, under the (crucial) assumption of zero/negative correlation between the trees, we know that the sample mean has variance that goes to zero as n goes to infinity. This is one key element of random forests; if I have a bunch of overfit trees (low bias, high variance) I can simply get rid of this high variance for each tree by combining the predictions of each individual tree and averaging. The result is hopefully low bias, and low enough variance for my final, combined model.

However, if there is positive correlation among the trees, the variance of the sample mean actually increases (as  $ n \rightarrow \infty$) and if n is large enough, the covariance between the trees actually starts to dominate the variance of each individual tree itself. What this results in is a decrease in predictive performance compared to just using a single (or relatively few) tree(s). After all, the whole point is to fit low bias, high variance base learners and to then reduce the variance by averaging. If we now cannot reduce the variance due to correlation between base learners (and start increasing it instead!), our model is likely to be severely overfit, even moreso than using a comparatively less overfit base learner.

So the question becomes...how do we reduce the correlation between the trees? One solution is to introduce randomness not only through the data sample itself (i.e. bootstrap/sample with replacement) but also through randomly selecting predictors at each split of a node. By doing this, we hopefully force each tree to learn something different about the data. Indeed, if only a few predictors are useful than chances are, all of the trees will pick up on this and stick to using these few predictors over and over again. The result is that all of the trees have learned exactly the same thing, substantial correlation among the trees, and not much purpose to using the trees together.

Bagging (which is what you refer to as the first level of randomness) and other ensembling based methods work best when we have what people call "diverse base learners". This means that in general, if we can find many models that are similar in performance but uncorrelated (that is, they have learned different, but equally important parts of the data) then combining the models in some fashion will almost certainly be better than using either one alone. Creating model diversity is difficult but some common ways are;

1) Do something similar to random forests; give each base learner a different set of features to use.
2) Use different algorithms that hopefully learn different parts of the data due to the differences in how they are fit; example: random forest + neural network + gradient boosting, etc.


Open-source interactive dashboard in Python

Have a look at IPyWidgets.
I've used it to create interactive dashboards in IPython/Jupyter. It's very concise and powerful.
You define a function:
def f(x):
return x

And you connect it to a widget using the Interact() function.
interact(f, x=10);

This generates a widget (a slider in this case) automatically and updates your function with new input when you interact with the widget. The type of widget is determined from the default value you specify (i.e. a checkbox for a boolean, a textbox for a string, etc.)


http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html

You can use the above tutorial to get acquaintance with text classification. Afterwards it should be easier to formulate nontrivial questions to move even further.


Technical question: how to build exe from python code?

You'll need to copy, or convert*, your IPython code (I think) into a Python script. After that you can use software such as py2exe to create a standalone application. Other hypothesis are:

cx_freeze
pyInstaller
py2App (Mac OsX)

It's possible to save data inside the scripts (it will be compiled the same way as code as long as it is valid). Other possibility is to save in an external file and just load the data on execution. If the data is in a class check pickle to save it to a file.

as @dmb suggested.

EDIT: Ah I forgot to mention the arguments. That is easily achievable. Check this Question in SO.


Is it a good practice to always apply `ReduceLROnPlateau()`, given that models benefit from reducing learning rate once learning stagnates?

I believe it is, in theory, always a good idea to use this method. I say in theory because the theory of gradient descent points to the fact that a minimum can only be reached when the learning rate approaches 0. Otherwise, with a permanent large learning rate, the model's performance (i.e. the loss metric) will bounce around the minimum – always overshooting it.

That being said, it is important to think practically and be aware of the dangers of using this method. There can be phases of training where you don't improve your metric by much, and so the learning rate is reduced. But this can happen prematurely - depending on the plateau-parameters you selected - and so actually impede/prevent the model from ever getting far enough down the loss curve! If you set a minimum learning rate, you may always get down to a minimum (local or otherwise), but it could take a lot longer if your learning rate was reduced prematurely.

In the end, trying out as many combinations as possible is likely your best best.

Alternatively, you could try combing this with simulated annealing, whereby the learning rate is boosted upwards again, according to a predefined rule, such as after a number of epochs or a %-reduction in the learning rate itself.


Measure correlation for categorical vs continous variable

By saying you want to "explain Y by X" it sounds that you try to build a classifier F that can map X values into expected Y: F(X) --> Y.
If so, you don't have to search for "correlation" necessarily.
There are various methods to build such a classifier. You can use logistic regression \ SVM \ Neural network \ etc.
Besides, if it make more sense for you, you can always first discretize the continuous variables into categorical vars and than use also other methods such as decision trees \ Naive Bayes and more.


when you "predict" something in KNN-classify problems, you are classifying new information.

yah, KNN can be used for regression, but let's ignore that for now.

The root of your question is why bother handling known data, and how can we predict new data.  Let's do KNN in R1, with two training examples.  The first one will be 0 and it will be class A, the next one will be 100 and it will be class B.

So, KNN is what's known as a lazy classifier. You actually aren't training, any hyperparameters, just loading the training data.  I've loaded two points, and now I want to classify a new point.  Let's consider the point 0.  Since we already have the point 0 labelled as class A, we know it's going to be class A.  What about the point 4?  It's closer to 0 than to 100, so it's class A.
Similarly, the point 95 is closer to 100, so it's class B.

KNN is really just doing those comparisons with more points and/or more dimensions.

EDIT: user asked the following: how would I predict if it is yes or no, based on the classification I have done before?

The most basic implementations of KNN check the distance from your new case to all of the points in the training data.  If you want to visualize it, it's possible with one, two, and maybe 3 dimensional data.

In python, you can do as follow (from scikit-learn.org):

X1 = np.random.normal(0,1,100) #values centered around 0
X2 = np.random.normal(6,.1,100) #values centered around 6
X = np.concatenate([X1,X2])
X = np.array(X).reshape(-1,1)
Y = [0 for y in range(100)]+[1 for y in range(100)] #2 classes
Y = np.array(Y).reshape(-1,1)

clf = neighbors.KNeighborsClassifier()
clf.fit(X, Y)

now, you can predict by calling predict

clf.predict(1)
clf.predict(4)
clf.predict(.1)

KNN visualized in 2D:

from: https://www.analyticsvidhya.com/blog/2014/10/introduction-k-neighbours-algorithm-clustering/


A class taxonomy should:

Serve the business needs
Be learnable

There is a potential tradeoff here. The more exact and specific the taxonomy, the more you'll know about the entities and you'll be able to server better the business needs. However, for a large taxonomy the classifier will have to model more complex rules, will have less samples for each case and the boundary samples will have more influence.

Check the number of samples per label in your dataset. A class label with not enough samples will be hard to learn. If you have many class label, the ratio of some of them will also be low, introducing an imbalance learning problem which is also harder.

Using a cost matrix, you can evaluate the extra cost due to merging some classes. merge classes when the cost on not distinguishing among them is low.

Building a taxonomy of class might can help working in many resolutions. Take for example animals, you can create a taxonomy of animals -> reptiles -> crocodiles and then try to distinguish reptiles from other animals or crocodiles from other reptiles.

In many some cases you might get many class labels due to a Cartesian product of some dimensions. Using the animals examples again, you might be interested in dimension like "is_flying?", "is_carnivore?" and learn them instead of the animal classes. These dimensions will probably be more balanced and can be learn with a certain degree on independence (reusing samples). The down side is the a flying carnivore will be a bird of prey and not necessarily a falcon.


Ideal score of a model on training and cross validation data

Accuracy on the training data basically doesn't count. I don't quite want to say to ignore it, because a train/test accuracy of 100/70 seems different to me than a train/test accuracy of 71/70, but you're mostly not interested in performance on the training data.

Using a test set mimics a real application of machine learning. Think about Siri or Alexa. The goal is to predict speech that it hasn't heard. There's no way to know how it will perform on such speech, so the next-best approach is to use some data where you know the answer but hide it from your model. After you train the model, assess how it performs on data where it has not seen the answer. If the model is accurate, then that's a good sign about its ability to perform on real speech recognition tasks.

Training data is like the practice questions or homework problems, while test data is the exam.


What is the most efficient data indexing technique

I think you messed up some things in your question. Lucene (I know nothing about Lucene,NET, but I suppose is the same) is a library used to analyze, split in tokens, and store documents in order to be able to query and retrieve them later. Lucene has a pretty old but effective model, it uses inverted trees to find and retrieve documents. Without further details, all documents are split in tokens (terms), and for each term is maintained a data structure, which stores all the documents which contains the given term. As a data structure could be used a BTree, a hash table and in the latest major revisions you can even plug in your own data structures.

A BTree (see Wikipedia page for further details), is a kind of a tree data structure, which is appropriate for working with big chunks of data and is often used for storing tree-like ordered structures on disk. For in-memory other trees performs better.

Murmur hash (see Wikipedia page for further details), is a family of hash functions used in hash table. The implementation of the hash table is not important, it could be a standard chained implementation or more advanced open hash addressing scheme. The idea is that the hash tables allows one to get fast a key, from an unordered set of keys, and can answer to tasks like: is this key part of this set of keys? which is the value associated with this key?

Now back to your main problem. You have one library (Lucene) and to data structures, both data structures are used in Lucene. Now you see that it is not possible to answer your question in these terms since they are not comparable.

However, regarding you footprint and performance part of the question. First of all you have to know which kind of operations you need to implement.

Do you need only get value for key, or do you need to find all elements in a range? In other words do you need order or not? If you do, than a tree can help. If you do not, than a hash table, which is faster could be used instead.

Do you have a lot of data which does not fit the memory? If yes than a disk-based solution would help (like BTree). If your data fit the memory, than use the fastest in-memory solution and use disk only as a storage (with a different structure, much simpler).


Using a neural network to learn regression in image processing

Yes - neural networks can be used to predict regression problems. The output layer will be a set of nodes that make continuous numeric predictions.
In your problem, it would be two nodes that learn the weights for $\theta_1$ and $\theta_2$.


Text generation using Tensor Factorization

Tensor Factorization would not work for text generation as a stand-alone technique. There is no way for the decomposition to model long-term dependencies in language. Without modeling long-term language dependencies, its results would be similar to low-order Markov chains.

Tensor Factorization could be used as another signal in a larger natural language generation system, for example improving word-embeddings.


I can mention two main reasons.

Complexity of dataset
high Bayes error

The former means your dataset is highly non-linear and the latter simly means you may have same input with contradictory outputs in your dataset.


How to interpreting the exponential coefficent in poisson regression with offset?

The offset means "exposure", so when you are calculating a regression with offset, instead of calculating absolute values, you are calculating ratios.

You are changing "number of thefts" for "number of thefts for every 10.000 vehicles"

$y = x1+x2+(1|cluster) + log(x_3)$: This is the equation you are calculating.

I think you are missing a log in $y$ because the offset and the endogenous variable have to form a ratio:

$log(y) = x1 + x2+(1|cluster)+log(x_3)$

$log(y)-log(x_3) = x1 + x2+(1|cluster)$

$log(\frac{y}{x_3}) = x1 + x2+(1|cluster)$

$log(ratio) = x1 + x2+(1|cluster)$


In seaborn, the hue parameter determines which column in the data frame should be used for colour encoding. Using the official document for lmplotprovided an example for this.

import seaborn as sns; sns.set(color_codes=True)
tips = sns.load_dataset("tips")
g = sns.lmplot(x="total_bill", y="tip", data=tips)

Adding `hue="smoker" tells seaborn you want to colour the data points for smoker and non-smoker differently.

>>> g = sns.lmplot(x="total_bill", y="tip", hue="smoker", data=tips)


At my work we've had some success just using averages of pre-trained embeddings (e.g. the GloVe vectors) for classifying short texts. Have you tried that?


How to label a dataset for Machine Learning?

If you have a smaller dataset, then I think manual labelling could work.

For larger datasets, you need to use a sentiment analyser engine which has been retained on a very large dataset.

For example, you can use TextBlob,

from textblob import TextBlob
obj = TextBlob( some_text )

print( obj.sentiment.polarity )

You can one by one load each review and sort them into three classes or continuous values.

You can classify the review into a categorical variable. For example, if the polarity is above 0 then the review is positive else it is negative. Here, you will perform classification.
Use the values from TextBlob as they are. Hence, they will form continuous labels. Here, you will perform regression.

Tip:

Also, you can search for similar packages which provide sentiment score for a text. You can use Natural Language Processing APIs.

import urllib

data = urllib.urlencode({"text": "I'm a very good boy "})
u = urllib.urlopen("http://text-processing.com/api/sentiment/", data )
the_page = u.read()

print (the_page)


Tensorflow RNN: Batching data of different length

I'd suggest padding your sequences to all be the same length. Obviously this isn't ideal as you end up wasting time processing fake data but it does solve the problem.


Applying machine learning algorithms to subset of attributes in dataframe

Applying a machine learning algorithm on only a subset of the data and including other subsets later does not allow the algorithm to assess the importance of each attribute equally.

For example, say you have a data set called A, which has subsets B and C. Without loss of genearality, if you fit a model ('apply an algorithm') on subset B, and then include subset C later, then you're saying 'given subset B is already in the model, assess the impact of including subset C'. Instead, if you apply the entire algorithm on the entire data set (A), then you're allowing the algorithm to discover which features are most important for the desired outcome.

That being said, it may be wise to process the different elements of your data set differently. That is, categorical covariates may be modelled differently from continuous covariates. If you're using something like a feed-forward neural network, then it's not a big deal, but if you're using a more traditional statistical model you may need to take that into account. For example, in R, you need to specify that a categorical covariate is in fact a 'factor' variable.


A quick answer would be No.

Let's walk through how FastText works internally:

For representation purposes, FastText internally initializes a dictionary. Dictionary contains all the collection of words. Besides words, it also maintains the count of all the words in the dictionary (and other information). Every time a new word is added to the dictionary its size is increased and word2int_ is updated to size_(which is increased after assignment).

The code below adds a word into the dictionary.

// adding of new word
void Dictionary::add(const std::string& w) {
  int32_t h = find(w);
  ntokens_++;
  if (word2int_[h] == -1) {
    entry e;
    e.word = w;
    e.count = 1;
    e.type = getType(w);
    words_.push_back(e);
    word2int_[h] = size_++; //  word2int_[h] is assigned a uniuqe value
  } else {
    words_[word2int_[h]].count++; // word's count is being updated here
  }
}

// funciton used to access word ID (which is the representation used)
int32_t Dictionary::getId(const std::string& w) const {
  int32_t h = find(w);
  return word2int_[h];
}

As mentioned in this medium article, word2int_ is indexed on the hash of the word string, and stores a sequential int index to the words_ array. The maximum size of word2int_ vector can be 30000000.

For embeddings matrix of M x N is created where M = MAX_VOCAB_SIZE + bucket_size. Where, M is total vocabulary size including bucket_size corresponds to the total size of array allocated for all the n-gram tokens and N is the dimension of the embedding vector, which means the representation for one word requires the size of 1.

The code below shows how to compute the hash and calculates the ID of the subword. A similar logic is used to access the subword vector. Note here h is an integer value which is calculated using dict_->hash(). This function returns the same h value which is used when adding a word in the dictionary. This makes the process of accessing word IDs only dependent on the value of h.

int32_t FastText::getSubwordId(const std::string& subword) const {
  int32_t h = dict_->hash(subword) % args_->bucket;
  return dict_->nwords() + h;
}

void FastText::getSubwordVector(Vector& vec, const std::string& subword) const {
  vec.zero();
  int32_t h = dict_->hash(subword) % args_->bucket;
  h = h + dict_->nwords();
  addInputVector(vec, h);
}

Long story short, FastText makes use of integer IDs assigned at the beginning and use those IDs to access the embeddings.

I hope this helps. All the code samples are taken from FastText repo. Feel free to dive in to understand more.


You do not need a wrapper for DBPedia, you need a library that can issue a SPARQL query to its SPARQL endpoint.  Here is an option for the library and here is the URL to point it to: http://dbpedia.org/sparql

You need to issue a DESCRIBE query on the United_States resource page:

PREFIX dbres: <http://dbpedia.org/resource/>

DESCRIBE dbres:United_States

Please note this is a huge download of resulting triplets.

Here is how you would issue the query:

from SPARQLWrapper import SPARQLWrapper, JSON

def get_country_description():
    sparql = SPARQLWrapper("http://dbpedia.org/sparql")
    sparql.setReturnFormat(JSON)

    sparql.setQuery(query)  # the previous query as a literal string

    return sparql.query().convert()


Find optimal P(X|Y) given I have a model that has good performance when trained on P(Y|X)

This response has been significantly modified from its original form. The flaws of my original response will be discussed below, but if you would like to see roughly what this response looked like before I made the big edit, take a look at the following notebook: https://nbviewer.jupyter.org/github/dmarx/data_generation_demo/blob/54be78fb5b68218971d2568f1680b4f783c0a79a/demo.ipynb

TL;DR: Use a KDE (or the procedure of your choice) to approximate $P(X)$, then use MCMC to draw samples from $P(X|Y) \propto P(Y|X)P(X)$, where $P(Y|X)$ is given by your model. From these samples, you can estimate the "optimal" $X$ by fitting a second KDE to the samples you generated and selecting the observation that maximizes the KDE as your maximum a posteriori (MAP) estimate.

Maximum Likelihood Estimation

... and why it doesn't work here

In my original response, the technique I suggested was to use MCMC to perform maximum likelihood estimation. Generally, MLE is a good approach to finding the "optimal" solutions to conditional probabilities, but we have a problem here: because we're using a discriminative model (a random forest in this case) our probabilities are being calculated relative to decision boundaries. It doesn't actually make sense to talk about an "optimal" solution to a model like this because once we get far enough away from the class boundary, the model will just predict ones for everything. If we have enough classes some of them might be completely "surrounded" in which case this won't be a problem, but classes on the boundary of our data will be "maximized" by values that aren't necessarily feasible.

To demonstrate, I'm going to leverage some convenience code you can find here, which provides the GenerativeSampler class which wraps code from my original response, some additional code for this better solution, and some additional features I was playing around with (some which work, some which don't) which I probably won't get into here.

np.random.seed(123)
sampler = GenerativeSampler(model=RFC, X=X, y=y,
                            target_class=2,
                            prior=None,
                            class_err_prob=0.05, # <-- the score we use for candidates that aren't predicted as the target class
                            rw_std=.05,          # <-- controls the step size of the random walk proposal
                            verbose=True,
                            use_empirical=False)
samples, _ = sampler.run_chain(n=5000)

burn = 1000
thin = 20
X_s = pca.transform(samples[burn::thin,:])

# Plot the iris data
col=['r','b','g']
for i in range(3):
    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')
plt.plot(*X_s.T, 'k')
plt.scatter(*X_s.T, c=np.arange(X_s.shape[0]))
plt.colorbar()
plt.show()

In this visualization, the x's are the real data, and the class we're interested in is green. The line-connected dots are the samples we drew, and their color corresponds to the order in which they were sampled, with their "thinned" sequence position given by the color bar label on the right.

As you can see, the sampler diverged from the data fairly quickly and then just basically hangs out pretty far away from values of the feature space that correspond to any real observations. Clearly this is a problem.

One way we can cheat is to change our proposal function to only allow features to take values that we actually observed in the data. Let's try that and see how that changes the behavior of our result.

np.random.seed(123)
sampler = GenerativeSampler(model=RFC, X=X, y=y,
                            target_class=2,
                            prior=None,
                            class_err_prob=0.05,
                            verbose=True,
                            use_empirical=True) # <-- magic happening under the hood
samples, _ = sampler.run_chain(n=5000)

X_s = pca.transform(samples[burn::thin,:])

# Constrain attention to just the target class this time
i=2
plt.scatter(*X_r[y==i,:].T, c='k', marker='x')
plt.scatter(*X_s.T, c='g', alpha=0.3)
#plt.colorbar()
plt.show()

sns.kdeplot(X_s, cmap=sns.dark_palette('green', as_cmap=True))
plt.scatter(*X_r[y==i,:].T, c='k', marker='x')
plt.show()

This is definitely a significant improvement and the mode of our distribution corresponds roughly to what we're looking for, but it's clear we're still generating a lot of observations that don't correspond to feasible values of $X$ so we shouldn't really trust this distribution either.

The obvious solution here is to incorporate $P(X)$ somehow to anchor our sampling process to regions of the feature space that the data is actually likely to take. So let's instead sample from the joint probability of the likelihood given by the model, $P(Y|X)$, and a numerical estimate for $P(X)$ given by a KDE fit on the entire dataset. So now we're... sampling from... $P(Y|X)P(X)$....

Enter Bayes Rule

After you hounded me to be less hand-wavey with the math here, I played around with this a fair amount (hence me building the GenerativeSampler thing), and I encountered the problems I laid out above. I felt really, really stupid when I made this realization, but obviously what you are asking for calls for an application of Bayes rule and I apologize for being dismissive earlier.

If you're not familiar with bayes rule, it looks like this:

$$P(B|A) = \frac{P(A|B)P(B)}{P(A)}$$

In many applications the denominator is a constant which acts as a scaling term to ensure that the numerator integrates to 1, so the rule is often restated thusly:

$$P(B|A) \propto P(A|B)P(B)$$

Or in plain English: "the posterior is proportional to the prior times the likelihood".

Look familiar? How about now:

$$P(X|Y) \propto P(Y|X)P(X)$$

Yeah, this is exactly what we worked up to earlier by constructing an estimate for the MLE that is anchored to the observed distribution of the data. I've never thought about Bayes rule this way, but it makes sense so thank you for giving me the opportunity to discover this new perspective.

To backtrack a tiny bit, MCMC is one of those applications of bayes rule where we can ignore the denominator. When we calculate the acceptanc ratio, $P(Y)$ will take the same value in both the numerator and denominator, canceling out, and allowing us to draw samples from unnormalized probability distributions.

So, having made this insight that we need to incorporate a prior for the data, let's do that by fitting a standard KDE and see how that changes our result.

np.random.seed(123)
sampler = GenerativeSampler(model=RFC, X=X, y=y,
                            target_class=2,
                            prior='kde',         # <-- the new hotness
                            class_err_prob=0.05,
                            rw_std=.05,          # <-- back to the random walk proposal
                            verbose=True,
                            use_empirical=False)
samples, _ = sampler.run_chain(n=5000)

burn = 1000
thin = 20
X_s = pca.transform(samples[burn::thin,:])

# Plot the iris data
col=['r','b','g']
for i in range(3):
    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')
plt.plot(*X_s.T, 'k--')
plt.scatter(*X_s.T, c=np.arange(X_s.shape[0]), alpha=0.2)
plt.colorbar()
plt.show()

Much better! Now, we can estimate your "optimal" $X$ value using what's called the "maximum a posteriori" estimate, which is a fancy way of saying we fit a second KDE -- but to our samples this time -- and find the value that maximizes the KDE, i.e. the value corresponding to the mode of $P(X|Y)$.

# MAP estimation

from sklearn.neighbors import KernelDensity
from sklearn.model_selection import GridSearchCV
from scipy.optimize import minimize

grid = GridSearchCV(KernelDensity(), {'bandwidth': np.linspace(0.1, 1.0, 30)}, cv=10, refit=True)
kde = grid.fit(samples[burn::thin,:]).best_estimator_

def map_objective(x):
    try:
        score = kde.score_samples(x)
    except ValueError:
        score = kde.score_samples(x.reshape(1,-1))
    return -score

x_map = minimize(map_objective, samples[-1,:].reshape(1,-1)).x

print(x_map)

x_map_r = pca.transform(x_map.reshape(1,-1))[0]
col=['r','b','g']
for i in range(3):
    plt.scatter(*X_r[y==i,:].T, c=col[i], marker='x')
sns.kdeplot(*X_s.T, cmap=sns.dark_palette('green', as_cmap=True))
plt.scatter(x_map_r[0], x_map_r[1], c='k', marker='x', s=150)
plt.show()

And there you have it: the large black 'X' is our MAP estimate (those contours are the KDE of the posterior).


[under/over]-sampling teaches model the wrong distribution?

Yes, teaching the model to expect a balanced distribution definitely will impact the results on the test set. Oversampling the minority class to balance the distribution will make the classifier more likely to predict that a given example is the minority class. Each iteration of gradient descent will push the model to a location where approximately half of the loss was contributed by examples from a minority class, which is not the situation when you're testing the model. There are a few things that I've seen that try to overcome this issue; focal loss is one example, where easily classified examples (predictions that the model has a lot of confidence in) contribute less to the loss than examples that are difficult to classify. Another approach that I've seen used is training a model on an artificially balanced training set, and then freezing all of the weights except the last layer (or last few layers), and finetuning the model on the true distribution. This usually increases the precision of the model.
Accuracy is a misleading metric in this situation. As you noted, the model trained with a balanced training set performs worse when compared to a model that predicts everything as class A. Despite the fact that the accuracy of the second model is higher, the predictions don't make sense, and don't reflect the nature of the data. I would encourage you to examine the precision and recall of the two models to get an idea of where the model is doing poorly.
Also, I think your validation split should have the same statistics as the test set. When the validation data doesn't reflect the test data, you're essentially optimizing your model to hit a different target than you actually want.


Does Amazon RedShift replace Hadoop for ~1XTB data?

tl;dr: They markedly differ in many aspects and I can't think Redshift will replace Hadoop.

-Function
You can't run anything other than SQL on Redshift. Perhaps most importantly, you can't run any type of custom functions on Redshift. In Hadoop you can, using many languages (Java, Python, Ruby.. you name it). For example, NLP in Hadoop is easy, while it's more or less impossible in Redshift. I.e. there are lots of things you can do in Hadoop but not on Redshift. This is probably the most important difference.

-Performance Profile
Query execution on Redshift is in most cases significantly more efficient than on Hadoop. However, this efficiency comes from the indexing that is done when the data is loaded into Redshift (I'm using the term indexing very loose here). Therefore, it's great if you load your data once and execute multiple queries, but if you want to execute only one query for example, you might actually lose out in performance overall.

-Cost Profile
Which solution wins out in cost depends on the situation (like performance), but you probably need quite a lot of queries in order to make it cheaper than Hadoop (more specifically Amazon's Elastic Map Reduce). For example, if you are doing OLAP, it's very likely that Redshift comes out cheaper. If you do daily batch ETLs, Hadoop is more likely to come out cheaper.

Having said that, we've replaced part of our ETL that was done in Hive to Redshift, and it was a pretty great experience; mostly for the ease of development. Redshift's Query Engine is based on PostgreSQL and is very mature, compared to Hive's. Its ACID characteristics make it easier to reason about it, and the quicker response time allows more testing to be done. It's a great tool to have, but it won't replace Hadoop.

EDIT:  As for setup complexity, I'd even say it's easier with Hadoop if you use AWS's EMR. Their tools are so mature that it's ridiculously easy to have your Hadoop job running. Tools and mechanisms surrounding Redshift's operation aren't that mature yet. For example, Redshift can't handle trickle loading and thus you have to come up with something that turns that into a batched load, which can add some complexity to your ETL.


Alternative regression model algorithms for machine learning

Multiple regression
multivariate regression
Mixed Models
elastic search
Partial Least Squares regression
Logistic regression
Bayesian regression and many more..

When you google you'll find first page suggestions like https://www.listendata.com/2018/03/regression-analysis.html , https://www.techfunnel.com/information-technology/regression-analysis-types/ and https://www.kaggle.com/eraaz1/a-comprehensive-guide-to-advanced-regression


How to prepare masks for multiclass semantic segmentation?

You should create a separate binary mask (1 for the pixlels belonging to that class and 0 for the rest of pixels) for each class. Therefore, your mask array should have a shape of (BATCH_SIZE, WIDTH, HEIGHT, NUM_CHANNELS), where NUM_CHANNELS is the number of class.


You can indeed use the conversion table:

conversion <- read.table(text = "ACT SAT
8   200
9   210
10  220
11  225
12  250
13  285
14  325
15  360
16  385
17  410
18  440
19  465
20  485
21  505
22  525
23  545
24  560
25  575
26  595
27  615
28  635
29  655
30  675
31  700
32  725
33  750
34  775
35  790
36  800", header = TRUE)

With the help of this table and mathematical/logical operators, you can create the values:

transform(df, newgrade = (sat | act) *
            (conversion$SAT[match(df$act, conversion$ACT)] ^
               as.logical(act) * !sat) + sat)

The result:

   student sat act newgrade
1        1 365  28      365
2        2   0  20      485
3        3 545   0      545
4        4 630   0      630
5        5 385  16      385
6        6 410  17      410
7        7   0  35      790
8        8 655  29      655
9        9   0  21      505

The value in newgrade will also be 0 if both sat and act are 0.


Check the columns inside your dataframe.

You can try:

X.count()

to see how many data entries each column has.

From there, you can try:

.fillna()

or go back to your data and fill the necessary information or use sklearn.preprocessing.Imputer encode mean / median imputation for missing values.

Make sure that each column doesn't have any string or letter in it.
Check also each column and get their max value, maybe you can convert them in much more smaller number (e.g. Normalizing)

Also, try this:

np.isnan(X)

to get a boolean form of your dataframe to check if there's any blanks


Alternatives to Heat Maps for Communicating Daily Data

Thanks macaw_9227, that's a great idea. Below are two other ideas I'm considering.


r glm - Error in names(coef) <- xnames only for 2 columns in data

The problem is with datatype integer64. I don't know why this is the problem, but change it to type integer.

ratings = as.integer(ratings)

Normally, I have not seen anyone using integer64 datatype.


What you are trying to do here is named-entity recognition. Namely, the task consists of classifying substrings into a set of named entities (i.e. person, location, etc.). From a more formal perspective, this is a sequence labeling task that classifies parts of a sequence.
This task can be approached in different ways:

gazetteers/string matching
regular expressions
machine learning

I would highly recommend using SpaCy. They allow you to customize and retrain a model with your own data and labels, and typically, for such a use case, the model will perform well without requiring massive amounts of data.


How to fill missing values by looking at another row with same value in one column(or more)?

Sorted and did a forward-fill NaN

import pandas as pd, numpy as np
data = np.array([[1,2,3,'L1'],[4,5,6,'L2'],[7,8,9,'L3'],[4,8,np.nan,np.nan],[2,3,4,5],[7,9,np.nan,np.nan]],dtype='object')
df = pd.DataFrame(data,columns=['A','B','C','D'])

df.sort_values(by='A',inplace=True)
df.fillna(method='ffill')


Random forest, predicts the class with highest probability estimate. The predicted class probabilities of an input sample is computed as the mean predicted class probabilities of the trees in the forest. The class probability of a single tree is the fraction of samples of the same class in a leaf.

Majority voting, which is also called Hard Voting, every individual classifier votes for a class, and the majority wins. In statistical terms, the predicted target label of the ensemble is the mode of the distribution of individually predicted labels.

Majority voting may works better in cases where there are some outliers. Consider these votes: $\{0.51, 0.51, 0.51, 0.01\}$ and $\{1,1,1,0\}$.


Using the example dataframe provided by@JahKnows:

import pandas as pd

df = pd.DataFrame(data={'shopify_customer_id': [1,1,2,3,4,5],
                        'financial_status': ['paid', 'refunded', 'paid', 'refunded','paid', 'paid'],
                        'count': [8, 1, 13, 1, 1, 1],
                        'order_amt': [19, 19, 0, 19, 0, 24]})

You could also do the following:

df=df.groupby(['shopify_customer_id','financial_status']).sum().unstack()
df.columns = ['_'.join(col).strip() for col in df.columns.values]
df.reset_index(inplace=True)
df.fillna(0,inplace=True)

That gives you this (if this is what you intend to have!):

   shopify_customer_id  count_paid  count_refunded  order_amt_paid  order_amt_refunded
0                    1         8.0             1.0            19.0                19.0
1                    2        13.0             0.0             0.0                 0.0
2                    3         0.0             1.0             0.0                19.0
3                    4         1.0             0.0             0.0                 0.0
4                    5         1.0             0.0            24.0                 0.0


One way or another any data can be represented in a table or even in a big binary string, since after all the physical memory of a computer is just one big binary sequence. But the question is whether the table format adequately represents the semantics of any data, and the answer is definitely no: while there are tabular representations for graphs, text, images or videos, these representations simplify and/or make assumptions about the nature of the data. Virtually any representation is a simplification of the data it represents, but some representations are more faithful to the original data than others.


Tensorboard with pytorch dont display a graph

I had the same problem.
Check out this thread:
https://github.com/pytorch/pytorch/issues/24157

TLDR: Update PyTorch to PyTorch-nightly and the problem should be solved.
https://pytorch.org/get-started/locally/


There is no gradient with respect to non maximum values, since changing them slightly does not affect the output. Further the max is locally linear with slope 1, with respect to the input that actually achieves the max. Thus, the gradient from the next layer is passed back to only that neuron which achieved the max. All other neurons get zero gradient.

So in your example, $\delta_i^l$ would be a vector of all zeros, except that the $i^*$th location will get a values $\left\{\delta_j^{l+1}\right\}$ where $i^* = argmax_{i} (z_i^l)$


Accuracy after selftraining didn't change

Well, that is a bit of a turn down but: your model has limitations.

If the 50.000 data forms a complete set for your problem that means that more data won't be needed or helpful.

What do I mean by complete set is: there are enough samples to form a full rank correlation matrix in your feature space. So from your samples you can get a set that can generate all other samples in your feature space by linear combination.

Also, while your data might represent everything a decision three needs to know for classificating your data in the generated feature space, there may be other feature spaces that benefit from the extra data (such as deeper trees or other models)

You might try helping you decision tree by providing a few normalizations for data and feature engineering


Removing strings after a certain character in a given text

For instance:

 rs<-c("copyright @ The Society of mo","I want you to meet me @ the coffeshop")
 s<-gsub("@.*","",rs)
 s
 [1] "copyright "             "I want you to meet me "

Or, if you want to keep the @ character:

 s<-gsub("(@).*","\\1",rs)
 s
 [1] "copyright @"             "I want you to meet me @"

EDIT: If what you want is to remove everything from the last @ on you just have to follow this previous example with the appropriate regex. Example:

rs<-c("copyright @ The Society of mo located @ my house","I want you to meet me @ the coffeshop")
s<-gsub("(.*)@.*","\\1",rs)
s
[1] "copyright @ The Society of mo located " "I want you to meet me "

Given the matching we are looking for, both sub and gsub will give you the same answer.


The main rule when doing mean encoding is not using the test data. As long as you don't use the test data, you'll be mostly alright.
That being said, if you encode with all your training data you're probably going to see a drop of performance in the test set, which will not happen if you encode in a cross-validated fashion. So, the second option has some advantages with respect to the first one, but the first one is simpler. Still, I'd go for the second one.
In addition, you can add regularisation to the mean as done in the python package category-encoders, and it will be more robust to overfitting. This package allows for other options like leave one out encoding, that ensures that a sample's target is not used to compute the encoding for that sample.


Finding best match/score between 2 datasets

Welcome to the site! It is hard to answer this without more details (how much data do you have? On what parameters do you want to score the data?).
If you have some previous training data, you can look at recommendation algorithms with side information.
If you don't have any training data, then you essentially want to compute text similarity between the job spec and candidate's skills. For this read up on text similarity measures, cosine similarity etc. The first step in building such unsupervised algorithms is representing your text data in terms of vectors, basics of which are explained here.