Varying results when calculating scatter matrices for LDA

The LDA crashes for the exact reason you suspected.  You have
complex eigenvalues.  If you use np.linalg.eigh, which was
designed to decompose Hermetian matrices, you will always get real
eigenvalues.  np.linalg.eig can decompose nonsymetric square
matrices, but, as you've suspected, it can produce complex
eigenvalues.  In short, np.linalg.eigh is more stable, and I would suggest using it for both PCA and LDA.
Dropping the complex part of the eigenvalues may have been acceptable in your specific example, but in practice it should be avoided.  Depending on the size of the complex part of the number, it can significantly change the result.  For example think of the multiplication of two complex conjugates.  $(3+ .1i)*(3-.1i)=9-.01=9.01$ comared to $9$ when droping the complex part is a relatively safe, but $(3-2i)*(3+2i)=13$ compared to $9$ is a significant miscalculation.  Using the above method for the eigen-decomposition will prevent this situation from arising.
Remember that one of the assumptions of LDA is that the features are normally distributed and independent of each other.  Try running print('Class label distribution: %s' % np.bincount(y_train)[1:]).  If the counts are not close to being equal, you've violated the first asumption of LDA, and the within-class scatter matrix must be scaled, in short divide each count by the number of class samples $N_i$.  By doing this it should be obvious that computing the normalized scatter matrix is the same as computing the covariance matrix $\Sigma_i$.  $$\Sigma_i=\frac{1}{N_i}S_W=\frac{1}{N_i}(x-m_i)(x-m_i)^T $$
Make sure your scaling your features before you do your PCA/LDA.
If the above doesn't fix your eigenvector verification step I suspect the problem is that the eigenvectors are scalled differently.  Remember from your linear algebra class that a single eigenvalue, $\lambda_i$, has infinitely many eigenvectors, each being a scalar multiple of the others.  $v_i=[1,2,3]$ and $v_i=[2,4,6]$ can both be the eigenvector of $\lambda_i$.  So while you may get different values when calculating values at any given step after the decomposition, the end result should be the same.

Below is a template I use for LDA data compression.  It assumes that you've split your data into a training and test set, the feature space has been properly scaled, and there are three classes in your label vector (you can adjust accordingly).  It plots the individual and cumulative "discriminability" of each linear discriminant and then relies on the lda package in sklearn to transform the feature space using the number of discriminants you intend on using (here I chose to use the first 2).  It also scales the within class scatter matrices by default.

LINEAR DISCRIMINANT ANALYSIS

calculate mean vectors

mean_vecs = []
for label in range(1, 4):
    mean_vecs.append(np.mean(X_train_std[y_train==label], axis=0))
    print('MV %s: %s\n' %(label, mean_vecs[label-1]))

calculate within-class scatter matrix

d = X_train_std.shape[1]
S_W = np.zeros((d, d))
for label, mv in zip(range(1, 4), mean_vecs):
    class_scatter = np.cov(X_train_std[y_train==label].T)
    S_W += class_scatter
print('Scaled within-class scatter matrix: %sx%s' % (S_W.shape[0], S_W.shape[1]))

calculate between-class scatter matrix

mean_overall = np.mean(x_train_std, axis=0)
S_B = np.zeros((d, d))
for i, mean_vec in enumerate(mean_vec):
    n = X_train_std[y_train==i+1, :].shape[0]
    mean_vec = mean_vec.reshape(d, 1)
    mean_overall = mean_overall.reshape(d, 1)
    S_B += n * (mean_vec - mean_overall).dot((mean_vec - mean_overall).T)
print('Between-class scatter matrix: %sx%s' % (S_B.shape[0], S_B.shape[1]))

eigen decomposition

eigen_vals, eigen_vecs = np.linalg.eigh(np.linalg.inv(S_W).dot(S_B))
eigen_pairs = [(np.abs(eigen_vals[i]), eigen_vecs[:,i]) for i in range(len(eigen_vals))]
eigen_pairs = sorted(eigen_pairs, key=lambda k: k[0], reverse=True)
print('Eigendecomposition: \nEigenvalues in decreasing order:\n')
for eigen_val in eigen_pairs:
    print(eigen_val[0])

plot discriminablithy and select number of linear discriminants

tot = sum(eigen_vals.real)
discr = [(i / tot) for i in sorted(eigen_vals.real, reverse=True)]
cum_discr = np.cumsum(discr)

plt.bar(range(1, 14), discr, alpha=0.5, align='center',
        label='individual "discriminability"')
plt.step(range(1, 14), cum_discr, where='mid',
         label='cumulative "discriminability"')
plt.ylabel('"discriminability" ratio')
plt.xlabel('Linear Discriminants')
plt.ylim([-0.1, 1.1])
plt.legend(loc='best')
plt.tight_layout()
plt.show()

from sklearn.lda import LDA

lda = LDA(n_components=2)
x_train_lda = lda.fit_transform(X_train_std)
x_test_lda = lda.transform(X_test_std)
print('Features projected onto %d-dimensional LD subspace' % lda.n_components)


I have been using one heuristic to detect outlier values that is very simple. Start by calculating an interval centered on the mean and with three standard deviations in radius. Next, calculate a second interval using Tukey's box and whiskers measures, placing the interval limits at the whiskers' limits. Finally, calculate the union of both intervals and use this new interval to detect your outliers: any observation outside this interval is a potential outlier. Please note that you may have to tweak both intervals to calibrate for your desired precision. Also, you may have to take into account possible trends and seasonalities in these intervals, depending on how the historic data behaves.


A Simple Explanation of ZooKeeper in Hadoop

I noticed you are asking several questions related to the Hadoop ecosystem. It is hard to judge how much you know while explaining the things you are asking in a very simplistic manner without knowing that. Might be worthwhile giving some background of what you are trying to understand. But I digress.

Unless you are building an application that requires coordination (think race conditions) you more than likely won't need Zookeeper directly. Many tools within the Hadoop ecosystem do use it for this exact purpose (e.g. HBase).

This is probably the best and most simplistic answer I can give.

  In a nutshell, ZooKeeper gives you the tools to help build distributed applications. For example, suppose you have a distributed web server application running on 10 nodes. Say, you want to get total real-time hit count. One way to do this is to write an application which connects to the 10 nodes, gets count from each and present the sum. Alternatively, you can have each web server application write their hit counts to ZooKeeper on regular intervals and then query ZooKeeper to get the count.

Source: https://10kloc.wordpress.com/2013/12/21/quick-intro-to-apache-zookeeper/

ZooKeeper Quorum is "A replicated group of servers in the same application is called a quorum, and in replicated mode, all servers in the quorum have copies of the same configuration file."

Source: https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html

A more detailed answer of what Zookeeper is and how it is beneficial can be found on SO (https://stackoverflow.com/questions/3662995/explaining-apache-zookeeper).


What happens if we use Z-Score(Mean Normalisation) in Neural Networks to standardize values

By normalizing the data the code is not likely to get negatively affected. This is because the network doesn't know a priori that images are being used as input data, it only receives a set of numeric values (which can represent any type of data) and finds the appropiate values of weights and biases that decrease our cost function.
Thereby even if we get negative values as the new input data, that doesn't mean that this preprocessing is going to cause a bad effect.
But, why normalization may be useful?
The useful thing about Z-score normalization here is that by doing it, all the "new pixels" have a value with zero-mean and the same variance. This is a good thing as explains Yann LeCun in his paper "Efficient Backprop" (page 8):

Why use data with features having a mean close to zero:

Convergence is usually faster if the average of each input variable over the training set is close to zero...  ....When all of the components
of an input vector are positive, all of the updates of weights that feed into a
node will be the same sign. As a result, these weights can only
all decrease or all increase together for a given input pattern. Thus, if a weight
vector must change direction it can only do so by zigzagging which is inefficient
and thus very slow

The expression that he is refering to is the one that is used to update the value of the weights of the first layer, which is given by:
$$ \frac{\partial C}{\partial w_{jk}^{l=1}} = \delta_j^{l=1}x_k$$
Where $x_k$ represents an input pixel value and $w_{jk}^l$ a weight that connects that input $k$ to the neuron $j$ of the first layer ($l=1$). So, given the scalar term $\delta_j^{l=1}$ it's clear that all the weights that connect the input layer to the first layer of neurons will be updated in the same direction if all $x_k$ are positive, as the quote explains.

Why use data with features having the same variance?:

Scaling speeds
learning because it helps to balance out the rate at which the weights connected
to the input nodes learn.

As we saw earlier, the expression that gives the updates on the weights $w_{jk}^{l=1}$ are proportional to the input $x_k$. So if e.g. $x_1$ samples have bigger values than $x_2$ samples, then the updates on $w_{j1}$ on $w_{j2}$ may not be balanced and therefore the updates are made with different rythms on both parameters.
Just a side note $\rightarrow$ The method of dividing each feature (each pixel intensity) by their maximum value is not a Z-score normalization but is another way of normalizing the data (in order to use the Z-score, the quantity that we would have to use to divide the features is the standard deviation of each pixel throughout all the samples).


Data Leakage when preprocessing categorical features?

Think of the data being organized in rows (instances) and columns (features). Any pre-processing step which mixes information between/across rows can lead to data leakage.
A typical example is standardization or normalization. Applying min-max-scaling, for example, on the whole dataset would leak information since it is an aggregation across rows/instances. Another typical example is encoding categorical variables. When you train your encoder before the split you're taking into considerations values which you should not have seen before. Because when applied on real data, you might be presented unknown data values too.
The guiding principle here is, that your test strategy should resemble the real application in order to estimate the ability of your model to generalize to unseen data.
The safest way to avoid data leakage is to split the data before applying any pre-processing. Moreover, this approach supports designing the pipeline in a way that you can apply it to validation/test and later production data the same way you applied it on training data.


There are multiple techniques that help with the problem you sketch, the applicability of which usually depends on the classification technique and the corpus. But I get the idea you would be helped by some practical examples. So let's go over some of them. Feel free to comment if I tread over familiar grounds, or you want me to elaborate on some of them. Or where to start experimenting with them

Stopping A simple technique is applying a stoplist: A list of common words that should be removed. There are pre-packaged lists, but most packages allow you to provide your own.
TF/IDF A technique that transforms your features by weighing words by their term frequency (how often do they occur in the document) divided by the document frequency (how often does the word occur in other documents. This way frequent words are made less relevant to the document
POS Many packages offer you a part-of-sentence-tagger, that will tag the words by their grammatical function (Verb for instance). You can leverage that in the tokenization step to filter out words (usually you'll look for verbs and nouns). Some vectorizers can do this straight out. (This could also be done with NER)
Stemming transforms inflections of words (eg: train/trains) to a stem. This might make some of your words a little more relevant by upping the chance a pair of them collides
Restricting your vectorizer: Most packages sport a vectorizer that you can instruct to either look for a minimum/maximum document count (ignore words that either occur in to many different documents, or to little different documents), or to cap the amount of features (words). Capping the amount of words usually selects for most frequently used words.
Encoding word/token based features to more semantic features: Word2Vec, but also older techniques such as LDA/LSI.
Picking the classification algorithm: Some algorithms are very capable of handeling large feature spaces (Naive Bayes for instance), some algorithms learn to transform the feature space to find better ways to weigh the features.

Packages such as Sklearn, NLTK and Gensim offer most of these techniques.

Let me know if this was helpful


Is it better to have higher train accuracy with lower test accuracy or higher test accuracy with lower train accuracy?

Test accuracy better reflects generalization error, so you want the one with higher test accuracy. In your first setup, the higher train accuracy indicates overfitting, as it's significantly higher than train accuracy. This is also kind of why it generalizes less well than the second one.


Google OR-Tools - Routing - Penalties - Adding Different Penalty to Different Location (Python)

Given your code, it looks like data['penalty'][manager.NodeToIndex(node)] is not an integer, probably a float, and therefore it doesn't match the function signature, as the error message suggests. Try converting it to an integer.


Is there any public implementation / publication with Hintons capsules idea?

Following the publication of Dynamic Routing of Capsules (https://arxiv.org/abs/1710.09829), there have been open source implementations of this idea released. See https://github.com/XifengGuo/CapsNet-Keras and https://github.com/soskek/dynamic_routing_between_capsules for two such examples.


Calculation and Visualization of Correlation Matrix with Pandas

Another alternative is to use the heatmap function in seaborn to plot the covariance. This example uses the Auto data set from the ISLR package in R (the same as in the example you showed).

import pandas.rpy.common as com
import seaborn as sns
%matplotlib inline

# load the R package ISLR
infert = com.importr("ISLR")

# load the Auto dataset
auto_df = com.load_data('Auto')

# calculate the correlation matrix
corr = auto_df.corr()

# plot the heatmap
sns.heatmap(corr,
        xticklabels=corr.columns,
        yticklabels=corr.columns)

If you wanted to be even more fancy, you can use Pandas Style, for example:

cmap = cmap=sns.diverging_palette(5, 250, as_cmap=True)

def magnify():
    return [dict(selector="th",
                 props=[("font-size", "7pt")]),
            dict(selector="td",
                 props=[('padding', "0em 0em")]),
            dict(selector="th:hover",
                 props=[("font-size", "12pt")]),
            dict(selector="tr:hover td:hover",
                 props=[('max-width', '200px'),
                        ('font-size', '12pt')])
]

corr.style.background_gradient(cmap, axis=1)\
    .set_properties(**{'max-width': '80px', 'font-size': '10pt'})\
    .set_caption("Hover to magify")\
    .set_precision(2)\
    .set_table_styles(magnify())


Machine Learning - How to predict set of fixed fields based on past features

It's a supervised classification problem: you're trying to predict the destination (class) based on some categorical features (input columns). I would suggest starting with some simple algorithms such as decision trees or Naive Bayes.

However I'm guessing that logistical shipments can evolve over time: maybe a shipper business grows with country X but decreases with country Y, etc. If this is relevant, it might make sense to look into more advanced methods which could take the chronological evolution into account (time series).


High probabilities of success for wrong predictions

It is difficult to diagnose without having more info. But based upon what you mention I point:

Since you have a class imbalance, you MUST use regularizarided models that penalize the cost function  made by the minority class, more strictly than those errors for the majority class (Boosting models, Linear models)

Try using another metric like ROC AUC or Balanced Accuracy to measure the performance of your model so that you can see an adequate point of view of how your model is performing.
An additional advantage of using ROC Score is that you can find the "optimal" threshold being the one that maximizes Area under the roc curve when you make predictions with your model

This is the most important part. Use a technique to debug your model like SHAP values,Partial dependence plot in order to understand how your model is learning and the relationships between the features and the target

If you are using ensemble methods, you can set something named Monotonicity constraints That will allow you to add business knowledge to the model in order to correct potential non-sense relationships that the model may find


Filling missing data with other than mean values

There are of course other choices to fill in for missing data. The median was already mentioned, and it may work better in certain cases.

There may even be much better alternatives, which may be very specific to your problem. To find out whether this is the case, you must find out more about the nature of your missing data. When you understand in detail why data is missing, the probability of coming up with a good solution will be much higher.

You might want to start your investigation of missing data by finding out whether you have informative or non-informative missings. The first category is produced by random data loss; in this case, the observations with missing values are no different from the ones with complete data. As for informative missing data, this one tells you something about your observation. A simple example is a customer record with a missing contract cancellation date meaning that this customer's contract has not been cancelled so far. You usually don't want to fill in informative missings with a mean or a median, but you may want to generate a separate feature from them.

You may also find out that there are several kinds of missing data, being produced by different mechanisms. In this case, you might want to produce default values in different ways.


Depends on content, but I would probably go for "observed" (vs. "unobserved"). A suitable direct antonym of "missing" might be "extant".


The answer to this question is that it depends. The primary approach is to pass in the tokenized sentences (so SentenceCorpus in your example), but depending on what your goal is and what the corpus is you're looking at, you might want to instead use the entire article to learn the embeddings. This is something you might not know ahead of time -- so you'll have to think about how you want to evaluate the quality of the embeddings, and do some experiments to see which 'kind' of embeddings are more useful for your task(s).


Mixed effect random forest model for Python Windows

I have tried installing "merf" on Windows 10 - it worked. I did it like this:

pip install merf


Model Dump Parser (like XGBFI) for LightGBM and CatBoost

Both LightGBM and CatBoost have functions that give you the models' feature importances. They are described in the respective docs here:

LightGBM: plot_importance and lightgbm.importance
Catboost: has get_feature_importance which is described in more detail on its concepts page


How to estimate student's level according to score and time spend

I think there will be different approaches, and different results, to this problem.
The first which comes to my mind is simple:

evaluate the difficulty of each topic, by the average of the time spent by the students on it;

use the evaluations at point 1) to weight the time spent by the students on the topics;

calculate the average weighted time spent on all the topics;

a) you can rank the students based on the average weighted times, or
b) you can plot an histogram of the average weighted times, and choose some proper tresholds to classify the students


How do I transform a file to .txt file using pandas?

pandas.to_csv() as you might know is part of pandas owned IO-API (InputOutput API).
Currently panas is providing 18 different formats in this context.
And of course pandas is also able to write txt-files. With other words pandas is a multi data converter:
Pandas IO tools
To do so with pandas use to_csv with the sep='\t' attribute:
df.to_csv('data.txt', sep='\t')
The next option you have to do it with numpy in that way, which is a bit laborious:

Call pd.DataFrame.to_numpy() to convert a pd.DataFrame to a NumPy array
Use np.savetxt(fname, X, fmt = "%s") with the string "%d" as fmt and the NumPy array as X to write the contents of X into the file fname.


Accuracy (for classification problems)
Precision
Recall
F1 Score
AUC-ROC, particularly for imbalanced datasets

Good performance on the training set and bad performance on the test set is due to overfitting. So you should try to find ways to tackle overfitting, such as regularization of parameters, parameter tuning using cross-validation etc...


Clearly, the optimization of your loss function led to it only predicting the most common words in English. The probability of other words are obviously small.
I am no expert, but I would check three things. 1. How long does it take for the loss to converge? With the output you show it is likely it does not take a long time. 2. Maybe try to filter out so you have different words in the sentences. 3. Check if it can help to optimize another loss function which would have a bias towards more diverse sentences.
I hope this can give you a few ideas at least.


How to do template matching without opencv?

import numpy as np
import matplotlib.image as img
import matplotlib.pyplot as plt
from skimage.metrics import structural_similarity as ssim

def rgb2gray(rgb):
    return np.dot(rgb[...,:3], [0.299, 0.587, 0.144])

full_image = rgb2gray(img.imread("img/full_image.png"))
sub_image = rgb2gray(img.imread("img/sub_image.png"))

full_w,full_h = full_image.shape[:2]
sub_w,sub_h = sub_image.shape[:2]

print(full_w,full_h)
print(sub_w,sub_h)

winW = 0
found = False
while winW < full_w - sub_w and found == False:
    winH = 0
    while winH < full_h - sub_h:
        window = full_image[winW:winW+sub_w, winH:winH+sub_h]
        if ssim(sub_image, window) > 0.80:
            found = True
            print("found", ssim(sub_image, window))
            plt.imshow(window)
            break
        winH += 2
    winW += 2


Rearranging data frame from column names to key value pairs

This is a simple gather() using package tidyr:

> library(tidyr)
>
> gather(a, key = "k", value = "v")
  k v
1 a 1
2 a 2
3 b 3
4 b 4


A the end of a big DS project, should I make trained models available on GitHub?

From what I've seen on Github while looking for open source projects is that people usually do both.

You can have a section where one loads the models and runs the inference, and another section where you let the user train the models from scratch using your code.

I recommend doing this since some people do not want to retrain the model, especially if it's memory consuming (such as computer vision problems), where they only need to see the same results that you had. People who need to train models usually have a different need than what you had in your results, therefore you should let them tailor the results for their needs.


Machine learning learn to work well on future data distribution?

It has been studied under various names likes Domain Adaptation,   Sample Selection Bias, Co-variate Shift.

Please go trough this survey paper on Transfer learning. It covers all the possible combination like

1)same distribution for train and test data

2) Gradual change between train and test distribution

3) Different but related distribution for train and test

It'll also give you all the necessary resources required to study further on this topic.


The number of hidden units is a hyperparameter which increases the model capacity. Therefore, increasing it not only increases time and memory requirements but can also lead to overfitting (also see section 11.4.1 in the Deep Learning Book).


Technology stack for linear regression on (not so) large dataset

Since no one has mentioned RAM-efficient methods yet:

Instead of loading everything into RAM, you can use online/out-of-core learning.

Python's Scikit-Learn for example has the SGDClassifier class. Set its loss function to "log" and you get logistic regression. Using the partial_fit function you can feed it small batches of data that you read straight from the database (or from some CSV file,...).

Vowpal Wabbit might also be worth a try. Its made for out-of-core learning - hardly uses any RAM and you won't find anything that's much faster.

You could also use Python's Keras library to build a neural network (or in the simplest case just logistic regression), which you can also feed with small batches of data instead of loading everything into RAM. Compared to the other two recommendations a neural network could also learn non-linear dependencies.

Besides that, try to start with fewer samples - plot the learning curves with 10k, 100k, 1M samples and see if 100M samples are even necessary to get a good score.


Bellman operator and contraction property

The proofs I have seen for contraction of the bellman operator (this page has a really nice run through, explicitly utilize the fact that an infinity norm is being used. This does not necessarily mean that contraction doesn't occur under other norms, but it does suggest one of two possibilities:

The contraction proof is only valid under the infinity norm.
The infinity norm is just the easiest metric to prove the contraction property.

When showing that the Bellman Operator converges to a fixed point it is satisfactory to simply show that it is a contraction, it doesn't matter what sort of contraction it is, so we would typically prove the contraction that is easiest to show.
That being said, intuitively, I would imagine that the question of whether or not this contraction holds under other norms would impact questions surrounding the rate of convergence to the fixed point (ie., how many times we must apply the bellman operator for convergence). After a cursory search I was not able to find any theory explicitly proving or disproving the contraction under other norms. This might be an interesting question to raise in the mathematics stack exchange.


Why CNN doesn't give higher accuracy over simple MLP network? [From Keras examples]

CNN (and RNN) models are not general improvements to the MLP design. They are specific choices that match certain types of problem. The CNN design works best when there is some local pattern in the data (which may repeat in other locations), and this is often the case when the inputs are images, audio or other similar signals.

The reuters example looks like a "bag of words" input. There is no local pattern or repeating relationships in that data that a CNN can take advantage of.

Your results with a CNN on this data set look reasonable to me. You have not made a mistake, but learned how a CNN really works on this data.


Terminology to distinguish between ML methods and optimization methods (PSO, ACO..)

In a wider context every machine learning method can be re-cast as some type of optimisation problem. For example for Neural Networks the associated optimisation problem is "find the weights which minimise some loss function of the data given an architecture". This is solved using back-propagation (which is a layered gradient-descent method) and when minima of the loss function are found we say the systen has "learned".
On the other hand there is nothing stopping us from re-casting optimisation problems as "learning" problems
.
A possible differentiation (based on what you mention in the question) is between analytic methods and non-analytic methods.
A. Analytic Methods: Gradient-based, Primal-Dual etc... (eg NNs, SVMs)
B. Non-Analytic Methods: Particle systems, Genetic/Evolutionary systems, Simulation methods, Stochastic methods, etc..
The basic differrence is both whether the form of the objective function is known and if analytic methods (eg gradient-descent vs particle systems) are used to find optima.
Again, there is nothing, in principle, stopping us from using these non-analytic optimisation methods to do machine learning.


How do I change the location of Google BigQuery

Based on my experience with how finicky CSV can be, I strongly suspect an encoding problem. A lot of CSV readers simply throw-away the data if they encounter illegal characters or when they can't figure it out. I don't know what BigQuery does in that regard, but considering their help on loading CSV files it's worth a shot to make sure your data is clean and according to how BigQuery wants it before attempting anything else.

  Limitations

  When you load CSV data from Cloud Storage into BigQuery, note the following:

  CSV files do not support nested or repeated data.
  If you use gzip compression BigQuery cannot read the data in parallel. Loading compressed CSV data into BigQuery is slower than loading uncompressed data.
  You cannot include both compressed and uncompressed files in the same load job.
  When you load CSV or JSON data, values in DATE columns must use the dash (-) separator and the date must be in the following format: YYYY-MM-DD (year-month-day).
  When you load JSON or CSV data, values in TIMESTAMP columns must use a dash (-) separator for the date portion of the timestamp, and the date must be in the following format: YYYY-MM-DD (year-month-day). The hh:mm:ss (hour-minute-second) portion of the timestamp must use a colon (:) separator.

  CSV encoding

  BigQuery expects CSV data to be UTF-8 encoded. If you have CSV files with data encoded in ISO-8859-1 (also known as Latin-1) format, you should explicitly specify the encoding when you load your data so it can be converted to UTF-8.

  Delimiters in CSV files can be any ISO-8859-1 single-byte character. To use a character in the range 128-255, you must encode the character as UTF-8. BigQuery converts the string to ISO-8859-1 encoding and uses the first byte of the encoded string to split the data in its raw, binary state.

BigQuery expects UTF-8, you provide CP-1252. CP-1252 should, in my opinion, have died out by now. So either it only looks like such or you have a program that doesn't care much about following recent standards (those exist, I use one of them every day).

Also, make sure your localisation is set correctly. Some CSVs are separated by ; instead of , and some locales consider , a thousands separator while others see it as decimal separator. So please verify whether the locale of your CSV is the same as the locale of your schema.

Your default location can be set in the .bigqueryrc file. The schema's location can be set using the bq command, like this:

bq --location=location load \
--source_format=format \
project_id:dataset.table_name \
path_to_source \
schema

  Where:

  location is the name of your location. The --location flag is optional. For example, if you are using BigQuery in the Tokyo region, you can set the flag's value to asia-northeast1. You can set a default value for the location using the .bigqueryrc file.
  format is NEWLINE_DELIMITED_JSON or CSV.
  project_id:dataset.table_name is your project ID.
  project_id:dataset.table_name is the dataset that contains the table into which you're loading data.
  project_id:dataset.table_name is the name of the table into which you're loading data.
  path_to_source is the location of the CSV or JSON data file on your local machine or in Cloud Storage.
  schema is the inline schema definition.

  Example:

  Enter the following command to load data from a local CSV file named myfile.csv into mydataset.mytable in your default project. The schema is manually specified inline.

bq load \
--source_format=CSV \
mydataset.mytable \
./myfile.csv \
qtr:STRING,sales:FLOAT,year:STRING

More on locations here.

Both your encoding and locale are suspect based on the details you have provided.


How to associate array of class probabilities with corresponding image in keras?

Keras Model.predict() method doesn't shuffle the data, so each row in p_valid corresponds to the row in x_valid. For instance, p_valid[2] holds the probabilities for the image x_valid[2].

By the way, note that the values in each row don't necessarily sum up to 1, because they correspond to the probability of class i being present, thus are not exclusive.


How to improve naive Bayes multiclass classification accuracy?

If you are hand-coding, and found that you got the same prediction for everything in your test set, it is possible that you are multiplying feature probabilities until you hit the floating point limitations of your environment, ending up with a zero value, which probably matches one of your labels.

e.g., $0.001 * 0.0001 * 0.0002 * 0.0003 ... = 0$

Try adding log values instead.

e.g., $log(0.001) + log(0.0001) + log(0.0002) ... $


Continuous variable to categorical by quartiles?

Panda's Categorical Data Type is designed for that type of analysis,  pandas.cut can divide by user-defined bins and pandas.qcut can create quantile-based discretization. Something like this:

import numpy as np
import pandas as pd

df = pd.DataFrame({'value': np.random.randint(1, 80, 20)})
df['group'] = pd.cut(df.value,
                     bins=[0, 5, 31, 51, 80],
                     labels=["very short", "short", "long", "very long"])


How to feed training set labels into Keras LSTM

Your approch so far is not exactly right. Assuming you want to use the many to one approcah, the way you do it is as follows:

You have a dataset like this:

And you prepare your data in a way like this:

Source

For a timestep of 3, you discard the first 2 occurrences of your labels and use $y_{3}$ for the inputs $\{x_1, x_2,x_3\}$.

Next you don't start in $x_4$. Instead you use the next three-item slice $\{x_2, x_3, x_4\}$ with the training label $y_4$.

You continue this way until the last imput $\{x_{n-2}, x_{n-1}, x_{n}\}$ to predict $y_n$. If you have $n=30000$ samples, you can generate up to $n-2$ (29998) training samples for your model following this logic.


How to approach Multilingual Text Classification?

In a similar situation, -after trying some alternatives- I had to build a language classifier in front of all learning and classification steps.

That is, for learning:

Detect the language of the input (say, an enumeration like "DE", "EN", etc.)
Apply language specific stemming to the words of the input.
Prepend words in the learning phase with the language identifier (i.e. "de_du", "en_you")
Use these words in a single training model.

In classification phase, use the same steps.


Maximum number of images to train a convolutional neural network

Theoretically speaking, there is no limit on how many data points you can use to train any neural network. This is true because neural networks that can be trained incrementally and do not need to "see" the entire dataset.

Obviously, it helps if your machine is able to store at least a single image and the network in memory.

To train on a large number of images, you will likely need to use batches. This means that each iteration happens on a subset of your data, which can be as small as a single image.


Then check out this tutorial generative/abstractive text summarization.


How to filter Named Entity Recognition results

The most advanced (and complicated) approach to this is some sort of weakly-supervised system like Holoclean. It seems promising, but not easy.

The other end of the spectrum are heuristics like the one you propose. If you want to use a string distance approach like you describe, I would create some metric that gives points for both string edit distance and having matching words. If you can break the pipeline into smaller steps, maybe you can use some sort of topic analysis, then look for two similar entities and that topic in a knowledge graph, and see if the point to the same area.

The 2018 Alexa Prize winner paper, Gunrock is full of clever heuristics that you might be able appropriate.

Edit: you should also look at Entity Linking, which is more or less the problem you describe.


Looks like it requires R 3.6.0 or higher. At least that is what the docs here say.


Improvements to video-game cover CNN classifier (keras)

Here are some points to consider:

  1) I've written a Metacritic Scrubber which takes PS2, PS3, PS4, Xbox360 and Xbox One games which have metacritic scores, downloads the artwork and labels it with the score. Duplicate games are removed (which may be a mistake since they sometimes have different scores based on platform). I'm pretty happy with this code.

Good!

  2) I've rounded scores on a scale of 0-9 instead of 0-100 to make the number of classifications smaller.

This also should be fine.

  3) images are 123x98 with 3 channels. All images have been stretched to this size. I wonder if this may be a source of problems because some covers have been stretched. Values are between 0-255 for each channel.

I would normalise the values between 0-1 (norm_values = values/255). Regarding the size of the files I don't think this should be a problem. For example, Cifar-10 are 32x32 or the average image resolution on ImageNet is 469x387 pixels although most approaches resize them to 256x256. I think your size should be fine.

  4) Games with square cover art (DLC, non-retail, etc) have been omitted.

Not much to add here.

  This leaves me with a data set of 3816 game covers. I figured this would probably be enough for an initial investigation.
  The model I've built has been based on the work by Iwana et al in this paper: https://arxiv.org/abs/1610.09204

Here is where I actually think we have some problems. First, your dataset is quite small. 3816 reduced to 1908 when splitting train/test... This is not good especially with such structure. In the paper you mentioned, they are using a network with around 2.5M parameters and they used ~137K samples. Your 1908 dataset seems tiny compared and you are using the same model structure...

In my opinion your model is not able to do better, simply put. It doesn't matter what parameters you choose you need (a lot) more data. You might try reducing the size of your network, also create more data using noise addition, mirror samples, etc... and see if these help somehow.

Finally, apart from the size, you have no way to tell if your data is representative enough so that any model can learn from it. Therefore, 35% accuracy is as good as any other value I am afraid. Anyway, your mission seemed to be able to complete your experiment and you did. And you learnt a lot about limitations of deep models, so I'd say: good work!


What is the correct procedure when "joining" data takes ~6 hours?

There is no ideal way of doing that type of join efficiently in Pandas.

One possible option is using numpy.piecewise to define matches based on the timestamps from each dataframe. An example of apply numpy.piecewise can be found here.


Number of Fully connected layers in standard CNNs

I think the confusion with the Inception module is the somewhat complicated structure. The point on the relevant CS231n slide (#37), saying there are no FC layers is partially correct. (Remember this is only a summary of the model to get the main points across!).  In the actual part of the model being explained on that slide, they are referring only to the Inception modules:

No FC layers!

Definitions will, however, play a big role in deciding whether or not there are FC layers in the model.

In the bigger scheme of things (beyond a single Inception module), we have first to distinguish between the train and test time architectures.

At train time there are auxilliary branches, which do indeed have a few fully connected layers. These are used to force intermediate layers (or inception modules) to be more aggressive in their quest for a final answer, or in the words of the authors, to be more discriminate.

From the paper (page 6 [Szegedy et al., 2014]):

  One interesting insight is that the strong performance
  of relatively shallower networks on this task suggests that the features produced by the layers in the
  middle of the network should be very discriminative. By adding auxiliary classifiers connected to
  these intermediate layers, we would expect to encourage discrimination in the lower stages in the
  classifier, increase the gradient signal that gets propagated back, and provide additional regularization.

The slice of the model shown below displays one of the auxilliary classifiers (branches) on the right of the inception module:

This branch clearly has a few FC layers, the first of which is likely followed by a non-linearity such as a ReLU or tanh. The second one simply squishes the 1000 input weights into whatever number of classes are to be predicted (coincidentally or not, this is a 1000 here for ImageNet).

However, at test time, these branches are not active. They were used simply to train the weights of the modules, but do not contribute to the final classification probabilities produces at the end of the entire model architecture.

This all leaves us with just the suspsicious looking block right at the end of the model:

There is clearly a big blue FC layer there!

This is where definitions come into play. It is somewhat subjective.
Is a fully connected layer one in which each $m$ weight is connected to each of $n$ nodes? Is it a layer in which representation are learned, and if so, does the layer require a non-linearity?  We know that neural networks requires the non-linearities, such as ReLU and tanh functions to be applied to the outputs of a layer (thinking in forward flow). Without these, neural networks would simply be a combinations of linear functions, and so going deeper wouldn't theoretically add any power as we essentially would just be performing a huge linear regression.

In this spirit, we can look at the final piece of the puzzle, and point out that tis final FC layer is noted to simply be linear! That is, it takes all the weights resulting from the preceding Average Pooling layer, and combines them into a linear combination of only 1000 values - ready for the softmax. This can all be understood from the tabular overview of the network architecture:

So, do you agree with the stanford guys or not? I do!


Books about the "Science" in Data Science?

Introductory:

Machine Learning: The Art and Science of Algorithms that Make Sense of Data (Flach)
Learning From Data (Abu-Mostafa et al.)
Introduction to Statistical Learning (James et al.)

Digging deeper:

Elements of Statistical Learning (Hastie et al.)
Pattern Recognition and Machine Learning (Bishop)

Some special interest examples:

Convex Optimization (Boyd)
Bayesian Reasoning and Machine Learning (Barber)
Probabilistic Graphical Models (Koller)
Neural Networks for Pattern Recognition (Bishop)

Broader reference works on machine learning (not really what you asked for, but for completeness):

Machine Learning: A Probabilistic Perspective (Murphy)
Artificial Intelligence: A Modern Approach (Russell & Norvig)

Bonus paper:

Statistical Modeling: The Two Cultures (Breiman)


Python memory error: compute inverse of large sized matrix

Well it does not matter the language you will use, I am surprised you can even store that matrice in memory... I had a similar problem while researching on kernel methods.

First, to put in perspective how huge that matrix is, I will assume each element is 32 bit word: $ 200,000 \times 200,000 \times 32 = 1.28 \times 10^{12} $ bits or 149 GB

The bitlenght is variable but even for a boolean matrix that is about 4,65 GB and you said your data is a bit more complex than that.

Solution 1

If you trying to solve a linear system there are many iterative solutions that might help you computing an 200,000 x 1 array aproximation of your system answer without having to store that absurdly large matrix in memory. But you should be aware that this might take a bit long because you might have to load information from the disk.

Solution 2

Another possible solution, if you really need that inverse is if that matrix can be expressed as sumation of outer products of vectors. If so, you can do this:

$$
M = \sum v_i.u_i^T
$$

Get the first set element of the sum and invert it so you have $M^{-1}_{bad aproximation}$ matrix by using pseudoinverse, instead of doing the outerproduct you can invert $u_1$ and $v_1$ and use Moore-Penrose Inverse property:

$$
M^{-1}_{bad aproximation} = (v_1.u_1)^\dagger  = v_1^\dagger . u_1^\dagger
$$

Combine this with Sherman-Morrison formula to add the other $v_i$ and $u_i$ vectors.

Sherman-Morrison formula gives:
$$
(A + uv^T)^{-1} = A^{-1} - \dfrac{A^{-1}uv^TA^{-1}}{1 + v^TA^{-1}u}
$$

Solution 3

You might try using Schur Complement which allows you to do the following:

Given a matrix $M$, it can be expressed as
$$
M =
    \begin{bmatrix}
    A & B  \\
    C & D  \\
    \end{bmatrix}
$$
where $A$ and $B$ must be square you can solve the inverse of $M$ blockwise as:

$$
M^{-1} =
    \begin{bmatrix}
    (M/D)^{-1} & (M/D)^{-1}BD  \\
    -D^{-1}C(M/D)^{-1} & (M/A)^{-1}  \\
    \end{bmatrix}
$$
where $(M/D)$ denotes the schur complement of block D of the Matrix M and is defines as

$$
(M/D) = A - BD^{-1}C
$$

also
$$
(M/A) = D - BA^{-1}C
$$

If $A$ or $D$ are singular, the matrice $M$ is singular and a pseudo inverse might be used.

This can give a reduction from inverting a $200,000 \times 200,000$ matrice to inverting 4 $50,000 \times 50,000$ matrices and some matrice multiplication, also this can be implement in parallel applications and there is way to do this recursively given certain conditions (enabling a reduction greater than 4 in size)

The $50,000 \times 50,000$ matrices store is only 9.3 GB each and will be easier to invert

Hope it helps. Also you might want to search answers in Math Stack or in Stack Overflow and even ask there.

  Edit: Added another solution with Schur Complement


Have you considered using the Mahalanobis Distance? It's can be thought of as the magnitude of a multi-dimensional Z-score.

The way I prefer to view the the Mahalanobis Distance is as the square root of the exponent of the Multivariate Normal Distribution. This is similar to the Z-score, which is the exponent of the univarate normal distribution before the square is applied. A big difference between these two is the Z-score is signed, while the Mahalanobis Distance is unsigned, which doesn't matter for finding outliers anyway.

I understand you don't have normally distributed data, but sometimes pretending your data is normally distributed can have good results, so using the Mahalanobis Distance can be worth investigating.


What feature engineering is necessary with tree based algorithms?

Feature engineering that I would consider essential for even tree based algorithms are:

Modular arithmetic calculations: e.g. converting a timestamp into day of the week, or time of day. If your model needs to know that something happens on the third Monday of every month, it will be nearly impossible to determine this from timestamps.
On a similar vein, creating new features from the data you have available can drastically improve your predictive power. This is where domain knowledge is extremely important - if you know of, or think you know of a relationship then you can include variables that describe that relationship. This is because tree based methods can only create splits that are horizontal or vertical (i.e. orthogonal to your data).
Dimension reduction is typically performed by either feature selection or feature transformation. Reducing the dimension through feature selection will likely not help much with the models you mention, but an algorithm may or may not benefit from feature transformation (for example principal component analysis) depending on how much information is lost in the process. The only way to know for sure is to explore whether feature transformation provides better performance.


One-class classifier for time series data classification

Apart from the approach @Rolf Schorpion mentioned, there are others.
For example, you could use a deep neural network, specifically, an auto-encoder (see here for a tutorial).

But there's an important catch to all purely "data-driven" approaches: if the figure of 30 time series you mention in the comments is a typical order of magnitude for your training set, the results will be more or less arbitrary. If you don't define "accordance" in any way, this is a classification problem with only positive training data which consists of 30 data points with more than 100 features each.

Unless your data is very, very special (e.g., all time series are identical), there is just too much freedom in the solution of this problem. Different algorithms (or different parameter sets for the same algorithm) will use this freedom in very different ways. So you will probably see very different solutions when you experiment with different methods.

So if you don't want to do more or less arbitrary experiments and choose from the solutions afterwards, you have to use a method in which you somehow define the meaning of "in accordance" in advance. You may try and use traditional time series techniques to find a common model that fits each of the time series well. You could then postulate that "accordance" is a good fit to this model.

Or you might just do some exploratory analysis of your data to decide upon a simple rule which could be used to detect accordance. There's lots of possibilities, and without further details on the application it's difficult to decide which one is appropriate. Delegating the decision of what's "in accordance" to some algorithm doesn't seem to be a good choice in this case, though.


Understanding Reinforcement Learning with Neural Net (Q-learning)

In Q-Learning, on every step you will use observations and rewards to update your Q-value function:

$$
Q_{t+1}(s_t,a_t) = Q_t(s_t,a_t) +  \alpha [R_{t+1}+ \gamma  \underset{a'}{\max} Q_t(s_{t+1},a') - Q_t(s_t, a_t)]
$$

You are correct in saying that the neural network is just a function approximation for the q-value function.

In general, the approximation part is just a standard supervised learning problem. Your network uses (s,a) as input and the output is the q-value. As q-values are adjusted, you need to train these new samples to the network. Still, you will find some issues as you as using correlated samples and SGD will suffer.

If you are looking at the DQN paper, things are slightly different. In that case, what they are doing is putting samples in a vector (experience replay). To teach the network, they sample tuples from the vector, bootstrap using this information to obtain a new q-value that is taught to the network. When I say teaching, I mean adjusting the network parameters using stochastic gradient descent or your favourite optimisation approach. By not teaching the samples in the order that are being collected by the policy the decorrelate them and that helps in the training.

Lastly, in order to make a decision on state $ s $, you choose the action that provides the highest q-value:

$$
a^*(s)= \underset{a}{argmax} \space Q(s,a)
$$

If your Q-value function has been learnt completely and the environment is stationary, it is fine to be greedy at this point. However, while learning, you are expected to explore. There are several approaches being $\varepsilon$-greedy one of the easiest and most common ways.


Conflicting directions of weights gradients in gradient descent?

That is, maybe Err reduces when w1 goes towards ∂Err/∂w1 alone, but it might well be the case that Err actually increases when w1 is updated along with w2, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err.
Isn't this the case?

Not exactly. Using back propagation, the weight gradients may be calculated precisely (for the given training data). It doesn't matter that there are many of them updating at the same time. The gradients don't "conflict" as such, they are 100% compatible. But they are only valid locally to the current values of $w_i$
An update to the all the weights at once in the opposite direction to the gradient is guaranteed to reduce the error value for that training data, with an important caveat that it is only fully guaranteed to make an infinitesimal small improvement, when the step size is also infinitesimal.
If you make an update step that is too large (for some value of "too large" which varies a lot depending on context), then the curve may not remain consistent over the step and your error value could increase.
This problem is not directly related to updating multiple weights at once. It can occur even when you have a single weight. However, when there is a more complex function, with more weights all changing at once, there can be more places where the function does this.
In practice, infinitesimal updates would take too long to train, so you need to find a larger value - but not so large it causes problems - as a compromise.
In addition:

The training data you have will usually allow you at best to create a rough approximation to the "true" function you are searching for. So you don't really want to find an actual global minimum in the error function for your training data set, as it would overfit.

Typically, using mini-batches, the gradient is also only a rough approximation, so updates only go roughly in the right direction (this can sometimes be good as it can help escape from local minima and saddle points)


Modelling population changes with years on a network graph

The Sankey diagram would be nicely suited for your problem statement.

The flow width can be the number of refugees migrated, and the end nodes can be the destinations (from and to) of the refugees, and the flow width would be the magnitude of refugee migration.

If you want to model the graph on a geographical map, it can look something like this:

I am not sure whether it can be done in NodeXL or not.  But, a google search has returned me this link.


Different approaches of creating the test set

It gets a little complicated, I've attached links at the end of the answer to explain as well.

def test_set_check(identifier, test_ratio, hash=hashlib.md5):
    return bytearray(hash(np.int64(identifier)).digest())[-1] < 256 * test_ratio

The hash(np.int64(identifier)).digest() part returns a hash value of the identifier (which we cast as an int of 8 bytes with np.int64()).

The bytearray method stores the hash value into an array. The [-1] represents getting 1 byte/the last byte from the array/hash value. This byte will be a # between 0 and 255 (1 byte=11111111, or 255 in decimal).

Assuming our test_ratio is .20, the byte value should be less than or equal to 51 (256*.20=51.2). If the byte is less than or equal to 51, the row/instance will be added to the test set.

Rest is explained well in these links:

https://stackoverflow.com/questions/50646890/how-does-the-crc32-function-work-when-using-sampling-data

https://github.com/ageron/handson-ml/issues/71

https://docs.python.org/3/library/hashlib.html


Identify Bad Products from given parameters using neural networks

No.  Classification requires labelled data.  Without labelled data there is no way to solve this.  How would you do anything at all, if you don't know which of the products in the training set are good and which are bad?  There's no basis for making a decision of any sort.


Error while using decompose and stl functions in R

The error is likely due to the fact that your x is an irregular time series: there are temporal gaps, e.g. no data on 2015-01-03, 2015-01-04. You will have to implement your own version of decompose, e.g. following this approach:

https://stackoverflow.com/questions/12623027/how-to-analyse-irregular-time-series-in-r


Decision trees: leaf-wise (best-first) and level-wise tree traverse

If you grow the full tree, best-first (leaf-wise) and depth-first (level-wise) will result in the same tree. The difference is in the order in which the tree is expanded. Since we don't normally grow trees to their full depth, order matters: application of early stopping criteria and pruning methods can result in very different trees. Because leaf-wise chooses splits based on their contribution to the global loss and not just the loss along a particular branch, it often (not always) will learn lower-error trees "faster" than level-wise. I.e. for a small number of nodes, leaf-wise will probably out-perform level-wise. As you add more nodes, without stopping or pruning they will converge to the same performance because they will literally build the same tree eventually.
Reference:
Shi, H. (2007). Best-first Decision Tree Learning (Thesis, Master of Science (MSc)). The University of Waikato, Hamilton, New Zealand. Retrieved from https://hdl.handle.net/10289/2317

EDIT: Regarding your first question, both C4.5 and CART are depth-first examples, not best-first. Here's some relevant content from the reference above:

1.2.1 Standard decision trees
Standard algorithms such as C4.5 (Quinlan, 1993) and CART (Breiman et al., 1984) for the top-down
induction of decision trees expand nodes in depth-first order in each
step using the divide-and-conquer strategy. Normally, at each node of
a decision tree, testing only involves a single attribute and the
attribute value is compared to a constant. The basic idea of standard
decision trees is that, first, select an attribute to place at the
root node and make some branches for this attribute based on some
criteria (e.g. information or Gini index). Then, split training
instances into subsets, one for each branch extending from the root
node. The number of subsets is the same as the number of branches.
Then, this step is repeated for a chosen branch, using only those
instances that actually reach it. A fixed order is used to expand
nodes (normally, left to right). If at any time all instances at a
node have the same class label, which is known as a pure node,
splitting stops and the node is made into a terminal node. This
construction process continues until all nodes are pure. It is then
followed by a pruning process to reduce overfittings (see Section
1.3).
1.2.2 Best-first decision trees
Another possibility, which so far appears to only have been evaluated in the context of boosting
algorithms (Friedman et al., 2000), is to expand nodes in best-first
order instead of a fixed order. This method adds the ”best” split node
to the tree in each step. The ”best” node is the node that maximally
reduces impurity among all nodes available for splitting (i.e. not
labelled as terminal nodes). Although this results in the same
fully-grown tree as standard depth-first expansion, it enables us to
investigate new tree pruning methods that use cross-validation to
select the number of expansions. Both pre-pruning and post-pruning can
be performed in this way, which enables a fair comparison between them
(see Section 1.3).
Best-first decision trees are constructed in a divide-and-conquer
fashion similar to standard depth-first decision trees. The basic idea
of how a best-first tree is built is as follows. First, select an
attribute to place at the root node and make some branches for this
attribute based on some criteria. Then, split training instances into
subsets, one for each branch extending from the root node. In this
thesis only binary decision trees are considered and thus the number
of branches is exactly two. Then, this step is repeated for a chosen
branch, using only those instances that actually reach it. In each
step we choose the ”best” subset among all subsets that are available
for expansions. This constructing process continues until all nodes
are pure or a specific number of expansions is reached. Figure 1.1
shows the difference in split order between a hypothetical binary
best-first tree and a hypothetical binary depth-first tree. Note that
other orderings may be chosen for the best-first tree while the order
is always the same in the depth-first case.


No generic approach in data handling. Role of DS is understanding data. As you know sometimes NULL value means something, you have to check all aproaches to handling data and I find out which is most suitable to yours data.
Usually I marked missing value and resampling datasets. So I getting one dataset without missing values and another one with marked missing value.

Another techniques
https://www.kdnuggets.com/2017/06/7-techniques-handle-imbalanced-data.html

Best


Which optimization technique to use with noisy and slow black-box system?

Are you a petroleum engineer by any chance? This sounds very like the sort of optimization you have to perform in oilfield waterflood operations; you have to allocate your limited budget of injection water and/or lift gas to maximize oil production, with the additional complication that the optimal set point changes as the reservoir depletes and/or wells get drilled or abandoned.

The oilfield case has the useful feature that the overall objective function (field oil flowrate) is a simple sum of well flow rates. Each production well is generally only sensitive (in the short term!) to a small local subset of the inputs (gas lift rate in the production well, and water injection rate in near-offset water injection wells).

I would generalise the solution employed in the oilfield as follows:

Regularly perturb the inputs
Measure the affected local outputs
Maintain a simple linear or low-order polynomial model of the system and its parts
Use the model to optimize day to day
Keep good records to ensure that the model is up to date with reality
Monitor local and global model fidelity to decide what to perturb and recalibrate next

This works because the small perturbations that you keep on making (for model calibration purposes) are of short duration and local effect, so they don't take the overall system very far away from the optimum. The trade-off between keeping the system optimized, and "de-optimizing" it to calibrate the model, works out favorably. The system reacts on a timescale of days to months, and the measurements generally take a day or two to accomplish.

Try Googling waterflood optimization and gaslift optimization for some more detailed pointers, although a lot of the trade literature is paywalled.

Looking forward to learning what approaches have been taken in other fields with this kind of problem.


Using R to organize/rearrange CSV - group by multiple columns?

This very simple using the pivot_wider function from tidyr:
library(tidyr)

df <- data.frame(
  Prop_ID = c(1, 1, 1, 2, 2, 2),
  deforYear = c(2010, 2011, 2012, 2010, 2011, 2012),
  deforHA = c(15, 0, 10, 35, 45, 0)
)

df %>%
    pivot_wider(names_from=deforYear, values_from=deforHA, names_prefix="deforYear_")

# result
  Prop_ID deforYear_2010 deforYear_2011 deforYear_2012
1       1             15              0             10
2       2             35             45              0


how to make decision based on users reports

You don't need a prediction model for this. Maybe if you have had users' data. But without anything else, then you just need labeled data. Historical data that you know if it was a real accident or not.

When you have your labeled data, then you can follow a process like this, which is still heavily dependent on the kind of your data.

Start iterating on your labeled dataset and calculate the accuracy of a real accident's report for different combinations (5, 10, 15, 20, 25, 30 ... mins) and (1, 2, 3, 4, 5, 6, 7, etc users).

You will have a 2D matrix. I guess, acting fast on an accident is important in your case. Set an acceptable accuracy and choose the combination with the smallest interval, above that accuracy.


What is the best way to scale a numerical dataset

There is pretty much mess in terminology in your question :).
Data Regularization is used for model selection, it is not about data processing. Here it is described in more friendly manner.

What you mean is Feature Scaling. It can be done in several ways including Rescaling, the method you described. You may also use Standardization (normalization) and Scaling to unit length.

These answers may be helpful:

Normalization vs Scaling
Normalization vs Standardization


what is the difference between multilabel and multilabel-multiclass classification?

Multiclass classification means a classification task with more than two classes; e.g., classify a set of images of fruits which may be oranges, apples, or pears. Multiclass classification makes the assumption that each sample is assigned to one and only one label: a fruit can be either an apple or a pear but not both at the same time.

Multilabel classification assigns to each sample a set of target labels. This can be thought as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A text might be about any of religion, politics, finance or education at the same time or none of these.

Multioutput-multiclass (or multilabel-multiclass) classification means that a single estimator has to handle several joint classification tasks. This is both a generalization of the multi-label classification task, which only considers binary classification, as well as a generalization of the multi-class classification task. The output format is a 2d numpy array or sparse matrix.

As each object of your data can be classified in one to multiple labels at the same time, it is a simple multilabel classification problem

Taken from : https://scikit-learn.org/stable/modules/multiclass.html


Find which type of log using machine learning?

You first need to preprocess your text to convert it into features that can be consumed by a machine learning algorithm. Then you can try this algorithm (a classifier, in this case) with the known examples you have. Then you can use that trained model to recognise logs it hasn't seen before.

If you don't have a preference of language you'd like to use, python with the scikit-learn library would be a good start.

For your problem, work through this tutorial which is very similar to what you're trying to achieve:
http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html


How to predict class label from class probability given by predict_generator for testdata?

You just take the class with the maximum probability. This can be done using numpy argmax function.


Generating Synthetic Image to improve the performance of classifier

I think it could work well, if you try to reduce the false positive error. Generating images for negative examples shouldn't be to difficult. Generating images for positive examples on the other hand would require a good classifier first.


Performing anomalie detection on a battery volatge using LSTM-RNN

I will use the "date" and "time" columns to pre-process your data and to construct your neural net input.
RNN does not work well for very long-term dependancies... so, for example, creating a time series with all minutes in a month, won't probably work.
You must select:

How many samples your input data will have
What is your sampling period (minute, hour, day, week...)
What do you want to detect: If the input data contains an anomally or you might want to predict  if an anomally will occur in the future...

When you have all of this, and perhaps something else, defined you will have to use your "date" and "time" columns to create the dataset (the time-series)
Furthermore, I don't know if all batteries are checked under the same circunstances, for instance, are all of them new stock ones? If not, you can also use those columns to compute battery age or "working time" as an extra feature.


How can I detect patterns and/or keywords or phrases?

Depends on what you want to do and what you define as a "pattern". If you are interested in frequent terms, then tokenize and count the words. If you want to compare various forms of the same terms, I suggest you build two term matrices, one where you take the input as is and one where you take a version of the input that has been transformed to lower case.

BTW, a term frequency matrix is simply a matrix where the rows are your examples (I guess the columns in your database) and the columns are the discovered tokens (ie, the words)

For example, in the phrase 'the cat sat on the mat', the corresponding row vector of word counts would be:

the 2
cat 1
sat 1
on  1
mat 1

To get frequencies, you just divide the resulting vectors by the total count of words.

[EDIT] After reading the comments, I would recommend that you explore solutions for anomaly detection. Here is a good paper that is tackling a similar problem as yours. The authors encode documents as bag of words using term frequencies (see above) and build a classifier to flag  a whole document as an outlier.

In your case, it is a little trickier because you want to flag specific terms as good or bad, and you don't have a labelled dataset. So either find a labelled dataset (no need for a fully labelled dataset, semi-supervised solutions exist but at least a few labels would be nice) or explore unsupervised solutions to outlier detection. As for pattern flagging, your best bet is to use recurrent neural networks, which are designed to process sequential data such as yours. That being said, I doubt you will achieve that with PHP, so maybe consider something else. Tensorflow / PyTorch / Neo4J are very good alternative for building such models and many other exist


Simply clustering and then labeling all the texts in cluster will yield very noisy results (and you might learn your clustering similarity function rather that the "real" one). while there are some techniques that handle noisy labels (e.g- this NIPS paper: http://papers.nips.cc/paper/5073-learning-with-noisy-labels.pdf), there are several other options to consider:

labeling a small amount of data (be sure to have label data from each category, assuming you know the categories), training a classifier, classifying new examples, adding the most certain one to the training set, than retraining the model and so on. For example, see this article: http://link.springer.com/article/10.1023/A:1007692713085
Active Learning can be useful in this scenario. E.g: http://www.kamalnigam.com/papers/emactive-icml98.pdf
Semi-supervised learning techniques. E.g: http://mlg.eng.cam.ac.uk/zoubin/papers/zglactive.pdf

It is a common practice to combine more than one method. Also, LDA might be useful here.


What happens to the left over unpicked data in Random Forest

I suppose it is possible that not all samples are selected during training, depending on the parameters you specify (or that are available in the implementation).

Looking at Scikit-Learn's RandomForestClassifier documentation, we can see that there is a bootstrap argument that can be set to False to ensure all data points are used to fit each of the trees. Otherwise, say you pick some arguments to all be 1 (num_estimators, max_depth, min_samples), then not much data would be used at all! Looking through the source code, there doens't seem to be a check that all data was used.

Another classifier, ExtraTrees (Extremely Randomised Trees) is generally designed to used all samples to train each estimator. However the SciKit learn implementation does allow you to disable that and use random bootstrapping, as is default with the other random forest algorithms.

So to answer your question; it seems the unused samples are simply left out!


How do I turn images dataset into a numpy array?

You encounter a FileNotFoundError, which means that a file can not be found.
I suspect that you forgot the filename extension of your image. Should '/content/drive/My Drive/Colab Notebooks/Train/Haze' not be something like '/content/drive/My Drive/Colab Notebooks/Train/Haze.png' or '/content/drive/My Drive/Colab Notebooks/Train/Haze.jpg'


Predictive modeling on big data set that can't fit into memory

Whether you're using this for a hobby or for your job, I would recommend EMR on Amazon Web Services. The name is anachronistic (it's no longer just for map reduce), but EMR enables you to quickly stand up an Apache Spark cluster of several machines. Spark comes with machine learning libraries for building trees, and can scale to however many machines you need (to the point that you no longer face memory constraints). EMR is cheap because you're charged hourly and don't have to buy your own hardware. Apache Spark is great for your resume, and it's also a practical skill because it's generally the first tool most data scientists turn to when working with data sets that are too large for a single machine.


From What are the advantages of stacking multiple LSTMs? (I'll only update the answer there):

  From {1}:

    While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that
    deep RNNs work better than shallower ones on some tasks. In
    particular, Sutskever et al (2014) report that a 4-layers deep
    architecture was crucial in achieving good machine-translation
    performance in an encoder-decoder framework. Irsoy and Cardie (2014)
    also report improved results from moving from a one-layer BI-RNN to an
    architecture with several layers. Many other works report result using
    layered RNN architectures, but do not explicitly compare to 1-layer
    RNNs.

  References:

  {1} Goldberg, Yoav. "A Primer on Neural Network Models for Natural Language Processing." J. Artif. Intell. Res.(JAIR) 57 (2016): 345-420.
  https://scholar.google.com/scholar?cluster=3704132192758179278&hl=en&as_sdt=0,5
  ; http://u.cs.biu.ac.il/~yogo/nnlp.pdf


Attempt to reuse RNNCell with a different variable scope

It looks like you might be trying to use MultiRNNCell as a recurrent neural network. This isn't really its purpose; that is what dynamic_rnn, static_rnn, and raw_rnn are for. MultiRNNCell is intended for creating a feedforward, multilayer network which is the "cell" to be iterated to produce a recurrent neural network.

The old version of your code, if I understand it correctly, creates a network consisting of a sequence of LSTM layers, each with its own set of weights. If your goal is to make the weights shared between layers (so that your network is recurrent), you want to instantiate a dynamic_rnn and pass it a single instance of one of the RNNCell subclasses. That instance generates all of the layers of the network by having its __call__ method invoked repeatedly, once for each layer. The dynamic_rnn instance handles the scoping and variable reuse issues for you and connects the layers together.

You have at least three options for the cell to pass to the dynamic_rnn constructor:

1) A DropoutWrapper which was created using a BasicLSTMCell (essentially, the output of one call to attn_cell, but get rid of that reuse option). Do this option.

2) A MultiRNNCell consisting of an LSTM layer, followed by a dropout layer (unneeded here, but this is what you would do in some more general case).

3) Subclass rnn_cell and create your own type of layer.


Python Keras NN, handling n_samples of float outputs

I'm confused. As you have said, the targets are ratings. It's definitely a regression problem to me, instead of a classification problem.
There's several problems in your code:

For regression problem, we usually use linear as activation function of the last layer (sometimes relu, even sigmoid).
And also we use mse as metric(sometimes mae, msle, etc). categorical_crossentropy is used for classification problem, and sparse_categorical_crossentropy is used for sparse input classification problem. Ref
KerasClassifier is used for classification problem, use KerasRegressor instead.
metrics=['accuracy'] is used for classification problem, and it's meaningless in regression problem. Ref
Assuming df is a Pandas DataFrame, then df.values is naturally a ndarrary, there's no need to cast np.array.

Now answering your question:

As I've never used wrappers for scikit-learn API, I'm not very sure why you have integer output, my best guess is because of KerasClassifier and .predict(in scikit-learn APIs, .predict returns integer, basically the predicted class and .predict_proba returns float, indicating the probability of each class). Try to use .predict_proba, it would help. BTW, you should really use KerasRegressor.
As you have mentioned, there's so many classes to predict. In fact, it's a regression problem.
Could you please add the links of these 3 papers? Neural networks having only 1 neuron in output layer seem to be a regression NN to me. Regression NN usually uses Dense(1, activation='linear') as the output layer.

Here's my version of your code, it might work:
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

X = df[FEATURES].values  # no need to cast np.array
Y = df["MTPS"].values

# define baseline model
def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(10, input_dim=(len(FEATURES)), init='normal', activation='relu'))
    model.add(Dense(1, init='normal', activation='linear'))  # one neuron, linear activation
    # Compile model
    model.compile(loss='mse', optimizer='adam')  # mse loss
    return model
#build model
estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=2)  # KerasRegressor for regression problem

#cross validation
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=seed)
estimator.fit(X_train, Y_train)

#print class predictions
print estimator.predict(X_test)
print Y_test

References


How do I merge two data frames in Python Pandas?

Pandas has a built-in merge function. Please refer to the documentation

End-result should be something like this:
df_merged = pd.merge(df1, df2, left_on=['name_indexcolumn_df1_here'],
              right_on=['name_indexcolumn_df2_here'],
              how='inner')


How/What Correlation for Position Ranking?

Sounds like you are looking for the Pearson correlation coefficient between the two variables. You can compute that by using Scipy's pearsonr method.


Why is np.where not returning '1'? Only returns '0'

You are first checking if the item contains the string, but then in np.where you are checking if the values are equal (item == s), which is obviously different. In addition, you set the whole column equal to the value from np.where (and overwriting it after each row), which results in the whole column getting the value based on the final row of the dataframe.
To avoid looping over the rows (which is relatively quite slow) you can use pandas.Series.str.contains to check if a string contains a certain value like this:
df["orc_4"] = np.where(df["indictment_charges"].str.contains("2907.04"), 1, 0)


From the Orange3 Pypi documentation :

To start Orange GUI from the command line, assuming it was successfully installed, run:
orange-canvas # or python3 -m Orange.canvas


How to handle highly Imabalanced classification?

have you already tried using only very little of the -ve cases? So for example to train your model on 900 points total, 600/300? Then stratified sampling should still work fine. Then I'd evaluate your model based on it's ability to predict -ve cases and just monitor the performance it get's on the (in your case) gigantic test dataset that the model hasn't seen before.
Also, you could generate "synthetic" +ve cases by bootstrapping from the real data to see how that influences the model. Similar to what is described here: https://sci2s.ugr.es/keel/pdf/specific/articulo/dupret_science.pdf


after augmentation validation accuracy going down?

You can not compare validation accuracies if the validation datasets are fundamentally different.

What you were doing initially was to validate your model on a limited set of images. Augmentation is applied to make your dataset more representative of your deployment environment.

If in your deployment environment you expect images to be drawn from the same subspace as your augmented dataset, the validation accuracy of the first case will not be a true representative of your model's real world performance.


How is y=mx+b different from hθ(x)=θ0+θ1x?

1) Your hypothese $hθ(x)$ is clearly a linear model with $b \leftrightarrow θ_0$ and $m \leftrightarrow θ_1$ as you expected (don't be too hesitant and your colleges could have been able to confirm this).

2) Lets call $hθ(x_i)$ the prediction $p_i$. A cost function $C$ can be any function that when you try to minimize it holds a good solution.
One example would be the mean squared error (mse):  $C(p_i, y_i)=1/2 \cdot (p_i - y_i)^2$. I'm quite sure the mse is used to optimize the linear model, however other cost functions are possible.

3) I suppose this is an assignment? You clearly gave a very good suggestion to approximate your data. I suppose the goal of your task was to come to the solution of the linear model yourself. Try to understand how $b$ and $m$ is estimated in the linear model.

If you are in school, it is very useful to understand how people before you came to certain formula. It seems like you are reinventing the wheel, but the insights you gather with this can only help your education :).

Good luck.


Limits of Javascript on the implementation of AI algorithms

First of all, learning other programming languages is always a good thing.  There are many reasons for this:

Learning new languages makes it easier for you to learn other new languages in the future - and sooner or later in your career you will almost certainly want to or need to.
Learning new languages helps you understand the language you're already working in.  If I told you JavaScript is a dynamically-typed language, would that mean anything to you?  Probably not so much if you've never used a statically-typed language.  If I told you it has first-class functions, would that mean anything to you?  It's like telling someone "night is dark" when they've never seen daylight.
Learning new languages teaches you about ideas from other languages that may eventually make it into your language. ECMAScript 6 added "arrow functions" to JavaScript, these are also known as lambdas and would already be familiar to you if you've used C# (from version 3 onwards), Java (from version 8 onwards), Scala, Haskell, or many other languages.
Learning new languages teaches you different ways to think about code. And this can only make you a better coder.
Learning new languages increases your employment opportunities.
Learning new languages is fun!

In addition to these, and other benefits I've surely omitted, there's this which is directly relevant to your question about machine learning in particular.

Different languages have different libraries and different communities of users that use them for different purposes.  Most people doing machine learning aren't working primarily in JavaScript.  Some are working in Python, some are working in R, some are working with Spark using Scala or Python, etc.

If you already know JavaScript, then learning Python shouldn't be too hard for you, and it has some great machine learning libraries like scikit-learn, with lots of great resources online to help you learn.  So I'd recommend starting there.  But I also recommend that you eventually branch out and learn some languages that are further outside your comfort zone.


options available for visualizing a matrix type data frame in R

I'm expecting your data in this form (use melt to transform the data):

   type product value
1   A1      p1   158
2   A2      p1   343
3   A3      p1   247
4   B1      p1   400
5   B2      p1   398
6   B3      p1   408

You may easily (using lattice package) draw a barchart diagram.

Personally, I'd switch to heatmap only after this diagram is no more feasible due to large number of levels in the product or channel factors.

library(lattice)
barchart(product~ value | type,
data =  dfp,
scales=list(x=list(rot=90, cex= 1 ),y=list(cex=1)),par.strip.text=list(cex=1), origin = 0,
ylab="Product", xlab="Volume",
main= "Matrix Visualization")

Of course you may also swap the dimensions by changing the formula. Use the parameter layout to align the barchart based on the number of levels.

barchart(type~ value | product,
data =  dfp,
scales=list(x=list(rot=90, cex= 1 ),y=list(cex=1)),par.strip.text=list(cex=1), origin = 0,  layout= c(5,1),
ylab="Channel", xlab="Volume",
main= "Matrix Visualization")


Analyzing Sentiments of Financial News related to a Company

But how do I tell my model that it is actually a good news for Shell.

You would probably need a complex semantic analysis from which you could build a knowledge graph, from which you could extract the logical implications of one news for other entities not mentioned in the news.

  P.S. Most important, is there any labelled dataset or any other pre-trained architecture which I can use to calculate the sentiments of financial news?

You could ask on https://opendata.stackexchange.com/.
As far as I know, really good resources (data and algorithms) for financial applications are very expensive and usually kept secret by the biggest financial institutions.


how to determine hashing bit length for multiple categorical features?

How can I efficiently optimize the hashing trick on that feature vector?

Use different size of hash, and hash function and see which work best.
One cannot tell in advance.

  Should I enumerate hash bit length for each feature independently (greedy)?

No, this is NEVER a good practice. The hash should be common to all features.

  Should I assume no hashing is necessary for features with small alphabet (ni is small)?

You can always choose not to hash specific features. It implies you think these features are more important, and should avoid collision.


The point is to learn useful variations of data instead of just splitting by large categorial variable. Each new row after encoding becomes immediately related with the output, while original categorial variable may be related only in indirect, latent manner. Plus, the interactions between output and the original variable are represented too, by definition. Think of it as of adding this interaction explicitly to add more intuitive justification to this method.

So I think this method will be perfect not only for divide-and-conquer RF-style methods but also for plain LR as well.

  if the encoding for test set is always 0.5, then what is the point of this encoding?

That's okay, as soon as you know how this should be related to the output.
0.5 in this example is just a rough approximation of input, but given that you have learned model even this approximation is still meaningful, as it instructs model -- which path to output to choose. If your model of choice is RF then 0.5 will land at about same set of leaves as more granular distribution ( if you were able to know it )

Meanwhile much stronger approximations are possible, for instance, one can use KDE to estimate response variable distribution and then draw samples from it at test time. Adding uniform random noise is just a hint in this direction. Maybe you want to follow it?

Knowing only particular level of categorial variable is way less informative, than this 0.5, isn't it?


How to apply Normalisation using the MinMaxScaler() to all Columns, but Exclude the Categorical?

If you want to apply the result of fit_transform, you must assign to your columns.
columns = ['S_LENGTH', 'S_WIDTH', 'P_LENGTH', 'P_WIDTH']

min_max = preprocessing.MinMaxScaler()
df[columns] = min_max.fit_transform(df[columns])
df.head()

Output
   ID  S_LENGTH  S_WIDTH  P_LENGTH  P_WIDTH    SPECIES
0   1       0.0      0.0       1.0      0.0  VIRGINICA
1   2       1.0      1.0       0.0      1.0     SETOSA


Are you asking about rmse or r-squared?

The size of rmse depends on the general range of your response feature(predicting target). The lower the better.

As to r-squared, both 0.9999 and 0.9769 are very likely to indicate overfitting. If you have to choose between these two models, my suggestion is to evaluate them using generalization error. You can also do cross-validation if no out-of-sample data are available.


You should definitely use nested cross-validation for model selection and performance estimation. I have also found that the AUC of the precision-recall (PR) curve (compared to that of the ROC curve) to be a better, that is, more stable, estimator of the performance of my Random Forest classifiers when I have a highly unbalanced dataset; there is research on this topic, showing that the AUC of the PR curve is more informative than that of the ROC curve. You can use average_precision_score() in scikit-learn to use the PR AUC score. Along the lines of resampling the data, you could try approaches like EasyEnsenble and BalanceCascade; search for the papers titled "Exploratory Undersampling for
Class-Imbalance Learning" and "Learning from Imbalanced Data" for more information.


From where does BERT get the tokens it predicts?

There is a token vocabulary, that is, the set of all possible tokens that can be handled by BERT. You can find the vocabulary used by one of the variants of BERT (BERT-base-uncased) here.
You can see that it contains one token per line, with a total of 30522 tokens. The softmax is computed over them.
The token granularity in the BERT vocabulary is subwords. This means that each token does not represent a complete word, but just a piece of word. Before feeding text as input to BERT, it is needed to segment it into subwords according to the subword vocabulary mentioned before. Having a subword vocabulary instead of a word-level vocabulary is what makes it possible for BERT (and any other text generation subword model) to only need a "small" vocabulary to be able to represent any string (within the character set seen in the training data).


How to calculate Cumulative Sum with Groupby in Python?

There are multiple entries for each group so you need to aggregate the data twice, in other words, use groupby twice. Once to get the sum for each group and once to calculate the cumulative sum of these sums.

It can be done as follows:

df.groupby(['Category','scale']).sum().groupby('Category').cumsum()

Note that the cumsum should be applied on groups as partitioned by the Category column only to get the desired result.


How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

You can check out Featuretools, which an open source python framework for automated feature engineering. Specifically for you, it can generate aggregation features such as count for your dataset.

After generating the new feature matrix with the desired column, you can use the matrix as you normally would within an Sklearn pipeline.


Sort a data frame when column name is assigned in a variable

Your first attempt was more accurate. Since I do not have your data I will just use the iris data set,

 iris_ordered_on_Sepal_Length <- iris[order(iris$Sepal.Length),]

or if you use data.table,

iris_dt <- data.table(iris)
iris_dt[order(Sepal.Length)]

data.table will make renaming easier too,

setnames(input, c("colname_11, "colname_17", "colname_23"), validoutcomes[1 : 3])


Handling underflow in a Gaussian Naive Bayes classifier

The standard answer is to work in log space, and manipulate the log of probabilities instead of probabilities, for exactly this reason. This classifier involves products of probabilities which just become sums of log probabilities.

You allude to that already, but the problem you suggest isn't a problem. Internally you don't calculate a probability and then take the log again. It stays in log space. So for very small P, log P is a large negative number while P itself may underflow to 0.0. But you may never need to evaluate P internally.


Curse of Dimensionality : How many dimensions is too many dimensions?

As karthikeyan mg mention in his answer, you could use the explained variance score to get an idea of how many columns you can drop. Unfortunately, there isn't a magic number to know in advance. If you write code in Python, you should read this blog post in towardsdatascience.com

An Approach to Choosing the Number of Components in a Principal Component Analysis

After you rescale your data to 0-1, you can run this snippet of code and get a plot of variance loss for each components number you choose.

#Fitting the PCA algorithm with our Data
pca = PCA().fit(data_rescaled)
#Plotting the Cumulative Summation of the Explained Variance
plt.figure()
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number of Components')
plt.ylabel('Variance (%)') #for each component
plt.show()

This is the result from the blog post. As you can see, the variance starts to drop a lot after the 5 components. So, this number might be the one you could use.


How to convert a pytorch geometric graph to Networkx Multigraph?

Check out the provided utility method torch_geometric.to_networkx (link to docs).
I am not sure how this would handle multigraphs.
There's also a from_networkx.


How to use ensemble of models in FM or FFM?

Most ensembling methods are independent from the internals of individual models, e.g. something like majority voting, bagging or model stacking can be directly applicable to any model.

Note: this was originally a comment to the OP.


Pre-trained models for finding similar word n-grams

First off, there aren't, to my knowledge, models trained specifically to generate ngram embeddings. Although, it would be very easy to modify the word2vec algorithm to accommodate ngrams.
Now, what can you do?
You could compute the ngram embedding by summing up the individual word embeddings. Potentially, you could apply weights based on tfidf for instance, but not required. Once you have 1 embedding, simply find a nearest neighbor using cosine distance.
Another approach, though more computionally expensive would be to compute the Earth Mover's Distance (also called Wasserstein) between ngrams and find nearest neighbors this way.


As you point out, the problem is not on the clustering algorithm, but on the features. So the question comes to the particular data you might be dealing with.
As an example, say you want to cluster different kind of animals. It is in general much easier to tell an elephant from a horse apart. But if you want to distinguish between races of horses, it gets much harder. But the bottleneck lies (mostly) on the features.


Retain similarity distances when using an autoencoder for dimensionality reduction

No it is not possible to preserve relative distances when reducing dimensions for arbitrary data. This is not due to a property of auto-encoders compared to e.g. PCA or T-SNE. It is due to geometry.

You can see this relatively easily by considering a reduction of dimensions from 3 to 2, and examining a tetrahedron where all four corner points are 1 unit apart. There is no two-dimensional shape that can place four points mutually equidistant (unless perhaps you consider non-Euclidean spaces). It should be clear that relative distances near those corner vertices would also be affected, and thus this special shape demonstrates a general property of dimension reduction.


ZeroR as baseline for balanced / imbalanced datasets

You should only undersample the data that is used for training. Test data should represent the true distribution.

Then you have at least two possible baselines. One is ZeroR (your second point), which always predicts the majority class. Given a true distribution of (0) 10% and (1) 90%, this would give you a recall of 100% for (1) and 0% for (0) and 90% accuracy.

If rows represent the true labels and the columns the predictions, the confusion matrix looks something like this:

$\begin{bmatrix}90 & 0\\10 & 0\end{bmatrix}$

The other would be to flip a coin with probability 90% for class (1) (your first point but without undersampling), which gives you a recall of 90% for (1) and 10% for (0), and 82% accuracy, which is a little more balanced:

$\begin{bmatrix}81 & 9\\9 & 1\end{bmatrix}$

If your classes are strongly imbalanced then recall and accuracy have the same problems as you noted above. F1 might be a better choice, but the best metric depends on your application.


Just CTR is not important. Also, cost-per-click should be applied. Hence, to rank sites in an ads auction, using $CPC\times CTR$.


Representation options of strings (keywords/topics) in models

Since (word-based) one-hot encoding and real-valued vector representations are already mentioned in the question, I would only add the n-gram representation, especially the character-based n-gram representation.

For word-based n-gram representations you consider not individual words, but their ordered combinations in the text and use the one-hot encoding for the combinations. E.g. for n=2 you might end up with the bigrams ["John likes", "likes to", "to watch", "watch movies"] and each of them would be assigned to some dimension using a static index.

This also works with characters, so you can represent the word "encoding" e.g. with those 3-grams: ["enc", "nco", "cod", "odi", "din", "ing"]. The one-hot encodings of n-grams are typically added, so multiple occurances of the same n-gram are recognizable in the resulting Bag-of-n-grams representation. This kind of representation is especially useful for languages with rich morphology and/or compound words. In a one-hot representation each single word form would be encoded in its own dimension whereas a character n-gram approach helps preserve similarity between different forms. An example in the English language would be the similarity between "encode", "encoded" and "encoding" which would stay preserved this way. Similar techniques are also used by some word embedding algorithms which consider subword information like e.g. FastText.

Also, although it's not directly an encoding, but depending on your use case and language it might be worth looking at different preprocessing options like lemmatization and stemming where you reduce different word forms to their base form. This would also affect the choice of representation, e.g. the word-based one-hot encoding might make more sense if you choose to use these preprocessing techniques.