How Sklearn-crfsuit interpret text features

CRF models are not DL models, they use text features in the traditional way as categorical data (equivalent to one-hot-encoding). However they are different from other supervised ML methods because they exploit the sequential information of the text.
Typically the text sequence is represented word by word, and each word can have several additional "features" usually represented as columns. For example:
The     DET   the
dog     NOUN  dog
chases  VERB  chase
a       DET   a
cat     NOUN  cat

The real features used by the CRF are computed based on the rules defined before training the model. They can involve the position of the current word or any other position relative to the current one. The features are usually simple binary values, for examples "doc[n-1][2] == 'DET'" would represent the condition "the previous word is a determiner": "previous word" is indicated as n-1 in the sequence, column 2 indicates the POS column. The exact syntax to specify the rules may differ with different CRF implementations but the principle is the same.


Is there any book for modern optimization in Python?

You should be able to translate code written in one language -- even pseudo-code -- to another, so I see no reason to avoid books for R. If you want one specifically for python, there's Machine Learning in Action by Peter Harrington.

One of scikit-learn's core committers is a releasing a book in October: Introduction to Machine Learning with Python: A Guide for Data Scientists.


Recurrent Neural Net (LSTM) batch size and input

Batch sizes are different to series length. Batch size refers to the number of observations, and are typically defined when one is performing mini batch stochastic gradient descent.

If you are doing N mini batch SGD, you need to ensure the inputs to your network are a N x 168 tensor. Batch sizes are normally 32-128 observations.

WRT your second question, the approach is fine, it's just different. See which works better.


Choice of weights for the Laplacian Eigenmaps algorithm

Recall the definition he makes for the graph Laplacian earlier, $L = D -W $. Now consider the map in the RHS parentheses which I'll call $L^*$,
$$ L^*f(x_i) := f(x_i) - \alpha \sum_{x_j, ||x_i-x_j||<\epsilon}e^{-\frac{||x_i-x_j||^2}{4t}}f(x_j).$$
The suggested weight matrix definition is natural because it lets us write
$$ L^* := I - D^{-1}W. $$ Here's a reference to a related paper with some easy to read exposition. Hope this helps!


collaborative filtering using graph and machine learning

I can only speak about Graphs:

Advantages:

Using graphs, you can easily find products bought/rated by users that bought or liked an item, or users that have similar "taste" to another user. From my experience the traversal process is fast enough.
Closely matched products are easy to find, depending on how you model your graph: e.g. (userA)-[]->(basketA)-[with]->(productA)<-(basketB)-[]-(userB). Finding similar products by basket is computationally cheap here

Disadvantages:

Depending on how you rank your results, you can easily run into the trap of constantly recommending the same products. Users that bought A, bought B. So you suggest B. People buy/like B. Next time you run the query, B will keep coming on top.
Your results will rarely uncover new products. You'd have to find smart approaches around this, e.g. find other products, within this price range, with this category
If you have very large data sets, you'll have to use a distributed graph database that performs well (this is not as easy as it sounds, unless you're willing to pay large sums of money)

My little experience with ML for collaborative filtering, is that when your data grows large (50GB+), building a model takes a considerable amount of time (hours, days), and you're not likely to get good recommendations on new products. Having to update your model becomes a huge problem too. From my experience, I lean towards graphs for small use cases.

Note that for both cases, you never recommend newly added products that have not been bought or liked by someone. I say this because the goal of recommendations is to help users uncover new products. There is a blog post I wrote, where I talk about modelling graphs, and I discuss recommendations in some examples here, so you can skip to that section to get an idea on how to model this kind of problem.

I advise you to read up on Amazon's paper on their approach to collaborative filtering, which is pretty simple, in theory, and yields good results for them. I think you'd always want to implement a few different approaches combined, to tackle different parts of the problem.

Good luck.


I have worked on a similar project with JDs, we basically created a word2vec model for words in JD, the result were good as we had lots of JDs.
Basically, what word2vec does is convert a word to vectorial representation which signifies context.
You can check the documentation here: https://radimrehurek.com/gensim/

You may extract skills or other stuff from CVs, and do a semantic similarity based on w2v model. You may use a custom formulae for comparing similarity. Other things could be education, experience, similar projects etc


Implementing Complementary Naive Bayes in python?

Naive Bayes should be able to handle imbalanced datasets. Recall that the Bayes formula is

$$P(y \mid x) = \cfrac{P(x \mid y) \, P(y)}{P(x)} \propto P(x \mid y) \, P(y)$$

So $P(x \mid y) \, P(y)$ takes the prior $P(y)$ into account.

In your case maybe you overfit and need some smoothing? You can start with +1 smoothing and see if it gives any improvements. In python, when using numpy, I'd implement the smoothing this way:

table = # counts for each feature
PT = (table + 1) / (table + 1).sum(axis=1, keepdims=1)

Note that this is gives you Multinomial Naive Bayes - which applies only to categorical data.

I can also suggest the following link: http://www.itshared.org/2015/03/naive-bayes-on-apache-flink.html. It's about implementing Naive Bayes on Apache Flink. While it's Java, maybe it'll give you some theory you need to understand the algorithm better.


What you're looking for is covered in the keras LSTM examples:

from keras.layers import LSTM, Dense
import numpy as np

data_dim = 16
timesteps = 8
num_classes = 10

# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32))  # return a single vector of dimension 32
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

# Generate dummy training data
x_train = np.random.random((1000, timesteps, data_dim))
y_train = np.random.random((1000, num_classes))

# Generate dummy validation data
x_val = np.random.random((100, timesteps, data_dim))
y_val = np.random.random((100, num_classes))

model.fit(x_train, y_train,
          batch_size=64, epochs=5,
          validation_data=(x_val, y_val))

Source: https://keras.io/getting-started/sequential-model-guide/#examples

In your case, timesteps = 200, data_dim = 2.
It sounds like you have a continuous outcome and don't have classes, so you probably want to switch out the last layer to a Dense(1) with a linear activation or something, and switch out your loss and metrics appropriately (probably mean_squared_error and mae or something).

Also please note the architecture is purely an example, you may not need stacked LSTMs (or even LSTMs) for your example, you may want dropouts, you may want more dense layers, etc. This is just an example to get you started,


Is it possible to reuse my CNN trained model over a new dataset with different number of classes?

The accuracy will downgrade since the model was trained on a dataset with 6 classes but is being tested on 4 classes.
If you want a good accuracy for the dataset with 4 classes, first train it on the data of 4 classes and then test it on some test data with 4 classes.
Hope this is of help.


How would you describe cluster 2 from this output of a run of the EM program?

Welcome to the community!

So, in clustering if the number of clusters you indicate Apriori, is not right (what is right indeed?!! it means the intrinsic number of clusters inside data) then some clusters will be broken down to more clusters and what you see here happens (and yes, you need to tell the number of desired clusters to most of clustering algorithms (including GMM that you use) Apriori!)

In GMM clustering using EM algorithm, you can simply plot the histogram of the data and try to count the number of single Gaussians, which summing together, build up the histogram. that is the best choice of number of clusters.

Histogram (he called it PDF because PDF is simply histogram divided by the integration of area under histogram curve) below is taken from this kernel in the Kaggle competition from which your data comes.
It simply shows (by arrows) that data inhibits 2 clusters intrinsically so using 3 clusters miss-partitions one cluster to two. What happened in your result.

Try the same run with two clusters and you will most probably see the problem solved :)

Hope it helped. Good Luck!


Preliminary note
as far as I know, people may use the term feature extraction in slightly different ways:

referring to automatic methods used for dimensionality reduction which involve transforming the features (and not only selecting a subset of features).
referring to the general process of designing and engineering the features before training/testing a model on these features.

Please be aware that I'm not 100% sure that the terminology I use myself is standard. (but I don't think it matters for the question you're asking).
Answer
Used in the context you mention, the term certainly refers to the general process of feature engineering (2nd point above).
In traditional ML models the model directly uses the features exactly as they are provided to the training algorithm. For example, a decision tree classification model tries to find the best conditions in the features to discriminate the data. This means that the algorithm selects a single feature and directly uses its range of values to build a condition, something like this:
 if color == 'green' then go to subtree 1 else go to subtree 2

Thus the type of information that can be used by a model is strictly limited by the algorithm and the features which are provided to it: there's no way to consider any information which is not directly available as a feature. As a consequence the stage of feature engineering is crucial. For example in text classification there are a lot of choices to make: provide the words frequency or TFIDF as features? Provide all the word or remove stop words? Lemmatize or not? Add POS tags? And these are just some very standard questions, specific tasks may require a very detailed analysis. Importantly, it's not possible to just provide all the features one can think of because this would almost certainly cause the model to overfit due to the high number of features. Generally in this stage the goal is to give the model the best and most precise clues that will help it make correct predictions, and also avoid giving it too many irrelevant features to prevent overfitting.
That's the stage that DL doesn't need: thanks to a much more complex architecture and algorithmic process, a DL model can itself select and combine the features it needs to perform well.

I thought discover it is part of the task of the machine learning algorithm.

Sure, but in traditional ML in order for the model to discover the most important patterns for a specific task, the features have to be prepared in a way which maximizes the chances of the model to find these. Simile: if a teacher tells their students to revise page 63 of the textbook for the test, it's likely that the students will perform better than if they have no clue about the topic and need to revise the whole textbook.


(Feature selection) In which cases it is legitimate to remove features manually?

Is it reasonable to remove this feature before building an estimator? I've experimented with the case when including and excluding this feature, and the later case showed a slightly(very slightly...) better performance.

This, ultimately, is the best way to get an answer. Trust in cross-validation; if you're doing it robustly and it says that ditching the feature improves performance, then ditch the feature.

  I am wondering if there is any machine learning principle, such as we should avoid 'human learning' and just let 'machine learning' do. What would be the best way to deal with this sort of features? I am currently using Scikit-learn.

Nope, human selection of features plays a part too. Some algorithms are resistant to junk features and can effectively ignore them, but they always present the potential for overfitting. The best way to deal with them is what you've done; try the model both with and without and see which performs better.


Ramer–Douglas–Peucker algorithm sounds like what you are looking for!
https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm
This algorithm allows you to simplify a sampled path. Basically, given a maximum allowed error, it proceeds by evaluating if a sequence of samples can be represented well enough (wrt the error) with a straight line between the first and the last point: if all the points in-between them falls under the error, all of them can be removed, otherwise it tries again with a shorter sequence.
This means that, setting a reasonable error value, on a straight road it will go for strong downsampling, while on an hairpin turn it may even decide to not downsample at all.
It seems there is a ready-to-go implementation in python https://github.com/fhirschmann/rdp
Eventually, you may find this talk from EuroPython 2016, about "Handling GPS Data with Python" by F. Wilhelm (I learn about the algorithm from this)


For small data, I think pandas.read_csv is the way to go.

For "medium" data, I recommend dask.read_csv

And for big data, I recommend spark.read.csv


This entirely depends on your data!

Generally, the more augmentation, the more situations your model will be exposed to during training and therefore the more robust it will be when being tested on unseen data.

However, what if we for example were working on a model for self driving cars? Using the vertical_flip just doesn't make sense, because the car will (hopefully!) be er be driving along on its roof.

I would suggest starting with no augmentation and slowly adding one possibility at a time. For example, you record an accuracy of 80% with no augmentation. Then addfeaturewise_normalizatiom and featurewise_std_normalization giving you an accuracy of 85%. Then adding horizontal_flip gets you to 90%. Finally you try adding zca_whitening and that send you back down to 86%.

The reverse approach may also work well for you, starting with all augmentation parameters turned on and removing them one by one. In any case, it is completely dependent on your specific problem and your available data.  Keras' ImageDataGenerator has a long list of parameters, so having a think about what makes sense will save you a lot of time.


Keras: Misunderstanding what Keras does when invoking a Model, providing an Input?

Keras does a lot of stuff when you call a model. I guess that the most important is that it defines the trainable variables and the graph based on the inputs and outputs. For instance actually making any dense layers, and not just the representations.

For instance, if you did mod1=generator_model(input_low_resolution) and mod2=generator_model(input_low_resolution), then if you train mod1 then mod2 would not be affected as they have different parameters. So the .define_model just makes a "handle" to your model, which when called actually "builds" it.

This functionality is very useful when you want to mix-and-match multiple layer configurations, or working with more complex structures like for instance GANs or ADDA.

I would argue that your implementation is not exactly canonical and actually I'd recommend looking at the tensorflow.keras.Model example. Here they introduce a very nice, canonical way of making your own model.

Furthermore maybe Keras Model class API and Getting started with the Keras functional API can be of help.


How to apply Gradient Descent to a summ of function

I came up with the simple answer after I just rewrite this idea on paper:

$$\frac{\delta \sum \Big[f(x,y)\Big]}{\delta x}  = \sum \frac{\delta f(x,y)}{\delta x}$$

so it's just partial derivative for each Experimental Dot summed.
Algo works quite well.


Incremental Learning with sklearn: warm_start, partial_fit(), fit()

Answering my own question after some investigation:

warm_start=True and calling .fit() sequentially should not be used for incremental learning on new datasets with potential concept drift. It simply uses the previously fitted model's parameters to initialize a new fit, and will likely be overwritten if the new data is sufficiently different (i.e. signals are different). After a few mini-batches with large enough sample size (datasets in my case), the overall performance converges to exactly that of simply re-initializing the model. My guess is that this method should be used for the primary purpose of reducing training time when fitting the same dataset, or when there is no significant concept drift in new data.
partial_fit on the other hand has an effect and can be used for incremental learning (especially for datasets too large to fit into memory and feeding in mini-batches). However, in datasets with potential concept drift or high noise, it performs worse than disregarding past observations and simply fitting on each dataset without any incremental learning.
For SGDClassifier, calling partial_fit repeatedly makes a difference.


Need help in improving accuracy of text classification using Naive Bayes in nltk for movie reviews

Without reading your code (sorry!) I can suggest that you drop the verbs-only approach and use a word embedding to account for word similarity. Your feature dictionary is bound to find very few "hits" in the documents, unless you have a humongous dataset, and even then YMMV because you are just using verbs. Also: what's the justifications for using verbs? "I liked" and "I didn't like" both contain the verb "to like" and they mean opposite things. Same goes for hate (because you only have binary classification: no nuances) and so on.

A popular python implementation of word2vec is gensim, but you could use that of tensorflow or some other embedding like the (allegedly superior) conceptnet numberbatch. If you want to summarize whole documents into numbers you can try doc2vec (aka paragraph2vec, paper here), also available in gensim, tensorflow, etc.

You will need a big dataset. Once you have this up and running you can try other classifiers.


Deriving new continuous variable out of logistic regression coefficients

Deriving a probability value (the continuous variable you are talking about) from the logistic model is a perfectly sound thing to do. The probability value is actually the main output from the model.

Getting from the probability value to a decision rule (e.g. from default probability to credit granting decision) is an another step that will also need to incorporate a number of business decisions concerning risk appetite - i.e. how risky clients are you willing to approve not to miss a potentially large amount of business, what the interest is going to be for different risk scores etc. The tradeoff this refers to is the well known sensitivity-specificity tradeoff which is, apart from the confusion matrix you are using, probably best visualised by the ROC curve.

From the confusion matrix it is also apparent that you were training the model on a balanced sample (default rate around 50%). This is quite unusual in credit risk modeling, usually the default rate is well below that. If that is the case, you will probably need to calibrate the probabilities, for example by fitting a yet another logistic regression model on your probabilities as a single variable (this works best when the probabilities are normally distributed). The calibrated probabilities that this second model will predict will accurately reflect the actual real-world probabilities of default.


Predicting Player Position from pervious positions and LSTMs

As I tried to predict the position without normalizing... the error was in the data. After normalizing the positions everything worked as expected.


yes, in detecting abnormal cases.

In the k-means based outlier detection technique, the data are partitioned into k groups by assigning them to the closest cluster centers.

Once assigned we can compute the distance or dissimilarity between each object and its cluster center, and pick those with largest distances as outliers.

k-means clustering for Outlier detection


Fully Convolutional Variational Autoencoder

In my experience, when people say "fully" convolutional, even when doing something typical like ImageNet classification, they are still typically referring to a network with at least one final dense layer.

If I understand your question correctly, you're looking to create a VAE with some convolutional layers which has the same sized output as input, but you're confused how to upsample in the decoder such that you go from fewer latent dimensions to an output of the same size as your input. There are a few ways people typically handle this, two of which are deconvolution and sub-pixel (description of the difference, good deconv tutorial). Both allow you to use something like a convolution to take the output from $ d << n $ latent dimensions and upsample it into an $ n $ dimensional output.

Here's an example written in Keras that does this using deconvolutional layers.


Why does map reduce have a shuffle step?

The shuffle step occurs to guarantee that the results from mapper which have the same key (of course, they may or may not be from the same mapper) will be send to the same reducer. So, the reducer can further reduce the result set.

for example, your WordCount mappers emit

(hot, 1)
(cold, 1)
(hot, 1)
(rain, 1)
(cold, 1)

the shuffle step will ensure that

(hot, 1), (hot, 1)

get to the same reducer.
So, you get your final answer

(hot, 2)

and the same applied for (cold,1) and (rain,1)


Choose threshold to get 90% precision classifier - ML Binary Classification problem

threshold_90_precision = thresholds[np.argmax(precisions >= 0.9)]

Above snippet is not doing what you are expecting it to do.
 Try these changes
precisions[precisions < 0.9] = 1
threshold_90_precision = thresholds[np.argmin(precisions)]

Also, I am not sure whether you are calculating the accuracy properly since z_scores is decision function, not Class.
This is a working example using  method='predict_proba' for 40%, you may change to 90%
model.fit(x_train, y_train)

from sklearn.model_selection import cross_val_predict
y_train_pred = cross_val_predict(model, x_train, y_train, cv=3)

z_scores = cross_val_predict(model, x_train, y_train, method='predict_proba')

from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds = precision_recall_curve(y_train_pred, z_scores[:,0])

import numpy as np
precisions[precisions < 0.4] = 1
threshold_90_precision = thresholds[np.argmin(precisions)]

y_train_pred_90percent_precision = z_scores[:,0] >= threshold_90_precision
from sklearn.metrics import precision_score
print(precision_score(y_train, y_train_pred_90percent_precision))


Result Dithering? Why randomly shuffle results?

This is not an answer but rather a comment. I would need 50 reputation to comment.

I think in its nature, a recommendation problem is a greedy algorithm to a search/optimization problem. However, the process is biased and local. The users see and react to what they can see, and the algorithm will only reinforce it. Dithering is a way to break up the bias and make the search more global.

What I have been doing is a bit different than dithering, but similar idea. Instead of ranking, I converted the matching score to log linear probability, and "sort" the item/person by probability. An example is, P(a)=0.2, P(b)=0.8, then b has 80% of chance on top, while a has 20% of chance on top.

Like to hear how others hear about it too.


Can pooling ever increase accuracy in convolutional neural networks?

I think this is a very interesting question. It tells me that you have a deep understanding of the basic concepts in machine learning, overfitting - underfitting, ... etc.

Let us start with making sure that we all agree that max pooling does not add any additional parameters to the network, max pooling is a well defined operation and there is no need to do any training to max pooling layers.

The main challenge in answering your question is that it is really difficult to address the effect of having max pooling as part of the network without considering other factors: the curse of dimensionality (the size of the output of the feature extraction part), the size of the network (especially the fully connected layers) and the overfitting issue.

In one of the experiment I did, I tried to answer a similar question. But first, I want to isolate the effect of using max pooling from changing the size of the network. But how to do this: I replaced the max pooling function with a custom version: instead of pooling 1-value from (4 x 4), I used a custom function that pooling (4 x 4) from (4 x 4) where all the values in the output are equal the max value in the input (4 x 4). I increased also the size of the convolution kernels  in later layers (to make sure that I am looking at a larger receptive field in later layers)

You can see that there will be many redundant information. I was able to isolate the advantage of changing representation from sparse to rich representations. Which can be beneficial (think about max pooling in sparse coding to understand how this works).

The answer from my own perspective:

generating rich representations can improve the classification accuracy.
if you do not have max pooling, the fully connected layers can be really huge.
if you do not have enough data, you will have overfitting problem.
in my case, the database I have contains ~ 12M images, since data is the best regularization technique, it was difficult for me to understand if there is any overfitting problems (which I believe it will be the case if you do not have enough images).

I totally understand that from information theory point of view, we are loosing information in max pooling, but it looks like max pooling keeps the most important information that can be used to build a strong multi-layer full connected network at the end.

please do not take my answer as an ultimate truth, I just wanted to share my concerns and thoughts about this based on one of the experiments I did before


Lowering learning rate makes my accuracy on the validation set go down

Optimizer functions in ML algorithms update themselves with learning rate to converge local minimums. When the step size (here learning rate = eta) gets smaller the function may not converge since there are not enough steps with this small learning rate (step size).

You should increase your learning rate or number of steps while keeping the learning rate constant to deal with the problem.

You may want to look at gradient descent: https://en.wikipedia.org/wiki/Gradient_descent

Also following image explains very well how to tune learning rate. (source: https://www.jeremyjordan.me/nn-learning-rate/)


What's a classifier capable of predicting a variable number of classes

This is an interesting problem. What is your xtrain? I guess it boils down to a multi label problem. In a simple setup, you would train on the presence of a certain label (present or not). So for n tags/labels, you would train n+1 (1 is no label) models.

I‘m not really in to multilabel problems, but I guess this is the way to go.


In a residuals vs fitted plot, how do I interpret a homoscedastic variance that is not randomly distributed above/below the line?

This suggests that there's a nonlinearity in your data that isn't reflected in your fitted model.  You haven't told us anything about your predictor variables, but for example if you've fit a model of the form $y=ax+b$, then perhaps you could try adding a quadratic term $y=ax^{2}+bx+c$.


How to implement gradient descent for a tanh() activation function for a single layer perceptron?

How can I circumvent this problem?

TLDR : Normalize your input data.

Why?

Notice how tanh actually works on the input data:

>>> np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000]))
array([-1. , -1., -1. , -0.76159416,  0.,0.76159416,  1. ,1.   , 1])

If your data input to tanh has a magnitude of 10 or more, tanh produces an output of 1. That means it treats 10, 50, 100, 1000 the same. We don't want that.

This would explain

  the input vector with the weights turns out to be too large, which makes tanh(x)=1tanh(x)=1 and 1−o2=01−o2=0, so I can't learn.

Instead, we normalize the data (by dividing the input data by 1000 ( there are other ways of doing this)):

>>> np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000]) / 1000)
array([-0.76159416, -0.09966799, -0.00999967, -0.001, 0. ,0.001,  0.00999967,  0.09966799,  0.76159416])

Now, tanh treats all of them differently.

Note
You are doing a classification task. Using mean squared error as a Cost Function won't yield the best results here. You should instead use Cross Entropy Loss with One Hot Vectors and softmax.

As to why should be a whole different answer.
Here is a link for the same.


What is the difference between observation and variable?

In your case, 'n' is the number of observations, and 'm' is the number of variables.  Think of a table in a database - each row is a 'record', and each record has various properties.  A record corresponds to an observation.  Each property is a variable or property of that observation.

In some contexts people make a distinction between data (i.e. observations, things observed) and variables (parameters to learn - things that vary in a mathematical sense).  But in the context of the link you sent, a variable is a feature.


How to use scikit metrics for a statsmodel or vice versa?

Interpreting the code (since the error received is not mentioned), it looks like the code is passing in the X matrix and y-pred to the metrics. According to the documentation, the metrics want the y-true and y-pred. This would lead to an error mentioning incorrect dimensions.

I have used statsmodels and call scikit metrics. Many of the scikit examples, like the above documentation, show arrays being passed in, not from a specific model.

If that is not it, please post the error received.


How can I predict student enrolment in September based on independent data available earlier in the year?

I would try out regression in Python's scikit-learn library to predict the September headcount given all those other variables you have. Here is a basic example using a linear model. Once you have that working, you could try a more sophisticated algorithm.

http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html


Feature name extraction directly from dataset

If you are reading in a csv, try;

data = pd.read_csv('filepath', header=None)

Then you can do what you were doing to read the features from the first line


The short answer is: don't do that; anything you come up with on your own is likely to leak information in subtle ways.

The longer answer is: talk to your data protection authority.  Only use a method if they consider it acceptable.  (Hint: it's very likely that the answer you will get back is "no, don't do that".)

The technical details are: there is an entire body of work on how to anonymize or de-identify data, so it can be revealed in a way that can't be linked to the people whom the data is about.  However, there's a long history of attempts to do this that failed; where it was actually possible to re-identify the people, violating their privacy.  See the following research paper:

Myths and Fallacies of "Personally Identifiable Information".  Arvind Narayanan and Vitaly Shmatikov. Communications of the ACM, 53:6, June 2010.

If you wanted to consider a scheme to do what you want, you'd need to tell us a lot about the nature of the data, what it reflects, what the privacy interests are, how the data is distributed, what other external sources might be available that might enable re-identification, and many other details.  Since you haven't provided any of that information, your specific question isn't answerable in its current form.

But ultimately, I hope I've convinced that what you're trying to do is a bad idea.  Don't endanger people's data by going "cowboy" with some homebrew scheme to obfuscate the data; instead, talk to your data protection authority about what their requirements are.


How to construct confidence bound for Time Series Prediction?

There is a technique called "Quantile Regression" which allows you to calculate a percentile of your $Y$ variable, the way a quantile regression works is by giving more penalty to the data far to the desired percentile.

Quantile Regression

As you used deep learning (I don't know in which software), maybe you could tailor a penalty function which does the same as quantile regression does or maybe the code you are using has it implemented.


Since you want to show so much data, I think that your best choice is going interactive. Check out this demo, it is close to what you want but not quite.

It is very difficult to show a lot of data in a single diagram, together with the finest details and the bird-eyes view. But you can let the user interact and look for the details. To show counts, one option is to use color-coding. Take a look at this image (code here): .

Here rgb channels have been used to encode (the logarithm of) the number of active events (red), events starting (green) and events ending (blue) for windows of different size. The X axis is time, and the Y axis represents window size, that is, duration. Thus, a point with coordinates (10, 4) represents the interval of time that goes from 10 to 14.

To make a lot of data more detailed, it could be a good idea to make the diagram zoomable (like in the demo before), and to give the user the possibility of visualizing just one channel/magnitude.


Matlab simulation through FIS and Markov Process

I don't really get why would you mix Fuzziness and Probabilities. HMMs already can give you probabilities without the need of adding Fuzzy systems into the mix.

I would just do a random walk with probabilities of transitions defined by the state of the lights.


Why is XGBClassifier in Python outputting different feature importance values with the same data across different repetitions?

Look at the 'max_features' parameter in the GradientBoosting Classifier in scikit learn, here: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier
That means that the algorithm is fitting different trees to try and account for the residual information after each sequence of trees, and that the features for each tree can be randomly permuted based on this parameter's selection.
Scikit learn does say that this can be made deterministic by setting random_state...


Detecting abundance of a certain periodic pattern in a time series?

Time warping is used to align different time series, is more useful for data preprocessing rather than for what you want.
"What I really want to do is to calculate a metric telling me to what extent there is a consistent '1 hour per day, once per week' pattern"
I don't know if you have a specific reason to use a 'metric', if what you need if to find periodical patterns across people what you could do is apply different moving averages to every time series to extract different periodical patterns, then you could evaluate the cross-correlation between different people time series to evaluate which patterns are subjective and which can be instead considered true across people.


Why not using a Multi-Model DBMS like OrientDB (I'm the author) or any others? In this way, with just one system you can store documents and graphs together.

It would be:
- much faster (because you avoid the double call to 2 systems where most of the times the 2nd call needs the output of the 1st call, so you pay a lot of latency)
- easier to manage: one DBMS, one set of skills, in the case of commercial support, only one vendor

Following your use case, with a Multi-Model DBMS like OrientDB, you could start with a FULL-TEXT query (the underlying technology is Lucene, the same of Elastic Search) and then navigate the resulting graph.

Look at the Social Travel Agency example about how you can mix multiple models at the same time: https://orientdb.com/docs/3.0.x/gettingstarted/demodb/DemoDB-DataModel.html.


Selecting most relevant word from lists of candidate words

There are many ways you could approach this problem

Word embeddings

If you have word embeddings at hand, you can look at the distance between the tags and the bucket and pick the one with the smallest distance.

Frequentist approach

You could simply look at the frequency of a bucket/tag pair and choose this. Likely not the best model, but might already go a long way.

Recommender system

Given a bucket, your goal is to recommend the best tag. You can use collaborative filtering or neural approaches to train a recommender. I feel this could work well especially if the data is sparse (i.e. lots of different tags, lots of buckets).

The caveat I would see with this approach is that you would technically always compare all tags, which only works if tag A is always better than tag B regardless of which tags are proposed to the user.

Ranking problem

You could look at it as a ranking problem, I recommend reading this blog to have a better idea of how you can train such model.

Classification problem

This becomes a classification problem if you turn your problem into the following: given a bucket, and two tags (A & B), return 0 if tag A is preferred, 1 if tag B is preferred. You can create your training data as every combination of two tags from your data, times 2 (swap A and B).

The caveat is that given N tags, you might need to do a round-robin or tournament approach to know which tag is the winner, due to the pairwise nature.

Recurrent/Convolutional network

If you want to implicitly deal with the variable-length nature of the problem, you could pass your tags as a sequence. Since your tags have no particular order, this creates a different input for each permutation of the tags. During training, this provides more data points, and during inference, this could be used to create an ensemble (i.e. predict a tag for each permutation and do majority voting).

If you believe that it matters in which order the tags are presented to the user, then deal with the sequence in the order it is in your data.

Your LSTM/CNN would essentially learn to output a single score for each item, such that the item with the highest score is the desired one.


Error with negative bins while melting dataframe

I am sorry, is this the kind of ggplot you want to obtain ?

If yes this is the code behind it:

library(reshape)
library(ggplot2)

d <- data.frame(breaks = c(-20, -15, -10, -5,  0, 5, 10, 15, 20, 25, 30),
                counts.x  = c(1, 0, 0 ,4 , 13, 18, 9, 2, 1, NA, NA),
                counts.y = c(1, 1, 5, 18, 27, 25, 12, 1, 7 , 0, 1))

d <- melt(d, id = "breaks")

ggplot(d, aes(x = breaks, y = value)) + geom_point(aes(colour = variable)) +
       labs(title = "Bins vs. Counts", x = "Bins", y = "Counts") +
       theme(plot.title = element_text(face = "bold"))


Comparing one small dataset with a big dataset for similar records

A way to speed up this process is to preprocess the large dataset, the goal being to store the documents from A in a way which avoids a lot of useless comparisons.

Store each document from A in an inverted index $m$, so that for any word $w$ $m[w]$ is the list of all documents in A which contain word $w$ (note that a document can appear several times in this data structure).
When comparing a new query against $A$, instead of iterating through all documents in $A$ just compare against the subsets which have at least one word in common, i.e. $m[w]$ for every word $w$ in the query.

Couple remarks:

Normally stop words would be excluded from the keys since they appear everywhere and they are not relevant for matching.
The key doesn't have to be a single word, it could also be an n-gram (or several n-grams) or even several words in case it fits in memory.

This kind of problem is frequent in the task of record linkage.


Where are WEKA installed packages stored

I don't know how WEKA stores the extra packages on other installations, but for my Windows 10 with WEKA 3.8 it was here:

WEKA stores the extra packages in a direcory called packages. This package was in my user directory and under the wekafiles folder.

Example: "C:\Users\myUser\wekafiles\packages....."

Hope this helps someone else.


Technical term to describe/differentiate the structure of an instance by the number of samples per instance ('tuple' vs 'time series')

We have a Dataset
 Dataset has Instances
Instances can be a Vector Or a Sequence.

If it is a vector, It has Features
Examples - Image, Tabular data
If it is a Sequence, It has Time-steps**
Each Time-step can have Features i.e. Uni/Multi-Variate
Examples - Video, Audio, Time-series data

**When we use text-data as a sequence, we call the words as "Tokens"


Making an ensemble model for high F1 score

In general this would mean that P predicts only a small number of instances higher than 0.9 whereas R predicts most instances higher than 0.9. Therefore a weighted average of the two scores will fall somewhere in the middle, likely resulting in a moderate precision and moderate recall.
This can give significantly better results but only if the two classifiers are complementary, i.e. they predict instances in a way different enough from each other. Otherwise it's equivalent to tuning the threshold on a single classifier.


Analyzing mobile usage. What kind of approach should I apply?

There are two straight forward (vanilla) ways without going for any fancy featurization:

Clustering:

Run a clustering algorithm. Something like k-means should work well with this kind of a dataset. While doing this, I would not feed the day_of_week information into the clustering algorithm.

I would suggest running k-means (after normalizing each of the columns). Choose a small number of clusters that is easy to investigate (or you could use the number of clusters that maximizes the BIC).

Investigate the clusters to understand membership by day_of_week in each of these clusters.

Multi-class Classification:

Treat the day_of_week as the response that you would like to predict. Build a decision tree of a fixed depth to predict the day_of_week given the columns. By examining this tree, you can easily tell, which decisions led to a set of leaves being labeled Sunday vs the set of decisions that led to a set of leaves being labeled Monday. These decisions will also help you understand the similarities between different days.


If you want to predict, you need an underlying model of the behavior. Then you can perform least-squares fitting to determine the model parameters. Linear regression just assumes that you have a linear relationship and predicts from that relationship - there's nothing magic about it, except that the model is simple and often effective. Polynomial regression gives bad behavior when extrapolating, so I wouldn't advise that.

Building a model function is usually a trial-and-error process (unless you have some prior knowledge about your system, which I guess you don't). I'd suggest looking at the locus of (x, y) points where z is maximized. It looks like they have an exponential decay relationship. A model like y = exp(-x/x0) might fit the data rather closely. Once you've determined the value of x0 from the data, you can rebuild a model for z.


How to upload a saved file from google colab to a site such as kaggle or github?

In your directory /miryam/myproject:

git init .
git add myfile
git commit -m "First commit"
git remote add origin remote https://github.com/yours_project
git push origin master


multi channel feature classification using deep feed forward neural network on tensorflow

You have not specified which deep net you are using. You can use any data set which is of the right format in TensorFlow. If you are using convolutional networks, You may need data which locality is important. In this case:

MNIST data set
ImageNet data set

may help you. If you are doing other tasks I highly recommend you taking a look at this question which is so much popular and has great answers.

If I want to quote, I quote the following answer. Network Repository an interactive data repository with over 600+ networks in 20+ collections; from large-scale social networks, web graphs, biological networks, communication and technological networks my help you.

  Data Sets

  Academic Torrents
  Quora
  hadoopilluminated.com
  data.gov
  Quandl
  freebase.com
  usgovxml.com
  enigma.com
  datahub.io
  aws.amazon.com/datasets
  databib.org
  datacite.org
  quandl.com
  figshare.com
  GeoLite Legacy Downloadable Databases
  Quora's Big Datasets Answer
  Public Big Data Sets
  Houston Data Portal
  Kaggle Data Sources
  A Deep Catalog of Human Genetic Variation
  A community-curated database of well-known people, places, and things
  Google Public Data
  World Bank Data
  NYC Taxi data
  Open Data Philly Connecting people with data for Philadelphia
  Network Repository An interactive data repository with over 600+ networks  in 20+ collections; from large-scale social networks, web graphs, biological networks, communication and technological networks, etc.
  A list of useful sources A blog post includes many data set databases

Data Sets From awesome-datascience


VC dimension for Gaussian Process Regression

The expressiveness of the gaussian process grows with a number of training points. So, the vapnik-chervonenkis dimension in fact is infinite (pretty much the same way it's infinite for k nearest neighbors) and unfortunately your rule of thumb is not applicable here.
You should probably rely on train/validation split to estimate the generalization. From my experience, GP generalizes much better than neural nets, but the exact required train size depends on data distribution complexity itself.


install_tensorflow() keeps overwriting virtualenv, making it impossible to install tensorflow and keras

Please check that which pip you are using to install keras. It is possible that your pip may be pip3( will find libraries for python 3.x), while pip2 aka pip (will find libraries for python 2.x).

Keras is supported in python 2.7:

Tensorflow also, they promote use of python 3.x, you can checkout this tow download 2.7 compatible whl. tensorflow 2.7

You can check the pip version by executing below cammand from terminal.

pip --version

To find location of python/ pip use:

which pip
which python

I would personally suggest use python 3.6 and above. As for python 2.x and python 3.4, support is going to be end soon.


why does making the target variable normally distributed helps?

In some cases, it may actually help getting better results (depending on the model type), but it is also likely that the improvement comes from the fact that the performance metric is computed differently. For instance, a skewed distribution will lead to high MSE values due to cases located on the other side of the distribution, while the MSE is limited if the data is transformed to a normal distribution. So when comparing the cases, make sure you evaluate the performance on the back-transformed target.

Cases where the model will actually perform better with a normally distributed target include, among others, Gaussian process regression, because of the underlying assumption of a Gaussian random variable. There should be quite a few other model types which somehow have similar assumptions, and thus perform better with transformed data.


Handling Concat and Shift Feature in Pandas for Data Science

As was pointed out by @Stephan Rauch in his comment, the names of the columns are stored in dataframe.columns - the OP had a typo.

Below is a working example with dummy data, getting the same output as the user - using instead a little loop to compute the shifted values.

from pandas import DataFrame

prices = dict(
    col1=[0, 1, 2, 3, 4, 5, 6],
    col2=[2, 3, 4, 5, 6, 7, 8],
    col3=[5, 6, 7, 8, 9, 10, 11],
    col4=[12, 13, 14, 15, 16, 17, 18])

dataframe = DataFrame.from_dict(prices)
print(dataframe)
new_col_names = ['t-2', 't-1', 't', 't+1']
dataframe.columns = new_col_names
print(dataframe)

# Number of columns we have
N = len(dataframe.columns)

for n, col in enumerate(dataframe.columns):
    shift_by = N - n - 1  # don't shift the final column
    dataframe[col] = dataframe[col].shift(periods=shift_by, axis=0)

print(dataframe)

# If desired, remove the new NaNs that appear in the first
final_dataframe = dataframe.drop(labels=dataframe.index[:N - 1], axis='index')
print(final_dataframe)


To me it looks like not tensorflow task at all. At least not at the first place.

"Normalize headlines" task (lemmatization). Spacy does nice job here and it has great documentation. Here is an example, have a look at the "lemma" property.
Use the food2vec as a database of topic names.
Parse sentence via spacy and find the phrase in the food2vec. Parsing should be done not word-by-word, but by phrase: first look up 3-words phrase in the dictionary; if not found - 2-word dictionary; than 1 word.

This should be enough to solve your task.


Custom loss function with both min(y, p) and max(y,p)

A loss function calculates the error over all the data presented to it. For neural networks, that is an average over the mini-batch. Your code might only handle scalars (hence working when batch=1), it should handle vectors.
The K looks like you are using tf.Keras. The documentation gives examples of custom loss functions. It helps to subclasses tf.keras.losses.Loss which expects a call() method that contains the logic for loss calculation using y_true, y_pred.
class CustomLoss(Loss):

  def call(self, y_true, y_pred):
    pass


libraries for multiple machine NN training?

TensorFlow and PyTorch both support distributed training.


If you are new to forecasting, I would recommend starting with a very simple model for your sales, e.g., Exponential Smoothing, which will use only the sales data and no external information, but will capture trend and seasonality. Here is an extremely good free online textbook.

Once you are comfortable with your simple model, you can look at more complex ones that do include external causal variables. For instance, you could run ARIMAX models (the X stands for eXternal or eXplanatory variables). For this, look at chapter 8 in the online textbook and check the auto.arima() function in the forecast package for R, where you can specify external variables using the xreg parameter.

Alternatively, you could look at other methods like Neural Networks, or possibly even Vector Autoregression. To learn more, you could look into standard econometrics textbooks.

Whatever you do, keep two things in mind:

If you include explanatory variables in your forecast, you will need to forecast those variables themselves. In an assignment-type situation, you may be able to work with actual future macroeconomic variables, but of course in a "real" forecasting situation, you don't know next month's sugar prices yet. Forecasting these to feed the sugar price forecast into your sales forecast model adds an additional bit of uncertainty.
You may just find out that your fancy model with lots of additional explanatory variables does not do a much better job at forecasting than your original simple model. This would not at all be surprising. Simple methods very often outperform complex ones in forecasting.


On the offline evalution of recommender system

Sequential data (i.e. data that can be understood in sequences, such as time series) require apposite train-test splitting. When dealing with sequences, the split must be temporal as well.

It's the same principle that that must be followed when training language models, to make the most common example. This is well known in the literature.

I suggest you to find a timestep threshold in your data, and use that to split between train and test set.


From sklearn:

  The features are always randomly permuted at each split. Therefore, the best found split may vary, even with the same training data and max_features=n_features, if the improvement of the criterion is identical for several splits enumerated during the search of the best split. To obtain a deterministic behaviour during fitting, random_state has to be fixed.

If you manually set the random_state variable when you create your tree object, you'll find that it does become deterministic.

In simpler terms, the data you are feeding it is a little small, and there are several splits that have the same information gain, so the split that is chosen is subject to random factors.


Does Pooling remove spatial information of image in CNN?

Your question text is not very clear but I try to give you what you need. Max-pooling layers give the CNNs spatial invariance ability through the layers due to the fact that they check the existence of something or not. If you stack them through multiple layers, your network will have a rough ability to be spatial invariance but this is not that much. Moreover, professor Hinton has stated somewhere that using pooling layers is a mistake and it's a disaster that they work; I didn't quote the exact words. If you want a network to be spatial invariance you should use spatial transformers which are differentiable modules and can be used in CNNs without any supervision. Take a look at here.

Finally, to answer your question, pooling layers somehow remove spatial information that is why capsnets are introduced.

About the fully connected layers, There are two main points. First the inputs, second, the outputs. In convolutional layers, the inputs of each layer is limited to the region which its is employed and for that input there is a single output. Consequently for a specific region, there is a single output which is responsible for that point. In MLPs, all the inputs go to a single neuron and for all the inputs there is just a single value. I guess this is why they don't keep the spatial information and are just used for classification tasks. Actually, they just try to classify the extracted features by the CNNs.


Create new performance indicators (error metrics)

In research, new evaluation measures are proposed all the time but very often for specific use cases: either because it's a new task for which there is no established evaluation standard, or because the new measure provably solves some issues/limitations with the previous evaluation standard.
The purpose of an evaluation measure (or performance indicator),  is to represent the quality of the predictions by any system for a given task. This implies that:

An evaluation measure is selected specifically for a task. Some measures are very common because many tasks can be framed in a standard way, for example as a classification problem.
The central problem of evaluation is how to define "quality" in a formal way. A performance indicator is always a simplified representation of "quality", but it must correlate at least approximately with what a human expert of the task would call "quality".

Naturally the first step for developing a new evaluation measure is to select the target task (or group of tasks). Then one has to demonstrate that the new evaluation measure brings some significant advantage in terms of measuring the quality of this task. Of course this is the hard part, and it always depends specifically on the task.
For example, there have been numerous evaluation measures proposed for Machine Translation (there's even a regular Shared Task devoted to the topic). The way a MT evaluation measure is itself evaluated is against human judgements: if the automatic measure correlates strongly with human judgements about the quality of translation, then it's a good measure.


Machine Learning for hedging/ portfolio optimization?

That is a rather broad question, and there is tons of literature about quantitative analysis and stock market prediction using machine learning.

The most classical example of predicting the stock market is employing neural networks; you can use whatever feature you think might be relevant for your prediction, for example the unemployment rate, the oil price, the gold price, the interest rates, and the timeseries itself, i. e. the volatility, the change in the last 2,3,7,..., days etc. - a more classical approach is the input-output-analysis in econometrics, or the autoregression analysis, but all of it can be modeled using neural networks or any other function approximator / regression in a very natural way.

But, as said, there are tons of other possibilities to model the market, to name a few: Ant Colony Optimization (ACO), Classical regression analysis, genetic algorithms, decision trees, reinforcement learning etc. you name it, almost EVERYTHING has probably been applied to the stock market prediction problem.

There are different fond manager types on the markets. There are still the Quants which are doing a quantitative analysis using classical financial maths and maths borrowed from the physics to describe the market movements. There are still the most conservative ones which do a long-term, fundamental analysis of the corporation, that is, looking in how the corporation earns money and where it spends money. Or the tactical analysts who just look for immediate signals to buy / sell a stock in the short term. And those quantitative guys who employ machine learning amongst other methods.


I don't know ludwig or even "parallel CNN" but there's a clear problem: this is not a confusion matrix, this is a heat map.

A confusion matrix shows the number of instances for every combination of predicted/true label. It's for categorical labels, you could indeed obtain TP/FP/FN/TN classification status from it.
A heatmap is for numerical values, as can be seen from the scale on the right side.

It's very likely that this heatmap is made of the probabilities of the labels for every instance, rather than the labels themselves (also the choice of colours is unusual but this doesn't matter).
The good news is that it appears that most instances which have 0 or 1 as true label are correctly predicted (high value for 1,1 and 0,0).
The bad news is that there's an issue in the labels themselves: apparently a large number of instances (at least a third I think) don't have any label and are considered 'UNKNOWN'. And apparently the system just considers these 'UNKNOWN' instances as a third class, so your labels are not binary actually. It manages to recognize this class quite well, but it's probably an issue you should investigate.


The best way to solve your problem would be to collect the right labels for your task. Currently, you have the input i.e. text of a review and the rating which can be used to predict the sentiment of the review, but not whether it mentions safety issues. To collect the right labels, you should look at each review and assign it a binary label - 0 means food safety issues are not mentioned, 1 means they are mentioned. The feasibility of such data collection depends on the size of the dataset and the importance of your project. How big is your dataset? How much time/money are you willing to commit to that project? Once you collect the labels, you can train and evaluate a neural network based on them. If this answer helps you, feel free to ask for elaboration, I don't know what's your level of data science expertise and don't want to state the obvious.


Parallel hyperparameter optimization techniques?

Bayesian optimisation is sequential in the sense that you need to know the value of the function for n point to decide through an acquisition criteria the next point to evaluate.

Maybe you could customize it to your problem so that the acquisition returns not one point but a batch of them, which you distribute at the next step.

You can also use an hybrid method. First run a classic grid search, distributed, and evaluate the function at many many points. Feed all this knowledge (points and objective value at these points) to a classic bayesian optimiser which will pick points one by one and finer tune the optimisation here. Not as optimal as the former, but less implementation work here.


MAD vs RMSE vs MAE vs MSLE vs R²: When to use which?

Well actually these can give you different insights into your models errors.
If $y$ is your target, $p$ your prediction and $e = p - y$ the errors:

Mean Error: $ME = mean(e)$

In (-∞,∞), the closer to 0 the better.
Measures additive bias in the error. Unbiased estimates should have the same mean as your target thus ME should be close to 0, if it's positive your predictions overestimate the target, if it's negative they underestimate.

Root Mean Squared Error: $RMSE = \sqrt{mean(e^2)}$.

In [0,∞), the smaller the better.
Measures the mean square magnitude of errors. Root square is taken to make the units of the error be the same as the units of the target. This measure gives more weight to large deviations such as outliers, since large differences squared become larger and small (smaller than 1) differences squared become smaller.

Mean Absolute Error: $MAE = mean(|e|)$.

In [0,∞), the smaller the better.
Measures the absolute magnitude of errors and it's units are the same as the units of the target. Makes for more easily interprectable errors and gives less weight to outliers. However a model with good $MAE$ can have punctually very high errors.

(Root) Mean Squared Log Error: $MSLE= mean((log(p+1)-log(y+1))^2)$.

In [0,∞), the smaller the better.
This is useful when dealing with right skewed targets, since taking the log transform makes the target more normally distributed. In practice it's usually achieved by changing the target to $\hat{y}=log(y+1)$ and then predicting as $y=e^\hat{y}-1$

Median Absolute Deviation: $MAD = median(e - median(e))$.

In [0,∞), the smaller the better.
This is a spread metric similar to standard deviation but meant to be more robust to outliers. Instead of taking means of squares as the sd, MAD takes medians of absolutes making it more robust.

R², coefficient of determination:

In (−∞,1] the closer to 1 the better
Is a measure of the ratio of variability that your model can capture vs the natural variability in the target variable.

In practice I usually use a combination of $ME$, $R^2$ and: $RMSE$ if there are no outliers in the data, $MAE$ if I have a large dataset and there may be outliers, $RMLSE$ if the target is right skewed.

This link offers a very nice overview on the matter:
http://www.cawcr.gov.au/projects/verification/#Methods_for_foreasts_of_continuous_variables


How discriminator knows if the image is real or fake at the initial phase?

This a good question. I'll post my own personal considerations that should not be taken as rigorous answers, I'd like to broaden the question a little bit
[Disclaimer: personal considerations, not rigorous answer]
The generator and discriminator are not strictly learning together, they are learning one against other. Indeed, when the discriminator is training, the generator is frozen and vice versa.
At the very beginning of the training phase, the generated outputs of the generator are expected to be very far away from the real samples. It is also true that the discriminator is expected to not have a good feature space for the classification, but I think it gets to tell them apart faster than the generator begins to produce closer fake samples. Then, the minmax game evolves...
I thougth once to use a pre-trained classifier to force the generator produce real good fake images, but I also got to the feeling that, in that way, the battle will not be fair and will be biased to the discriminator side, regarding this:

Which input data should should be fed to the discriminator in its pre-training phase? I mean, to pre-train the discriminator, some pre-fake images are needed. One approach is to pre-train for a few steps the generator and produce pre-fake images which are far away from real images

Since the discriminator is still learning, are they chances it becomes even better?

The discriminator is so good it won't learn, is the game closer to nash equlibirum and so, closer to its end?
Taking into account the enormous problems when training GANs, I've never tried it out.

It would be fantastic if more experienced GANners offer more insight into these considerations


There are two solutions to this problem, though I'm only going explain the second one because I think is the best option.
First option:
10 Scatterplots of speed of vehicle on the x-axis and predicted speed of vehicle on the y-axis. The only scenario I see this plot been useful is for clustering different types of vehicles with their speeds (if there actually are many vehicle models). Naturally you could also aggregate all data points in a single scatterplot and color-code the data set (ex. dataset1 = blue, dataset 2 = red, etc.)
Second option:
Calculate the difference of vehicle speed and predicted speed for each data point and then bind all the results into a single data table so that each column contains the difference between actual speed and predicted speed for each data set.
Then plot each column as a bar in a boxplot, here you will be able to describe all differences including signs for outliers and mean. As you might already know boxes represent standard deviation (variance) and the central line represents the mean. You can find more information about boxplots here:
Wikipedia
Sample code in R and ggplot2
dt1 <- data.frame(data = c(runif(20, min = -1, max = 1),
                       runif(20, min = -1, max = 1),
                       runif(20, min = -2, max = 1)),
              datasetName = c(rep("dataset1", 20),
                              rep("dataset2", 20),
                              rep("dataset3", 20)))

ggplot(dt1, aes(x = datasetName, y = data)) +
geom_boxplot(fill = "white", colour = "#3366FF") +
labs(x = "Dataset Name", y = "Difference Between Speed and Predicted Speed",  title = "Datasets distribution")

The resulting image should look something like this:


Pls. refer Pattern recognition and Machine Learning for details - section 1.6: Information Theory.

Left graph is Information(events) vs Probability. And the right graph is Expected Information/(Uncertainity) vs Probability.

If we look carefully at the left graph; the variation at either extremes of the curve dies down, representing less change. Translated in current context, it means not much Information flows-in at extremes.

Staying with the left curve, it's somewhere in the middle (45-degree, from vertex) where there's relatively more "meat" in the sense that more change happens/ translated, more information flows-in.

Same thing is shown by the right curve: at two extremes, Change/ Expected Information is low at extremes, and highest at centre.

A crude Graph using Excel:


Take the example of time series: $\mathbf{x}1,\mathbf{x}2,\ldots,\mathbf{x}10$ where each $\mathbf{x}i$ is let 5 dimensional. The 'timestep' here will be the window chosen such that value at time instant is dependent on previous $p$ lags. So data passed to LSTM will be of the form $\mathbf{x}1,\mathbf{x}2$ as input with $p=2$ lags. In your case as well, each word is dependent on previous words. So timestep will be the number of previous words you need to pass to LSTM in context of current words. Batchsize argument is the number of examples after which the backpropagation will happen. It is a free parameter you can control it but it should be completely divisible by number of training examples. Feature will be 839, the length of embedding vector.


Multi Class + Negative Class Image Classification Strategies

There is no general strategy to anything in machine learning. Moreover, research is scattered over multiple domains so it becomes harder to get your head around, "the best strategy for a case". You can solve a single problem in multiple ways. Coming back to the multi class plus a negative class classification issue, this is one of "many approaches" you can follow:

Hierarchical classification: This is still an open problem. As the name suggests you perform classification on multiple levels starting from a root to internal node to leaf nodes (similar to tree or DAG structure). For eg. Given a digits data set ( 0,1,2,3...9) mixed with some letters (a,b,c....z). On a higher level you need to first classify if its a digit or not i.e. meta-class and if its a digit then you perform lower level classification task to identify the right digit.

There are different variants depending on your knowledge of labels/ groups at different level of hierarchy. For eg. if you don't have a pre-defined meta-class then you might have to perform clustering of similar classes at the higher level before final classification.

Also, you may want to tune the weights in your loss function to identify a particular class efficiently at any level of the classification.

P.S. My knowledge is limited in neural nets context. Definitely there would be more strategies specific to them.


How do I interpret my result of clustering?

K-means don't modify the underlying structure of your data. K-means will just provide the 'color' part of your graph.

To answer the question about why do you get a cuboid, it's because your underlying data are a cuboid. Not necessarily by construction, but that's what happen when you cap your data. As an exemple, look at the following code :

X1 = c(rnorm(1000))
X2 = c(rnorm(1000))
q95_1 = quantile(X1,0.95)
q95_2 = quantile(X2,0.95)
q5_1 = quantile(X1,0.05)
q5_2 = quantile(X2,0.05)
X1[X1>q95_1]=q95_1
X2[X2>q95_2]=q95_2
X1[X1<q5_1]=q5_1
X2[X2<q5_2]=q5_2
plot(X1,X2)

The code simulate two random gaussian and cap them at 5% and 95%.

this is what you get :

Notice the squaroid pattern ? This is why you get a cuboid in 3D.

Ps: I can't help but say that's what you get when you do k-means without properly looking at your variables (see: What value can I gain by doing exploratory data analysis on features (and thus data) before doing clustering? for an infinite loop).


Gradient Descent - how many values are calculated in loss function?

Gradient descent is based on sources: your data and your loss function.

In supervised learning, at each training step the predictions of the Network are compared with the atcual, true results. The value of a loss function is calcualated, which is telling your model "how wrong it is". At this point, the weights of the Network must be updated accordingly.

In order to do that, a formula based on the chain rule of derivatives calculates retrospectively the contribution of each weight to the final loss value. The value of each weight is then changed, based on their impact on the loss function (mathematically, it's a first partial derivative for each weight). This process is called backpropagation, since it logically starts from the bottom of the Network and is computed backwards up to the input layer.

This process has to be done for each of the Network's learnable weights. The higher the number of parameters, the higher the number of partial derivatives that are computed at each training iteration for the weight update. When all the weights are updated, the position of the Gradient will (hopefully) be a small step closer to the global minimum of the loss function.

Coming to your doubts: grid search on the loss function values is something you can do only in theory, not in practice. It could take an impossible amount of time to do that, even for the most powerful computers. At the moment, the Gradient descent algorithm is the main tool to train Neural Networks. As far as I know, other methods such as Monte Carlo and Genetic algorithms are feasible but not practical, therefore not state-of-the-art.

I suggest you two great posts to understand how Gradient descent and backprop work: this one by Andrej Karpathy and this one by Colah.


This answer is borrowed from-

https://stackoverflow.com/questions/36232334/plotting-3d-decision-boundary-from-linear-svm

from sklearn.svm import SVC
import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets
from mpl_toolkits.mplot3d import Axes3D

iris = datasets.load_iris()
X = iris.data[:, :3]  # we only take the first three features.
Y = iris.target

#make it binary classification problem
X = X[np.logical_or(Y==0,Y==1)]
Y = Y[np.logical_or(Y==0,Y==1)]

model = svm.SVC(kernel='linear')
clf = model.fit(X, Y)

# The equation of the separating plane is given by all x so that np.dot(svc.coef_[0], x) + b = 0.
# Solve for w3 (z)
z = lambda x,y: (-clf.intercept_[0]-clf.coef_[0][0]*x -clf.coef_[0][1]*y) / clf.coef_[0][2]

tmp = np.linspace(-5,5,30)
x,y = np.meshgrid(tmp,tmp)

fig = plt.figure()
ax  = fig.add_subplot(111, projection='3d')
ax.plot3D(X[Y==0,0], X[Y==0,1], X[Y==0,2],'ob')
ax.plot3D(X[Y==1,0], X[Y==1,1], X[Y==1,2],'sr')
ax.plot_surface(x, y, z(x,y))
ax.view_init(30, 60)
plt.show()


You matrix will be sparse, so the memory usage will be inefficient. Try to look on LSH - locality sensitive hashing for clustering.

https://en.wikipedia.org/wiki/Locality-sensitive_hashing


Why does using a standard scalar on my tf idf matrix make it perform better?

I am not sure what you mean by two-fold normalized. TFIDF values are not normalized per se. They will fall within a somewhat constant range of values, but this completely depends on the dataset.
The StandardScaler performs normalization by removing the mean and scaling to unit variance. Since TFIDF values may vary from dataset to dataset, using the StandardScaler will have an effect on the values, which explains the difference in performance.


First it's not a matter of duration, it's a matter of number of data points that can be collected: if there are only a few users every day, it's going to take much longer to collect enough data points than if there are millions.
Now how many data points do you need? There's no simple answer since it depends what exactly you are going to test and how big the difference turns out to be between the two cases.
A good workaround is to start with some hypothetical scenarios: imagine for instance 100, 1000 or 10000 users and an outcome with a large, a medium or a small difference. Run the test in all these different scenarios and check when a significant difference is obtained. Based on this, choose a target number of users which would be sufficient to obtain a clear result in most of the scenarios.
Finally the duration can be approximated based on the average number of daily users.


What is the difference between majority vote, and greedy action in ensembling?

Ensemble learning basically means combining several base models to produce one optimal solution.

And from my understanding,
usually in Majority vote, we train the models with all the data and wait for all the models to result in a prediction and at the end we choose the major result (result that appeared more than any other) as the final output.

But in Greedy way, we usually iterate on the data, run prediction on a subset of the data, pick the best result of this subset and move to the next, and at the end we only consider the best results of each iteration. So we don't look at the big picture here but only a collection of subsets. Not the Global best result but Local best results.

Hope this helps.


Shifting dataPoints up by a constant (Is there an issue with too many 0's for features?)

From your question I am not sure how you are using logistic regression. The "vanilla" version of logistic regression gives a binary classifier, but your target has three values: a positive value, a negative value, and zero. The Apache site, on the page for logistic regression refers to the thesis of Paul Komarek, in which "logistic regression" refers to the binary classifier. So your problem might be that there your target variable has 3 values.

Assuming that that is the case, you have many options, for example:
 - a) use "multi-nomial logistic",
 - b) use ordinary logistic regression 3 times to fit 3 models, one per level, to distinguish it from the other two combined


Gradient boosting vs logistic regression, for boolean features

You are right that the models are equivalent in terms of the functions they can express, so with infinite training data and a function where the input variables don't interact with each other in any way they will both probably asymptotically approach the underlying joint probability distribution.  This would definitely not be true if your features were not all binary.

Gradient boosted stumps adds extra machinery that sounds like it is irrelevant to your task.  Logistic regression will efficiently  compute a maximum likelihood estimate assuming that all the inputs are independent.  I would go with logistic regression.


If you want to give the means, then it's not clustering anymore.

The name-giving part of k-means is computing the mean.

Instead, you are doing nearest neighbor classification. Your predefined means are your "training" data.


GIFs are just a sequence of images

Given that you just need to find right quantification/pooling of these images to display a GIF. When you displayed a single GIF it will be a collection of different images, where flashing images will have sudden changes in pixel intensity, exactly that is what you want to catch in set of images=GIFs that are malicious.  You just need binary classification from there on.

have a look at here


It seems to me that your premise for doing this is potentially flawed. It sounds like you're trying to replicate some information that power companies can generate but they are working with more broad datasets than you have presented here. In turn, this can be a reason why your accuracy scores as so poor.

For example, consider that weather has an effect on power usage. So unless you're wrangling in weather data at some point, you will (1) most likely never achieve the most accurate model possible (might just be "adequate") and (2) most likely never approach similar results to what your power company can generate.

So, I would take a step back and consider your current data points; it's highly likely that you just don't have the right factors there in place to create the accurate model you seek.


Confused about the different aspects in Machine Learning

Good question and welcome to Datascience

Imagine you have the tree as follows.

                 Machine Learning Models
                          |
           ----------------------------------------------------
           |                                                  |
       Supervised                                         Unsupervised
           |                                                  |
   - --------------------                                  Clustering
   |                    |
Regression          Classification

A model is indeed a trained version of the algorithm you are chosing,
regardless of whether its Regression, Classification or Clustering. The above
tree is ofcourse a simplified version of all ML algorithms.


I tried to construct the following simple example (mostly for my self-understanding) which I hope could be useful for you. If someone else notices any mistake please let me know. This is somehow based on the following nice explanation of gradient boosting http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/

The example aims to predict salary per month (in dollars) based on whether or not the observation has own house, own car and own family/children. Suppose we have a dataset of three observations where the first variable is 'have own house', the second is 'have own car' and the third variable is 'have family/children', and target is 'salary per month'. The observations are

1.- (Yes,Yes, Yes, 10000)

2.-(No, No, No, 25)

3.-(Yes,No,No,5000)

Choose a number $M$ of boosting stages, say $M=1$. The first step of gradient boosting algorithm is to start with an initial model $F_{0}$. This model is a constant defined by $\mathrm{arg min}_{\gamma}\sum_{i=1}^3L(y_{i},\gamma)$ in our case, where $L$ is the loss function. Suppose that we are working with the usual loss function $L(y_{i},\gamma)=\frac{1}{2}(y_{i}-\gamma)^{2}$. When this is the case, this constant is equal to the mean of the outputs $y_{i}$, so in our case $\frac{10000+25+5000}{3}=5008.3$. So our initial model is $F_{0}(x)=5008.3$ (which maps every observation $x$ (e.g. (No,Yes,No)) to 5008.3.

Next we should create a new dataset, which is the previous dataset but instead of $y_{i}$ we take the residuals $r_{i0}=-\frac{\partial{L(y_{i},F_{0}(x_{i}))}}{\partial{F_{0}(x_{i})}}$. In our case, we have $r_{i0}=y_{i}-F_{0}(x_{i})=y_{i}-5008.3$. So our dataset becomes

1.- (Yes,Yes, Yes, 4991.6)

2.-(No, No, No, -4983.3)

3.-(Yes,No,No,-8.3)

The next step is to fit a base learner $h$ to this new dataset. Usually the base learner is a decision tree, so we use this.

Now assume that we constructed the following decision tree $h$. I constructed this tree using entropy and information gain formulas but probably I made some mistake, however for our purposes we can assume it's correct. For a more detailed example, please check

https://www.saedsayad.com/decision_tree.htm

The constructed tree is:

Let's call this decision tree $h_{0}$. The next step is to find a constant $\lambda_{0}=\mathrm{arg\;min}_{\lambda}\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$. Therefore, we want a constant $\lambda$ minimizing

$C=\frac{1}{2}(10000-(5008.3+\lambda*{4991.6}))^{2}+\frac{1}{2}(25-(5008.3+\lambda(-4983.3)))^{2}+\frac{1}{2}(5000-(5008.3+\lambda(-8.3)))^{2}$.

This is where gradient descent comes in handy.

Suppose that we start at $P_{0}=0$. Choose the learning rate equal to $\eta=0.01$. We have

$\frac{\partial{C}}{\partial{\lambda}}=(10000-(5008.3+\lambda*4991.6))(-4991.6)+(25-(5008.3+\lambda(-4983.3)))*4983.3+(5000-(5008.3+\lambda(-8.3)))*8.3$.

Then our next value $P_{1}$ is given by $P_{1}=0-\eta{\frac{\partial{C}}{\partial{\lambda}}(0)}=0-.01(-4991.6*4991.7+4983.4*(-4983.3)+(-8.3)*8.3)$.

Repeat this step $N$ times, and suppose that the last value is $P_{N}$. If $N$ is sufficiently large and $\eta$ is sufficiently small then $\lambda:=P_{N}$ should be the value where $\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$ is minimized. If this is the case, then our $\lambda_{0}$ will be equal to $P_{N}$. Just for the sake of it, suppose that $P_{N}=0.5$ (so that $\sum_{i=1}^{3}L(y_{i},F_{0}(x_{i})+\lambda{h_{0}(x_{i})})$ is minimized at $\lambda:=0.5$). Therefore, $\lambda_{0}=0.5$.

The next step is to update our initial model $F_{0}$ by $F_{1}(x):=F_{0}(x)+\lambda_{0}h_{0}(x)$. Since our number of boosting stages is just one, then this is our final model $F_{1}$.

Now suppose that I want to predict a new observation $x=$(Yes,Yes,No) (so this person does have own house and own car but no children). What is the salary per month of this person? We just compute $F_{1}(x)=F_{0}(x)+\lambda_{0}h_{0}(x)=5008.3+0.5*4991.6=7504.1$. So this person earns $7504.1 per month according to our model.


Will the MAE of testing data always be higher than MAE of training data?

Test MAE of a model can be lower than Train MAE when it falls under below cases:

It is possible when you have not sampled the data or split the test train data perfectly.
It is possible when your test data is small and its not a good representative of train data, then there may or may not be a case when for that test data it behaves good and gives low error.
There can be a case when you use regularization and your train data move away from overfitting i.e. increased train error but generalizing good on test data i.e. low train error.


Prediction approach on unique data or progressive data

Creating model on second scenario is difficult but more information driven and give you more prediction capibility.

What you can do is use second scenario and create some featured variable. These variable will give information like after marrying in how many months attrition happen and so on.

So think what other info you can take out of this trend.

First scenario is not of much use, because it's just giving you last information of employee which hides many important relevant information which are actually driving attrition.


What is the best practice to classify category of named entity in sentence

I'm not sure I fully understand your question, but it seems to me that you're trying to determine the category of the string/entity "titanic" out of context. Your data tells you that "titanic" could be a book, a movie, or a product, and you want to figure out which one is correct -- is that what you're trying to do?

If so, the problem is that you've dropped the context in which the string/entity "titanic" appears in your original text. For example...

In the sentence "I couldn't stop reading Titanic," the word "titanic" refers to a book.
In the sentence "Titanic was one of the highest-grossing films of all time," the word "titanic" refers to a movie.
In the sentence "The Titanic was the world's largest ocean liner," the word "titanic" refers to a product.

Without that context, there's no way to know which is the correct category. I'd suggest looking into how named entity recognition tools like Stanford NER work -- that will help you better understand how to do something like this. You'll see that the input to an NER tool generally needs to be a sentence, in order to take advantage of the context to properly categorize the extracted entities.


Using sklearn DictVectorizer in real-time systems

If you're doing anything at the millions of records per second level, you're going to need something more high-performance than Python. Spark/PySpark is ideal for this since it interfaces with your existing Python knowledge and supports HDFS/distributed technologies that are made to take on the millions per second sized tasks.

As for sklearn: you might be out of luck here. The fit_transform() or fit() methods necessarily have to consider every record that you pass to them to keep the hash up to date, so if you have millions of data points coming in you're going to have to do millions of computations to keep moving, whether you do them one at a time or in a batch. The source code for the transform() method should convince you that doing either is equivalent.

One caveat is that if you have predictable features, e.g. maybe you're receiving counts of foo's bar's and baz's, like in the first link:

D = [{'foo': 1, 'bar': 2}, {'foo': 3, 'baz': 1}]

Then you don't need to update the hash every time, all you have to do is transform D into the vectorized format and add it to your collection. As long as you don't need to fit new features this can be done relatively fast. However, if you're doing something less structured like NLP this won't give you hardly any useful information.


I had a similar problem in my last job. My solution was to build features via (transformation(s) + comparison) * many combos and feed to models, then aggregate and model, i.e. 2 layer model. The key is encoding and similarity scores as features.

Transforms: remove vowels (great for certain roots), remove end vowels, remove double characters, convert to phonetic string (IPA, soundex, https://pypi.org/project/Fuzzy/), replace characters that either sound similar or have different sounds in other languages ($J$ in East Europe sounds like $Y$ in US, $C$ can sound like $K, D~T, T\sim TH$, etc), ... The strategy is to handle lots of weirdness/irregularity in people's names.

Comparisons (similarity and difference): try [character level, block/root/[pre/suf]fix level, word level (may not apply to you)] similarity and difference scores. Try Dice's coefficient, Levenshtein, Needleman–Wunsch, Longest common (non)contiguous substring, character histogram similarity, # characters matching, not matching (each left and right), etc. You could try using an RNN/LSTM and have it learn similarity for each transform. Use the output of the trained model(s) as another feature.

Experiment with different combos of the above and select a few that seem to have value. You could simply take all the scores and fit with Logistic Regression (or Neural Net) or you could build statistical models and output percent rank based on a small training set to normalize it. Another way to preprocess the raw scores is to use calibration encoding via logistic function. Then add summary stats from the normalized scores as additional features. Push all this into the final model.

Will you handle names that are derived from Arabic, Spanish, French, etc names? This is just extra, but consider downloading the Social Security and US Census name stats data to enhance your project with more name variations. I'll leave the how to you, but it helps knowing about the likely possibilities. Be aware that simply using Levenshtein doesn't work so well with William->Bill, Dianne->Di, Larry->Lawrence, Mohammed->Muhamed and Hamed, Danielle->Daniela, Thomas->Tom, and Jimmy->James. The strategy I mentioned should help you with all the variation.

Additional resources to explore:
https://github.com/jamesturk/jellyfish
https://nameberry.com/list/276/If-You-Like-Danielle-You-Might-Love
https://pypi.org/project/phonetics/


A class is simply a label you use to categorize a bunch of objects. For example, if you were trying to create an email filter, you might have a spam class and non-spam class.

A Gini index is used in decision trees. A single decision in a decision tree is called a node, and the Gini index is a way to measure how "impure" a single node is.

Suppose you have a data set that lists several attributes for a bunch of animals and you're trying to predict if each animal is a mammal or not. You would have two classes, mammal, and not-mammal. You start making your decision tree by asking if an animal is warm blooded or not and split your data set into two groups based on this splitting criteria. If an animal is cold blooded, it belongs to the not-mammal class, however, if an animal is warm-blooded, it may or may not belong to the mammal class. This new node (e.g., decision) might contain a mix, or group, of animals that may or may not be mammals (i.e., the group could contain birds and mammals). A 50/50 split between mammals and non-mammals at this node would mean the node is impure (with a Gini index of 0.5). A completely pure node would have a Gini index of 0 and would indicate a node is made up of only 1 class.


What does it mean when we say most of the points in a hypercube are at the boundary?

Speaking of '$99\%$ of the points in a hypercube' is a bit misleading since a hypercube contains infinitely many points. Let's talk about volume instead.

The volume of a hypercube is the product of its side lengths.
For the 50-dimensional unit hypercube we get $$\text{Total volume} = \underbrace{1 \times 1 \times \dots \times 1}_{50 \text{ times}} = 1^{50} = 1.$$

Now let us exclude the boundaries of the hypercube and look at the 'interior' (I put this in quotation marks because the mathematical term interior has a very different meaning). We only keep the points $x = (x_1, x_2, \dots, x_{50})$ that satisfy
$$
0.05 < x_1 < 0.95 \,\text{ and }\, 0.05 < x_2 < 0.95 \,\text{ and }\, \dots
\,\text{ and }\, 0.05 < x_{50} < 0.95.
$$
What is the volume of this 'interior'? Well, the 'interior' is again a hypercube, and the length of each side is $0.9$ ($=0.95 - 0.05$ ... it helps to imagine this in two and three dimensions).
So the volume is $$\text{Interior volume} = \underbrace{0.9 \times 0.9 \times \dots \times 0.9}_{50 \text{ times}} = 0.9^{50} \approx 0.005.$$
Conclude that the volume of the 'boundary' (defined as the unit hypercube without the 'interior') is $1 - 0.9^{50} \approx 0.995.$

This shows that $99.5\%$ of the volume of a 50-dimensional hypercube is concentrated on its 'boundary'.

Follow-up: ignatius raised an interesting question on how this is connected to probability. Here is an example.

Say you came up with a (machine learning) model that predicts housing prices based on 50 input parameters. All 50 input parameters are independent and uniformly distributed between $0$ and $1$.

Let us say that your model works very well if none of the input parameters is extreme: As long as every input parameter stays between $0.05$ and $0.95$, your model predicts the housing price almost perfectly.
But if one or more input parameters are extreme (smaller than $0.05$ or larger than $0.95$), the predictions of your model are absolutely terrible.

Any given input parameter is extreme with a probability of only $10\%$. So clearly this is a good model, right?
No! The probability that at least one of the $50$ parameters is extreme is $1 - 0.9^{50} \approx 0.995.$
So in $99.5\%$ of the cases, your model's prediction is terrible.

Rule of thumb: In high dimensions, extreme observations are the rule and not the exception.


Create a prediction formula from data input

I'm not sure if I understood your question! Probably better to plot a scheme at least. But according to what I guess from your question:

Q2- You probably need a simple MLP (Multilayer Perceptron)! it's a traditional architecture for Neural Networks where you have $n$ input neurons (here 20-25), one or more hidden layers with several neurons and 3 neurons as output layer. If you use a sigmoid activation function ranged from 0 to 1, the output for each class will be $P(Y=1|X=x)$.

Q1- So your question probably is: how many training data you need for learning a model? and to the best of my knowledge the answer is as many as possible!

and about the last question, I really could not figure out what you mean. You apparently have a very specific task so I suggest to share more insight for sake of clarification.

I hope I could help a little!


Supervised learning for variable length feature-less data

This is a typical scenario in language processing, e.g. if you want to read a product review representing each word with a number and output a 1-5 star rating.

You can use a recurrent many-to-one RNN/GRU/LSTM network: After feeding each word it generates a feature vector which can be fed into a classifier.
https://en.wikipedia.org/wiki/Recurrent_neural_network


Histogram indicating 20~25% in a certain range

Histograms are not reliable methods to communicate summary statistics because it depends on bin size choice.
The best option to answer numerical questions to find the raw numerical data.


How would you categorize email subjects to find similar emails?

As per your problem and the description, I'll suggest you to try Text Mining - Bag of Words approach. I did something similar using the same approach and it was really helpful.

More details about the approach are here.

Give it a try. Cheers!


Is empirical risk the same thing as loss function?

I found the accepted answer pretty hard to understand. Here is a simplified version that cleared it up for me:

Loss Function: a loss or risk function, $\mathcal{L}(\hat{y}^i, y^i)$, quantifies how well (more accurately, how badly) $\hat{y}$ approximates y. smaller values of $\mathcal{L}(\hat{y}^i, y^i)$ indicate that $\hat{y}$ is a good approximation of $y$

Empirical Risk: the empirical risk is the average loss over the data points. $$\mathcal{L} = \frac{1}{n}\sum_{i=1}^n \mathcal{L}(\hat{y}^i, y^i)$$

See these Stanford Lecture Slides for more details.


which loss function (if any) optimizes the calibration graph

Log-loss optimizes your predictions in terms of their probability so in essence, yes it should be optimizing your calibration curve.  i.e. if you predict probability of 0% but it's actually labelled as the true class, this is penalised more than probability of 5% being true when it is the true class and so forth.


How to get a (descriptive) overview of a large database?

There's no way to have a complete summary of a large dataset like this, you have to analyze what can be relevant, decompose into more specific pieces of information and then find the best way to visualize each specific part on its own.
The first thing would be to plot the distribution of this parameter of interest across subjects and/or observations.
If you want to look at the individual level and there are too many values, you can simply pick a random subset (say 100 subjects) and plot these. Then you do it again with a different random subset in order to distinguish real patterns from variations due to chance.


I think the number of elements in each lists are different. (26 , 22 , 26 , 18)

Zip() function uniformly maps all elements of lists of same sizes. (And that's why the 1's are missing and would round off to 18 elements)

Try maintaining the same number of elements in each List and try to find the mean again.


What do I need to learn in order to really understand Machine Learning?

From somebody with a PhD in Probability working with AI/ML for a living. Basics of Probability theory, maybe wikipedia/cousera/... for the very basics if you never had a class in it, followed by e.g. “Probability with Martingales” by Williams.
The papers here will also give you a good feel.
As for books, this one on “classical” machine learning is pretty good and free For Deep Learning https://www.deeplearningbook.org/, this one also has the basics of probability theory.
For reinforcement learning http://incompleteideas.net/book/the-book-2nd.html.
For the applied side, slides from https://web.stanford.edu/class/cs224n/ and http://cs231n.stanford.edu/.
As for statistics don’t bother, I’ve never seen any machine learning work reference a theorem in “pure” statistics, if there is such a thing.
Of course standard undergraduate calculus and linear algebra. And learn some Python while you’re at it