Which machine learning algorithms support a feedback ingestion loop?

A few other methods support incremental learning. (Sorry it should be a comment instead but I don't have enough reputation ;)

http://scikit-learn.org/stable/modules/scaling_strategies.html#incremental-learning


Confused about steps_per_epoch and validation_steps in Keras

There is an additional functionality with this method, way to think about it is a chance to use mini-epochs if loss stagnates too much

Usually it is equal to n_samples // batch_size, BUT

steps_per_epoch give you the chance to feeding process of the NN when updating the learning rate using ReduceLROnPlateuau(). This callback checks the drop of the loss once each epoch has finished and if the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than n_samples // batch_size without affecting the overall number of training epochs of your model.


Encoding features for multi-class classification

Color is a categorical feature.
One of the most common methods to encode categorical features is one-hot encoding. Color could be encoded as an indicator vector. The color of the current car would have a 1 at the appropriate index. For example, [1, 0, 0, …, 0] for a red car and [0, 1, 0, …, 0] for a blue car.
There are other options for encoding categorical features such as binary, count, hash, or label encoding.


How to start the process of coming up with the predicted math score?

What you are looking for is a machine learning algorithm. Although the easiest way would be to take the average scores and use that, there are much more accurate ways to make predictive models.

This was the first data science tutorial I did. It's perfect for getting started. Here is it in R and in Python.

If you're looking for a short answer, something you can just look up how to implement, I'd check out Random Forests.


Positive semidefinite kernel matrix from Gower distance

You may use nearPD in R to convert a matrix to its Nearest Positive Definite counterpart.


How to determine if my GBM model is overfitting?

The term overfitting means the model is learning relationships between attributes that only exist in this specific dataset and do not generalize to new, unseen data.
Just by looking at the model accuracy on the data that was used to train the model, you won't be able to detect if your model is or isn't overfitting.

To see if you are overfitting, split your dataset into two separate sets:

a train set (used to train the model)
a test set (used to test the model accuracy)

A 90% train, 10% test split is very common. Train your model on the train test and evaluate its performance both on the test and the train set. If the accuracy on the test set is much lower than the models accuracy on the train set, the model is overfitting.

You can also use cross-validation (e.g. splitting the data into 10 sets of equal size, for each iteration use one as test and the others as train) to get a result that is less influenced by irregularities in your splits.


To me (coming from a relational database background), "Big Data" is not primarily about the data size (which is the bulk of what the other answers are so far).

"Big Data" and "Bad Data" are closely related. Relational Databases require 'pristine data'. If the data is in the database, it is accurate, clean, and 100% reliable. Relational Databases require "Great Data"  and a huge amount of time, money, and accountability is put on to making sure the data is well prepared before loading it in to the database. If the data is in the database, it is 'gospel', and it defines the system understanding of reality.

"Big Data" tackles this problem from the other direction. The data is poorly defined, much of it may be inaccurate, and much of it may in fact be missing. The structure and layout of the data is linear as opposed to relational.

Big Data has to have enough volume so that the amount of bad data, or missing data becomes statistically insignificant. When the errors in your data are common enough to cancel each other out, when the missing data is proportionally small enough to be negligible and when your data access requirements and algorithms are functional even with incomplete and inaccurate data, then you have "Big Data".

"Big Data" is not really about the volume, it is about the characteristics of the data.


Sliding window leads to overfitting in LSTM?

Although the previous answer by @Imran is correct, I feel it necessary to add a caveat: there are applications out there where people do feed a sliding window in to an LSTM. For example, here, for framing forecasting as a supervised learning problem.

If your data are not very rich, then you may find that any LSTM at all overfits. There are a lot of parameters in an LSTM (in fact, there are $4(mn + n^2 + n)$ parameters, where $m$ is the input length and $n$ is the output length, according to this post).

Since LSTMs do not require fixed size input, they can find the optimal lookback number on their own. However, if you've done a prior autoregressive analysis and decided that, for example, the current time step is most correlated with the 10th previous time step, and not correlated with the 11th or any time steps further in the past, then you could perhaps save yourself some training time by feeding in fixed-length sequences. However, that kind of defeats the purpose of an LSTM.

If your data are not rich enough for an LSTM, I would recommend trying something much simpler, like an autoregressive model, and working your way up.

EDIT (responding to a comment):

Overlapping sequences are used as input, especially when the sequence is very long (although, of course, 'long' is relative). Although LSTMs are better than a vanilla RNN for long sequences, they can still have some trouble remembering time steps from the beginning of a sequence if the sequence is very long. That led to things like the bidirectional LSTM, which reads the sequence forwards and backwards, improving the exposure of the network to the beginning and end of each input sequence. The principle is the same with overlapping sequences, although I would argue that overlapping sequences is more intuitive.


If you are talking about exploring your data for patterns of missing data you can try using Self-Organizing Maps [https://en.wikipedia.org/wiki/Self-organizing_map], which are a special flavor of neural networks. Here is a small research paper explaining the concept a bit. Here is another [link][3] with historical info and links to some original papers about the topic, specifically by Kohonen et al.

From Wikipedia:

  Self-organizing maps are different from other artificial neural
  networks as they apply competitive learning as opposed to
  error-correction learning (such as backpropagation with gradient
  descent), and in the sense that they use a neighborhood function to
  preserve the topological properties of the input space. This makes
  SOMs useful for visualizing low-dimensional views of high-dimensional
  data...

Essentially this method will allow you to look for patterns in your data. From there you can determine what level of completeness you are dealing with in your data set.

Specifically for your example for Twitter data, I would imagine there are many fields in the JSON data with missing values. Maybe some users choose not to fill in their gender or age etc. Visualizing your data as well calculating summary statistics will help you to paint an overall picture of your data. And when you have high dimensional data using tools to visualize it in a lower dimensional space can always be handy. Hope this helps you!


Predicting when a partition in Oracle database will be archived

I'd suggest first visualize your data.

The time to archive a partition could be defined as a difference between the partition start (first possible insert) and archiving time (last possible update).

Plot the total distribution of this data for some time interval - check if it follows some distribution. Is the average, median or sd meaningful?

Make a boxplot for each hour to see inter quartile interval and the outliers.

Plot it as a time series - does it have a trend or daily/weekly pattern?

In your simple set up (with one predictor and one response) this will give you impression about the accuracy that you can expect from a model. The choose of the model depends on the result of the visualization.

I personally would start with the simplest models:

all partition archived after $N$ hours;
independent model for each hour predicting say $.9$ quantile of some
history interval;
more advanced 2.: consider the weekday and/or trend and/or
ignoring outliers.

Always check the accuracy and stop if expectations are met.


Error in lmest: missing data in the covariates affecting the initial probabilities are not allowed

I'm answering my question here. The issue was eventually addressed by reordering the structure of my dataset. In my original dataset, the response variables were placed at the first few columns, followed by a list of covariates. To use the LMest package, response variables should be placed towards the end of the dataset.


No, you don't train against 3 separate databases, because for each dataset, you would need to do the whole work from scratch ( exploratory data analysis , feature engineering etc..). You need a valid training dataset, from which you create your validation dataset and test dataset ( a hold-out set as a final verification step ). Yeah sure, you deploy your model for example as a webservice on a test server to make sure is working as expected. But model verification, including avoiding overfitting or underfitting, having a good accuracy and all that is done prior to any deployment.


Learning character sequences and predicting sequences

I think you might want to check Andrej Karpathy's work around charRNNs some pretty cool work is being done. The github link has all the relevant code too.
If you are looking for a more applied way - you can check my blog on deep learning gender from name which effectively uses character level LSTM RNNs to learn patterns :
medium.com/@prdeepak.babu/deep-learning-gender-from-name-lstm-recurrent-neural-networks-448d64553044

Hope this help.


How to build an encoder using a distance matrix

This is essentially what t-SNE does. It supports transforming data from any input dimension to any output dimension, and it tries to preserve distances between all pairs of points.

From the author's website:

  Can I use a pairwise Euclidean distance matrix as input into t-SNE?

  Yes you can! Download the Matlab implementation, and use your pairwise
  Euclidean distance matrix as input into the tsne_d.m function.


How can my Pytorch based GAN output pure B&W with no grayscale?

The output is going to be "continuous" if you don't use it for "classification".

You can follow few approaches here:

This would be the simplest approaches but you probably need some post processing to fill holes in lines:

Define a Threshold to make the image binary;
Redefine the last layer to output a image with 2 channels and define your final image by taking the index of the maximum channel value for each pixel (as it was a classification problem)

You can also redefine your problem:

PS:Every time a mention "line" from now on I actually mean line segment.

Given that your images are a set of geometric figures, which can all be broken down to lines and any line can be described by a set of 4 numbers, you can use a RNN to generate the set of lines (pair of points) that can be drawn easily afterwards.

To do so you need to rewrite your training set:

Define a line width for your images
Based on that line width you can set a number of line orientations possible (for example, with width of one pixel you can only draw lines in 4 orientations 0, 45, 90 135 degrees)
Use that to find every point that define line (start and end) by finding points where these orientation changes
To transform the set of points into a set of lines you can check if there is a black pixel in that orientation that is in the interval.

Binary Layer Issue

Your binary layer implements a Heaveside Step function which will kill the gradient since it's derivative is 0 for all values other than x=0. In x=0 the derivative is actually not defined (limit goes to infinity).

Thus your weights can't update, so every forward pass will filter data with random weights, causing random effects to the output.

Check this answer to understand more about why we don't use Heaveside Step function as an activation function on neural networks


How can I do web scrapping of all the articles containing certain keywords mentioned in a dictionary, using Python?

if you want to just pull news articles from BBC news, then you could use their API.

they provide client libraries in different languages (including python)

quoting documentation:

You can search for articles with any combination of the following criteria:

Keyword or phrase. Eg: find all articles containing the word 'Microsoft'.
Date published. Eg: find all articles published yesterday.
Source name. Eg: find all articles by 'TechCrunch'.
Source domain name. Eg: find all articles published on nytimes.com.
Language. Eg: find all articles written in English.

I believe the first one is what you are looking for.


How to understand features impact in a non linear case?

As far as I know, kernel methods cannot deal with categorical variables (don't now whether it is the case). In addition, you will have to use indirect methods to evaluate the variable importance. This could work, although I havent tested it yet:

Giam, X., Olden, J.D., 2015. A new R2-based metric to shed greater insight on variable importance in artificial neural networks. Ecol. Modell. 313, 307–313. http://dx.doi.org/10.1016/j.ecolmodel.2015.06.034

I would definitively go for a tree-based approach. Since you already know that there are correlated variables, I would advocate for conditional Random Forests (which solve many drawbacks of the standard random forests implementation). Check:

Strobl, C., Hothorn, Zeileis, A., 2009. Party on! R J. 1 (2), 14–17.

And references therein. At least in R there are complementary packages (https://cran.r-project.org/web/packages/pdp/index.html) that allow plotting the impact of each predictor variable over the target variable (house prices). That complements the variable importance ranking pretty well.

Good luck.


Joining two dataframes on the basis of specific conditions

Follow mentioned link for your problems.

https://www.shanelynn.ie/merge-join-dataframes-python-pandas-index-1/


Machine learning for Point Clouds Lidar data

First, CNNs are great for image recognition, where you usually take sub sampled windows of about 80 by 80 pixels, 27,000 input neurons is too large and it will take you forever to train a CNN on that.

Furthermore, why did you choose CNN? Why don't you try some more down to earth algorithms fisrst? Like SVMs, or Logistic regressions.

4500 Data points and 27000 features seems unrealistic to me, and very prone to over fitting.

Check this first.

http://scikit-learn.org/stable/tutorial/machine_learning_map/


You are minimizing the entire loss equation. If it contains regularization, you force the weights to be small too. Having small weights is favorable characteristic because the algorithm is not focusing strongly on one feature, all happen to be important, so the risk of overfitting to some feature is smaller.

So it is really some tradeoff as always, between the big margin but small focus on dataset distinct examples.


Your input is not standardized
The learning rate is way too high, start with the Default i.e. 0.001

Other suggested changes -

Use "relu" in the hidden/input layer
OHE the target
If the target is multi-class, the output layer should have same number of Neurons with softmax as activation
Data points are very less, Neural Net might not be the best option


Text classification for data with multiple labels per observation

There are different options:

Multi-class classification: each tweet can be labelled with several classes
Majority voting: each tweet is labelled with the class chosen most often
Consider tweets with (too many) different labels as ambiguous and remove them from the dataset
...

Additionally it's usually useful to calculate the inter-annotator agreement, because this indicates how subjective the task is and therefore how accurate one can expect the predictions to be.


Why is there such a wide range of prices for whole genome sequencing services?

Quality
Certificates (is "for research use only" ok for you?)
There is no "common" or "average" genome, some of the prices are for bacterial some for human(4,6 x 10 ^6 bp (E.coli) vs. 3.3 x 10^9 bp(human))


You would indeed need to calculate the difference in days between the Date Entered and the Date Closed column. You can then either (1) group directly by the number of days and look at the percentage of records where the Status is equal to Open or Lost or (2) first create buckets and then perform the same step as in (1). You are then free to choose the percentage which you feel equals "likely" (50%, 80%, 99%?) and then use the difference in days as the threshold.


I got 100% accuracy on my test set,is there something wrong?

There may be a few reason this is happening.

First of all, check your code. 100% accuracy seems unlikely in any setting. How many testing data points do you have? How many training data points did you train your model on? You may have made a coding mistake and compared two same list.
Did you use different test set for testing? The high accuracy may be due to luck - try using some of the KFoldCrossValidation libraries that are widely available.
You can visualise your decision tree to find our what is happening. If it has 100% accuracy on the test set, does it have 100% on the training set
?


Why can't we sample without replacement for each tree in a random forest if the subsample size is large enough?

No, the samples will not be independent, there is possibility the data samples will be skewed.
For example, imagine a class-imbalanced binary problem, once the minority class is already sampled (large possibility that this can happen given $n$ and $m$) then, without replacement, the rest trees will only sample from the majority class which will produce skewed trees.
Some references:

“Bagging based on resampling with and without replacement is equivalent”, is it?

For random forests, generally, the concept of replacement is
considered essential. This is because the underlying concept of random
forests is bagging to prevent overfitting, i.e., bagging builds an
ensemble of estimators trained on data with a high variance (with
regard to the training data they have seen).

Why does random forest use sampling with replacement instead of without replacement?

The basic idea of bootstrapping is that you use your sample as a
population. And from it you sample repeatedly, with replacement, to
build other samples of the same size as your original sample.
Replacement is an integral part of this process because you are trying
to create other possible sample distributions which could have come
for your original population based on the sample you have.


how to make new columns using another column in pandas?

Here is an example (I just through it together hastily). You can probably do this a little bit smoother.

import pandas as pd
import numpy as np
from random import shuffle

a = np.repeat(['a', 'b', 'c', 'd', 'e'], 6)
x = np.random.randn(30)
y = np.random.randn(30)
z = np.random.randn(30)

shuffle(a)

a = a.reshape(30, 1)
x = x.reshape(30, 1)
y = y.reshape(30, 1)
z = z.reshape(30, 1)

data = np.concatenate((a, x, y, z), axis=1)
df = pd.DataFrame(data, columns=['item', 'x', 'y', 'z'])

mapping = {'1': ['a', 'b'], '2': ['d'], '3': ['c', 'e']}

for k in mapping:
    df.loc[df[df['item'].mask(
        ~df.item.isin(mapping[k])).notnull()].index.tolist(), 'category'] = k


In Sepp Hochreiter's original paper on the LSTM   where he introduces the algorithm and method to the scientific community, he explains that the long term memory refers to the learned weights and the short term memory refers to the gated cell state values that change with each step through time t.

edit: quote from paper "Recurrent networks can in principle use their feedback connections to store representations of recent input events in the form of activations ("short-term memory", as opposed to "long-term memory embodied by slowly changing weights)"


I got your problem like this way:
You want to show labels on x and y axis on seaborn heatmap. So for that sns.heatmap() function has two parameter they are xticklabels for x-axis and yticklabels for y-axis labels.

follow below code snippet

import seaborn as sns # for data visualization
flight = sns.load_dataset('flights') # load flights datset from GitHub seaborn repository

# reshape flights dataeset in proper format to create seaborn heatmap
flights_df = flight.pivot('month', 'year', 'passengers')

sns.heatmap(flights_df)# create seaborn heatmap

Output >>>

Now, we are changing x and y axis labels using  xticklabels and yticklabels sns.heatmap() parameters.

x_axis_labels = [1,2,3,4,5,6,7,8,9,10,11,12] # labels for x-axis
y_axis_labels = [11,22,33,44,55,66,77,88,99,101,111,121] # labels for y-axis

# create seabvorn heatmap with required labels
sns.heatmap(flights_df, xticklabels=x_axis_labels, yticklabels=y_axis_labels)

Output >>>

for in depth explanation follow seaborn heatmap tutorial.


Clustering sequences of sentence embeddings

The approach taken for anyone coming across this has been to first cluster each web page / node before passing it into SGT. This means we can encode more information into the sequence prior to using SGT.


approach to classify text with natural language processing methods

This problem seems like a multi-class multi-label problem. The questioner seems to be comfortable in building a detailed ontology. These lead the author to propose the following approach. Please note that a detailed explanation of this can be found in an article here.

Steps to solve the problem:

Build a taxonomy file as a csv file as given below. Please note, the column headings should be identical to whats given below.

Put all your content in another csv file that looks like below. Please note, the column headings should be identical to whats given below.

In the following python code, please enter the path to content in the path to df and path to taxonomy in the path to df_tx. These steps are present near the comment import data for mapping. Add another path value for the output towards the end of the code.

Run the python code below. Please note that this code runs fine on Python 2.7 in Windows 10 machine. Please iron out any technical issues yourself as the author may not be of much help for such issues.

#Invoke Libraries
import pandas as pd
import numpy as np
import re

#import data for mapping
df = pd.read_csv("path to content csv");
df_tx = pd.read_csv("path to taxonomy csv");

#Build functions
#function that identifies taxonomy words ending with (*) and treats it as a wild character
def asterix_handler(asterixw, lookupw):
    mtch = "F"
    for word in asterixw:
        for lword in lookupw:
            if(word[-1:]=="*"):
                if(bool(re.search("^"+ word[:-1],lword))==True):
                    mtch = "T"
                    break
    return(mtch)

#function that removes all punctuations. helps in creation of set of words
def remov_punct(withpunct):
    punctuations = '''!()-[]{};:'"\,<>./?@#$%^&*_~'''
    without_punct = ""
    char = 'nan'
    for char in withpunct:
        if char not in punctuations:
            without_punct = without_punct + char
    return(without_punct)

#function to remove just the quotes(""). This is for the taxonomy
def remov_quote(withquote):
    quote = '"'
    without_quote = ""
    char = 'nan'
    for char in withquote:
        if char not in quote:
            without_quote = without_quote + char
    return(without_quote)

#split each document by sentences and append one below the other for sentence level categorization and sentiment mapping
sentence_data = pd.DataFrame(columns=['slno','text'])
for d in range(len(df)):
    doc = (df.iloc[d,1].split('.'))
    for s in ((doc)):
        temp = {'slno': [df['slno'][d]], 'text': [s]}
        sentence_data =  pd.concat([sentence_data,pd.DataFrame(temp)])
        temp = ""

#drop empty text rows and export data
sentence_data['text'].replace('',np.nan,inplace=True);
sentence_data.dropna(subset=['text'], inplace=True);

data = sentence_data
cat2list = list(set(df_tx['Category2']))
data['Category'] = 0
mapped_data = pd.DataFrame(columns = ['slno','text','Category']);
temp=pd.DataFrame()

for k in range(len(data)):
    comment = remov_punct(data.iloc[k,1])
    data_words = [str(x.strip()).lower() for x in str(comment).split()]
    data_words = filter(None, data_words)
    output = []

    for l in range(len(df_tx)):
        key_flag = False
        and_flag = False
        not_flag = False
        if (str(df_tx['Keywords'][l])!='nan'):
            kw_clean = (remov_quote(df_tx['Keywords'][l]))
        if (str(df_tx['AndWords'][l])!='nan'):
            aw_clean = (remov_quote(df_tx['AndWords'][l]))
        else:
            aw_clean = df_tx['AndWords'][l]
        if (str(df_tx['NotWords'][l])!='nan'):
            nw_clean = remov_quote(df_tx['NotWords'][l])
        else:
            nw_clean = df_tx['NotWords'][l]
        Key_words = 'nan'
        and_words = 'nan'
        and_words2 = 'nan'
        not_words = 'nan'
        not_words2 = 'nan'

        if(str(kw_clean)!='nan'):
            key_words = [str(x.strip()).lower() for x in kw_clean.split(',')]
            key_words2 = set(w.lower() for w in key_words)

        if(str(aw_clean)!='nan'):
            and_words = [str(x.strip()).lower() for x in aw_clean.split(',')]
            and_words2 = set(w.lower() for w in and_words)

        if(str(nw_clean)!= 'nan'):
            not_words = [str(x.strip()).lower() for x in nw_clean.split(',')]
            not_words2 = set(w.lower() for w in not_words)

        if(str(kw_clean) == 'nan'):
            key_flag = False
        else:
            if set(data_words) & key_words2:
                key_flag = True
            elif(bool(re.search('"',df_tx['Keywords'][l]))==True and quote_handler(key_words, comment) == 'T'):
                key_flag = True
            elif(asterix_handler(key_words2, data_words)=='T'):
                    key_flag = True

        if(str(aw_clean)=='nan'):
            and_flag = True
        else:
            if set(data_words) & and_words2:
                and_flag = True
            elif(bool(re.search('"',df_tx['AndWords'][l]))==True and quote_handler(and_words, comment) == 'T'):
                and_flag = True
            elif(asterix_handler(and_words2, data_words)=='T'):
                and_flag = True

        if(str(nw_clean) == 'nan'):
            not_flag = False
        else:
            if set(data_words) & not_words2:
                not_flag = True
            elif(bool(re.search('"',df_tx['NotWords'][l]))==True and quote_handler(not_words, comment) == 'T'):
                not_flag = True
            elif(asterix_handler(not_words2, data_words)=='T'):
                not_flag = True

        if(key_flag == True and and_flag == True and not_flag == False):
            output.append(str(df_tx['Category2'][l]))
            temp = {'slno': [data.iloc[k,0]], 'text': [data.iloc[k,1].strip()], 'Category': [df_tx['Category2'][l]]}
            mapped_data = pd.concat([mapped_data,pd.DataFrame(temp)], sort = False)

#output mapped data
mapped_data = mapped_data[['slno', 'text', 'Category']]

mapped_data.to_csv("Path here/mapped_data.csv",index = False)

Final output looks like this:


Not quite sklearn but have you tried xgboost?

The XGBRegressor in xgboost accepts many different objective functions including poisson count:poisson for count data.

It also plays nicely with sklearn so can be used with grid search, pipelines etc.


You just confused the actual and predicted. Every row represents actual values of distinct elements in your array and columns represent predicted values of them. That is,

First row: There are 2 ants, and 2 samples are predicted as ant.
Seconds row: There are 1 bird and 1 sample is predicted as cat.
Third row: There are 3 cats, 1 sample is predicted as ant, 2 samples are predicted as cat.


Insignificant input variable but high R2

This can happen, but only with very small amounts of data. An example (in R) with three data points:

x <- c(0,1,2)
y <- c(-0.1,0.7,1.2)
summary(lm(y ~ x))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.0500     0.1118  -0.447   0.7323
x             0.6500     0.0866   7.506   0.0843 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1225 on 1 degrees of freedom
Multiple R-squared:  0.9826,    Adjusted R-squared:  0.9651
F-statistic: 56.33 on 1 and 1 DF,  p-value: 0.08432

Notice the R-squared is very high, but the x parameter is not 95%-significant.

With lots of data, one cannot have a very high R-squared parameter without the corresponding parameter being significant.


How to explain the connection between the input layer and H1 of this CNN Architecture?

Yes, the spatial dimensions (height and width) are reduced: the input is 16x16, H1 is 8x8 and H2 is 4x4.
Also see the first paragraph in the architecture section:

Source
In modern terms you would say that they use a stride of 2. Which reduces the spatial dimensions accordingly.
EDIT (based on your comment)
The formula for the spatial output dimension $O$ of a (square shaped) convolutional layer is the following:
$$O = \frac{I - K + 2P}S + 1$$ with $I$ being the input size, $K$ being the kernel size, $P$ the padding and $S$ the stride. Now you might think that in your example $O = \frac{16 - 5 + 2*2}2 + 1 = 8.5$ (assuming $P=2$)
But take a closer look at how it actually plays out when the 5x5 kernel of layer H1 scans the 16x16 input image with a stride of 2:

As you can see from the light grey area the required and effective padding is actually not 2 on all sides. Instead for the width or height respectively it is 2 on one side and 1 on the other side, i.e. on average $(2+1)/2=1.5$.
And if you plug that into the equation to calculate the output size it gives: $O = \frac{16 - 5 + 2*1.5}2 + 1 = 8$. Accordingly the convolutional layer H1 will have spatial dimensions of 8x8.


Temporal Difference Learning Getting Stuck

Sounds like an interesting application. To debug RL applications, I like to perform rollouts. First identify a state where the current policy appears to be clearly wrong; (maybe sample some states and look at them individually). Then switch off the learning and run a sample from that state for both the current RL policy and the action you think is correct. This should give you the true action values. Hopefully, that will give you a hint as to what's going wrong. If it's an exploration problem, then the current policy won't be trying any actions similar to the action you think is correct. If the form of the value function is wrong, then it won't fit the values coming from your samples.


Is there any conceptual relationship between 'kernel' in SVM and 'kernel' in convolution neural net?

There is no direct relationship between these two concepts. However we can find some indirect ones.

According to Merriam Webster,

  kernel means a central or essential part

which hints why they are called "kernel". Specifically, deciding "how to measure point-point similarity (a.k.a. kernel function)" is the central part of kernel methods, and deciding "what array, matrix, or tensor (a.k.a. kernel matrix) to be convoluted with a data point" is the central part of convolutional neural networks.

A kernel function receives two data points, implicitly maps them into a higher (possibly infinite) dimension, then calculates their inner product.

A kernel matrix (or array, or tensor) is convoluted with one data point to map the data point explicitly into an often lower dimension. Here, we are ignoring a subtle difference between filter and kernel (a filter is composed of one kernel per channel).

Therefore, these two concepts are indirectly related based on mapping to a new representation. However,

Kernel functions map implicitly, but kernel matrices map explicitly,
Kernel functions cannot be stacked over each other (shallow representation), but kernel matrices can be since the input and output (explicit representations) has the same structure (deep representation),
The non-linearity of map is integrated into kernel functions, but for kernel matrices, we should apply a non-linear activation function after the (input, kernel) convolution to reach a similar non-linearity,
Implicit representations cannot be learned for kernel functions, a specific function implies a specific representation. However, for kernel matrices, representations can be learned by adjusting (learning) the weights of kernels, and can also be enriched by stacking kernels over each other.


Once a predictive model is in production, how it can be evaluated?

This is in fact a very good question.
The answer is simple, but depends on the case.
In general, what we do after pushing a model to production we apply an audit process. Let me explain: in reality machine learning models that are being pushed to production are pushed to replace another process (e.g, manual process- this is the case of automation). At the beginning everything predicted by machine learning model are audited through another process (e.g, manual), we call this stage the pilot stage. By comparing the model performance to the manual process we establish the quality of the model. Once we are happy, we start reducing the audit percentage from 100% to 5% or so ( there is some math behind what should be the audit percentage). This audit will never go away and will always be used to measure the performance of the model and to establish ground truth data for new samples that can be added to the training set.
In fact, training models in theory is something and using them in production is something else. It is really a complex process. Just to mention: we also like to implement some protection mechanisms to protect the model. For example data drift detection, uncertainty detection and so on.


The best method to find themes in a collection of documents is topic modeling. Topic modeling finds the hidden (aka, latent) themes beyond just keyword counts.

There are many approaches to topic modeling. Latent Dirichlet allocation (LDA) is a standard topic modeling approach. LDA is a probabilistic graphical model that assumes that each document is a mixture of a small number of topics and that each word's creation is attributable to one of the document's topics. The number of topics is a selected hyperparameter.


Well, Unless your goal is to build a neural net to solve the problem. This can be done in a much simpler way, Like in case of country name you can just check against a list of country names, and so on.
At best some NLP could give you what you want. A neural net solution might be a little overkill.

If a neural net is compulsory, Then I think You could get a better answer if some details were specified. Are you looking for a fixed set of fields, what kind of text content do the pdfs contain etc.

Also just in case, if you were thinking a neural net will give you a json as output (just in case if you were thinking that). That will not be the case. you would have to convert it to json from the neural nets output, but that conversion stuff is very trivial, so i should not even be talking about that.

I know i have not answered your question. But i hope you got some direction.


How should i build a Knowledge Graph for a custom dataset?

You can build a Knowledge Graph from existing data sources too using
RML.
RML stands for RDF Mapping Language and allows you to transform heterogeneous
data sources into RDF and extends R2RML.
I created an example with JSON data as a data source, but other formats are
possible such as CSV, XML, relational databases, etc.

RML (RDF Mapping Language)

Turning your existing data sources into RDF goes as followed with RML:

Write RML mapping rules which instruct an RML processor how the data should
be transformed into RDF.
You can find many example in the RML documentation how
to write your mapping rules for your data sources.

RML rules consist of Triples Maps
which on its own turn have the following parts:

Logical Source: How the data source
should be accessed.
Subject Maps: How a subject of a triple
should be created.
Predicate-Object Maps: How a predicate
and its object should be created.

Logical Source

rml:logicalSource [
    rml:source "people.json" ;
    rml:referenceFormulation ql:JSONPath ;
    rml:iterator "$.people.[*]" ;
] ;

The data source people.json is accessed using JSONPath expressions defined as
$.people.[*].
The expression allows the RML processor to iterate over the JSON data.

Subject Maps

rr:subjectMap [
    rr:template "http://ex.com/Person/{firstname}_{lastname}" ;
    rr:class foaf:Person ;
] ;

Every subject created by this SubjectMap will look like
http://ex.com/Person/{firstname}_{lastname} where firstname and lastname
are replaced by the corresponding JSON values during the execution of an RML
processor. The subject has the class foaf:Person.

Predicate-Object Maps

rr:predicateObjectMap [
        rr:predicate foaf:givenName ;
        rr:objectMap [
            rml:reference "firstname" ;
        ]
    ] ;

This map generates a predicate foaf:givenName where the object will receive
the JSON value firstname during the mapping process.

Execute your mapping rules with an RML processor.
An example of an RML processor is the
RML Mapper or the
RML Streamer.
Other RML processors can be used too, if they comply with the
RML specification.

An RML processor will generate the following triples from the JSON data based on
the mapping rules shown previously:

<http://ex.com/Person/John_Doe> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person>.
<http://ex.com/Person/John_Doe> <http://xmlns.com/foaf/0.1/givenName> "John".
<http://ex.com/Person/John_Doe> <http://xmlns.com/foaf/0.1/familyName> "Doe".
<http://ex.com/Person/Jane_Smith> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person>.
<http://ex.com/Person/Jane_Smith> <http://xmlns.com/foaf/0.1/givenName> "Jane".
<http://ex.com/Person/Jane_Smith> <http://xmlns.com/foaf/0.1/familyName> "Smith".
<http://ex.com/Person/Sarah_Bladinck> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://xmlns.com/foaf/0.1/Person>.
<http://ex.com/Person/Sarah_Bladinck> <http://xmlns.com/foaf/0.1/givenName> "Sarah".
<http://ex.com/Person/Sarah_Bladinck> <http://xmlns.com/foaf/0.1/familyName> "Bladinck".

Full example

I created a small demo to create a FOAF Person with a first and last name:

RML Mapping rules

@base <http://example.com> .
@prefix rml: <http://semweb.mmlab.be/ns/rml#> .
@prefix rr: <http://www.w3.org/ns/r2rml#> .
@prefix ql: <http://semweb.mmlab.be/ns/ql#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .

<#PersonMapping>
    a rr:TriplesMap ;
    rml:logicalSource [
        rml:source "people.json" ;
        rml:referenceFormulation ql:JSONPath ;
        rml:iterator "$.people.[*]" ;
    ] ;

    rr:subjectMap [
        rr:template "http://ex.com/Person/{firstname}_{lastname}" ;
        rr:class foaf:Person ;
    ] ;

    rr:predicateObjectMap [
        rr:predicate foaf:givenName ;
        rr:objectMap [
            rml:reference "firstname" ;
        ]
    ] ;

    rr:predicateObjectMap [
        rr:predicate foaf:familyName ;
        rr:objectMap [
            rml:reference "lastname" ;
        ]
    ] .

JSON data

{
    "people": [
        {
            "firstname": "John",
            "lastname": "Doe"
        },
        {
            "firstname": "Jane",
            "lastname": "Smith"
        },
        {
            "firstname": "Sarah",
            "lastname": "Bladinck"
        }
    ]
}

Note: I contribute to RML and its technologies.


Principle Component Analysis on multiple functions

Variations of PCA can still be applicable.

If the data is not linear, use nonlinear PCA.

It is not an issue there a multiple "y"s for every "x". PCA is unsupervised, there is no notion of targets. In PCA, there are only dimensions.

Typically, the dimensions are standardized so each dimension can be weighted independent of scale.

One caveat - PCA is domain agnostic. Given that you have physical land data, you can use domain-specific methods such as hierarchical spatial indexing.


MNIST has 28x28 pixel greyscale images, so there are $28\times28 = 784$ features per image.

CIFAR10 contains colour images that are 32x32 pixels. At first glance it may seem like there are only $32\times32 = 1024$ features per image, but there are red, blue and green channels in the image, which means each image is actually $3\times32\times32 = 3072$ features.

Usually, these channels are arranged as three separate 32x32 pixel images, rather than a single 32x32 image with RGB pixels. In keras, you can load the CIFAR10 dataset like so:

from keras.datasets import cifar10

(x_train, y_train), (x_test, y_test) = cifar10.load_data()

However, this will load the train and test dataset in the shape (num_samples, 3, 32, 32). In order to input these into an MLP, we need to flatten the channels and pixel arrays to form an array of shape (num_samples, 3072), just like with MNIST. We can do this in python like so:

x_train = x_train.reshape(-1, 3072)
x_test = x_test.reshape(-1, 3072)


How to train neural networks with large sized data sets?

Here are some of the things that influence your training speed:

Number of weights in your network
Speed of your CPU
Package you are using (mostly engine it is working on, in PyLearn this is Theano)
If all your data fits in memory or you are reading from disk in between batches

With regards to network design the only thing you can really do is make the network more shallow to reduce the number of weights. To reduce the number of epochs there might be other options like adding residual connections but that will not decrease the training time of 1 epoch.

Without more information it is unclear where the bottleneck is, but 20 hours for one epoch seems a bit high. The easiest and biggest improvement you will be able to get is to use a good GPU, which should be possible using pylearn since it is built on top of Theano.


Does it makes sense to apply feature scaling on timestamp

For time series analysis.  Yes

But, turning data into a computable object for using in the ML computation?  No

Using the data as a feature.  Then, Yes

I would give you a general time series example:

Consider the number of days in months. The irregularity causes friction while analyzing the model.

Consider this:

So, such type of transformations would be helpful which analyzing time series', which can help reduce friction in the models sensibly.
Link of the explanation


How to specify different columns stacked vertically within CSV using pandas?

You can use pd.read_csv funcation with perameters of nrows and skiprows to read two different data frame and then concat them horizontally
import pandas as pd
df = pd.read_csv('your csv file path',nrows=4)
df1 = pd.read_csv('your csv file path',skiprows=5)
df = pd.concat([df, df1], axis = 1, sort = False)
print(df)

__output__

   crit_rating      DPS   DPS-Error  agility     cDPS   cDPS-Error
0        -1250  4439.51     11.2203    -1250  1400.14      3.62795
1        -1225  4438.28     11.3488    -1225  1473.09      4.28675
2        -1200  4439.30     11.2504    -1200  1547.47      4.47641
3        -1175  4442.01     13.0366    -1175  1620.25      4.63525

same you can implement with a single line of code i.e.
df = pd.concat([pd.read_csv('your csv file path',nrows=4), pd.read_csv('your csv file path',skiprows=5)], axis = 1, sort = False)


Image Super-resolution Connecting Subimages

You're probably seeing those artifacts because your model doesn't see those pixels immediately outside your tile and so can't know how to "blend" things.  (I'm assuming your tiles have a stride equal to the input size)
A typical approach I've seen used (and used myself) is, at inference time, to keep only a central portion of each tile and then to have overlapping windows so you can "fill the space".
The more overlap (and less of the output you use) the less apparent the artifacts will be (or have been in my experience), but more computation is needed.
Here's what I mean in cartoon form:
Current situation(as I assume):
input tile1 #####----------
input tile2 -----#####-----
output tile1 ##########--------------------
output tile2 ----------##########----------
My suggestion:
Here, for the output # means the value there is copied to the output,
and * means that the model makes a prediction there but it's not used.
input tile1 #####----------
input tile2 ---#####-------
output tile1 **######**--------------------
output tile2 ------**######**--------------
If you go with this idea, you could even simplify the model so that it doesn't attempt to make predictions at locations you would ignore anyway.
And please comment if things need clarification.


How label smoothing and label flipping increases the performance of a machine learning model

Label flipping is a training technique where one selectively manipulates the labels in order to make the model more robust against label noise and associated attacks - the specifics depend a lot on the nature of the noise. Label flipping bears no benefit only under the assumption that all labels are (and will always be) correct and that no adversaries exist. In cases where noise tolerance is desirable, training with label flipping is beneficial.

Label smoothing is a regularization technique (and then some) aimed at improving model performance. Its effect takes place irrespective of label correctness.

From 1 Adversarial Perturbations of Deep Neural Networks, 2016:

  Without label smoothing, a softmax classifier is trained to make infinitely confident  predictions on the training set. This encourages the model to learn large weights and strong responses. When values are pushed outside the areas where training data concentrates, the model makes even more extreme predictions when extrapolating linearly. Label smoothing penalizes the model for making overly confident predictions on the training set, forcing it to learn either a more non-linear function or a linear function with smaller slope. Extrapolations by the label-smoothed model are consequently less extreme.

From Regularizing Neural Networks by Penalizing Confident Output Distributions, 2017:

  Confident predictions correspond to output distributions that have low entropy. A network is over-confident when it places all probability on a single class in the training set, which is often a symptom of overfitting. The confidence penalty constitutes a regularization term that prevents these peaked distributions, leading to better generalization.

As a result of label smoothing, the model becomes more robust in general. Its increased ability to deal with incorrect labels is just part of the overall improvement. However, one cannot claim that the effects of label smoothing are purely beneficial.

From When Does Label Smoothing Help?, 2019:

  Despite having a positive effect on generalization and calibration, label smoothing can hurt distillation. We explain this effect in terms of erasure of information. With label smoothing, the model is encouraged to treat each incorrect class as equally probable. With hard targets, less structure is enforced in later representations, enabling more logit variation across predicted class and/or across examples. This can be quantified by estimating mutual information between input example and output logit and, as we have shown, label smoothing reduces mutual information.


CNN Multi-class vs Binary Class Image Classification

I would train one classifier with 3 classes:

cat, dog, neither

Use cross entropy as your cost function.

There are many examples of this online. If you Google MNIST neural network you'll find many simple examples. Or you can take one of Google's trained networks and retrain the last layer.

Here is a convolutional being trained on the Cifar set of images.

Using a pre-trained network to tell two house cats apart.


Large mean squared error in sklearn regressors

that's possibly due to poor parameter tuning.
Try reducing C for SVR and increasing n_estimators for RFR.

A nice approach is to gridsearch through the parameter, and plot the metric result.

Another thing that might help is to normalize the parameters (sklearn.preprocessing.StandardScaler) and to remove the skew from the target (usually log transform or 1/target transform works better)


No, that is not what is happening, the definition of the Softmax classifier is: $$z \in \mathbb{R}^K; \sigma(z)_j = \frac{e^{z_j}}{\sum_{k = 1}^{K} e^{z_k}} $$ Let's talk first about what the softmax function aims to do, but before we can understand that, we need to understand the aim of classification in general. Classifiers are functions which take a vector in $\mathbb{R^n}$ and output one of $k$ classes. The softmax function takes in a vector $z$ and outputs a value in $\mathbb{R^+}$ whose values sum to one.

Now let's get to your question, if I were to just take $\log(Wx + b)$. It is actually almost exactly that, we just take the exponent so that our values range from $0$ to $1$. However, these values are not normalized and in order to normalize it we need to divide by the sum and we take the exponents to maintain a positive value.


finding start and end time of each session id in r

Assuming you have access to dplyr package, you can do the following.

data %>% group_by(session_id) %>% summarise(start_time = min(timestamp), end_time = max(timestamp), unique_items=length(unique(item_id)))


The final range of emotion is completely arbitrary. No matter the interval [a, b], you can adjust the emotions to fit inside. [-100, 100] is perfectly reasonable and is common. An example of use is from GDELT, which provides this interval for average tone of news documents.

Asking if equally distancing the emotions is statistically correct does not make sense. This entirely depends on your use case and opinions. Also, there is absolutely no reason why you can't use negative numbers in a weighted average.

If you mentioned what you are doing and how you are evaluating emotion, there might be more to say.


How to analyse player and enemy position for data analysis

Either do unsupervised learning with something like k-means clustering or DBScan where you attempt to segment students into groups and see if you can discern any insights based on the cluster generated or pick a threshold for certain categories, create a class column and label each student, and do a classification model.


If your old data is representatively of the underlying population and there is no radical shift of the underlying true model over time. Than you don't need to update you model as new data comes in. You might want to retrain you model with the new dataset periodically, but you don't need to do it in real time on your production machine.

If the underlying true does change radically overtime and you don't have any existing data to captures those changes then doesn't matter what algorithm you use, it's not going to work. The information is simply not there.


Is there a scientific method for grouping continuous numbers for this problem

Because you care about the ordering you can represent your points in two dimensions (where the x coordinate is just rescaled sequence number) and apply K-means.

import numpy as np

numbers = np.array([0.867, 0.808, 0.740, 0.746, 0.674,
                    0.669, 0.648, 0.722, 0.781, 0.612,
                    0.575, 0.566, 0.500, 0.555, 0.818,
                    0.800, 0.500, 0.500, 0.666])

#you can experiment with other step sizes
step_size = (numbers.max() - numbers.min())/len(numbers)
numbers2d = np.array([(step_size * i,x) for (i,x) in enumerate(numbers - 0.8)])

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=6)
kmeans.fit(numbers2d) #random init - possibly different result each time

print(kmeans.labels_)

import matplotlib.pyplot as plt

for cluster in set(kmeans.labels_):
    x = np.where(kmeans.labels_==cluster)[0]
    y = numbers[x]
    x = numbers2d[:,0][x]
    plt.scatter(x,y)

plt.show()

Output:

[3 3 3 3 1 1 1 1 1 4 4 4 4 4 5 5 2 2 0]


What is the best way Reinforcement learning, RNN or others to predict the best action we have to take to maximize sales?

I don't know if reinforcement learning is right for this problem.  A reinforcement learning model needs a simulated "game" environment where it can interact and select actions initially at random.  But real customer reactions are hard to simulate, and you don't want the model randomly sending out spam e-mails before it's trained.

There are other ways to train a model to recommend an action based on existing data.  If you want to keep it simple, you can perform a regression to estimate each action's impact on likelihood of a sale.  If you have more data and want to get more complex, perhaps look into recommender systems (Netflix, Amazon, etc.)


How to get DATE column from dataframe in pandas without using CSV

You need to reset index using df.reset_index().This will make your Dates attribute/column and not an index as it is now.
import pandas as pd
import pandas_datareader.data as web
import datetime as dt

start = dt.datetime(2019, 2, 1)
end = dt.datetime.today().strftime('%Y-%m-%d')
slug = "tsla"
df = web.DataReader(slug, 'yahoo', start, end)
modified = df.reset_index()
print(modified['Date'])


Adding dense layer to CNN causes to stop learning

Having dropout after convolution is rather unusual. Try to put it between the dense layers.

Explanation:

Dropout forces neuron to work with a random subset of the preceding neurons' activations. By doing so it breaks the co-adaptation of neurons and improves the generalization of the network. However, for convolutional layer co-adaptation is not an issue, as neighboring pixels of a feature map share their weights and guarantee to have different activations given different inputs in their receptive fields. It may only occur if two layers will learn the same kernel weights, but even then the problem is not so significant. Little number of weights in a convolution layer is a pledge of good generalization.

Here is a short article showing what happens when you use batchnorn/dropout between convolutions: Don’t Use Dropout in Convolutional Networks


Welcome to the DataScience Stack exchange.

Since your target variable has many categories your problem is Multi class classification. On how to implement and find a solution for this problem you can find on google. The following 2 links are for reference:

Link-1
Link-2

Here I'm not giving much information on which algorithm to use. If you need any additional help you can raise one more question so that we can help you better as we don't now much about your data and business problem.

So, to answer your question 2, you can convert them into number it is one line code,I've added the link for your reference.

Please let us know if you need any additional information.


Finding an appropriate binary classification algorithm for time series data intervals

You could do logistic regression combined with an additional moving average/ clustering step.
Combine your data into training rest and active data into a single array X, and have an additional array y which would represent the labels e.g. 0 - active / 1 - resting for each row in your training dataset.
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(random_state=0).fit(X, y)

Then  predict for each step whether it is resting or active.
The predictions of the test set would look something like:
[0,1,1,1,0,1,1,0,0,0,0,1,0,0,0]

The additional step would be to find the groups. You could do this with a moving window like "if 5 active states in a window of 7, label it as active". You could also solve that final step with clustering, e.g. "if 0 is surrounded by 1s it is 0", or find local clusters of binary values.


Regression Model Evalation with unbalanced target; alternative to R-squared?

Looks like your data are severely unbalanced if they are almost all zeros. Try data balancing or some form of cross-validation. To your original point: there is not a simple R^2 metric fix.


The type of neural networks you are looking for to predicting timeseries are called recurrent networks; the previous neuron states in the network affect the new neuron states. Examples of such recurrent networks are LSTM, GRU and NARX.

If you have the data from t0 to t10, and you want to predict t11, then you need to input the values from t0- t10 one by one into the network. You have to train the network so that the output of tn is tn+1. So after you have inputted t10, the output will be the values predicted for t11.

Real life example: I want a network to keep on decreasing it's output until it has reached 1, and then make it output 0 to start over again:

in: 0.0, out: 0.2
in: 0.2, out: 0.4
in: 0.4, out: 0.6
in: 0.6, out: 0.8
in: 0.8, out: 1.0
in: 1.0, out: 0.0

And that is what your training data should look like

If it is still to complicated, I recommend you to play around with this front-end neural network library for the browser: Neataptic. It is very easy to fiddle around with. Example


Cannot see what the "notation abuse" is, mentioned by author of book

The mathematical expression is completely legit. The abuse is in the fact that the function $p$, which is defined first time in equation 3.2, which:

  The function $p: S$ x $R$ x $S$ x $A \rightarrow [0,1]$. is an ordinary
  deterministic function of four arguments...

is re-defined slightly differently just two lines after this definition (equation 3.4), as a three-argument function $p: S$ x $S$ x $A \rightarrow [0,1]$.

If they used $p$ to represent the regular probability measure, there would be no abuse. In the authors' notation, $p$ is a deterministic function, while the regular probability function is denoted as $Pr$; and keeping the same name for slightly different functions, is where the "innocent" notation abuse comes from.


I think that you just need:
feature_importances = rf_gridsearch.best_estimator_.feature_importances_

This provides the feature importance for all the attributes in your dataset. For more information on this as well as other options, you may also refer to the Scikit-learn official documentation.


How can the labels of AgglomerativeClustering be re-computed?

One option to speedup computation with different thresholds is caching the results using the memory option.
Something like this:
from sklearn.cluster import AgglomerativeClustering

clustering = AgglomerativeClustering(compute_full_tree=True,
                                     distance_threshold=1.0,
                                     memory='mycachedir',
                                     n_clusters=None,)


Is this scheme correct for logistic regression with stochastic gradient descent

Hard to say without more detail, but isn't your update wrong? you need to subtract rather than add the gradient. Unless alpha is negative, this is wrong.


I realize this question was asked more than a year ago, but I think one possibility is to use the bias-variance decomposition to calculate a lower bound on the error rate.

Essentially, the error rate is written as the sum of three terms, the bias, the variance, and the irreducible error. One good source for learning about these terms is An Introduction to Statistical Learning.

Assume that the true function ($f(x)$) lies within the family of functions that our machine learning model is capable of fitting, and take the limit as the amount of training data we have goes to infinity. Then, if our machine learning model has a finite number of parameters, both the bias and the variance will be zero. So, the actual error will simply be equal to the irreducible error.

As an example, suppose our true data is linear with Gaussian noise: $y \sim N(a + bx, \sigma^2)$. One of the the optimal estimators is obviously linear regression, $\hat{y} = \hat{a} + \hat{b}x$, and, as we add more training examples, the estimated coefficients $\hat{a}$ and $\hat{b}$ will approach $a$ and $b$, respectively. So, the best error (assuming squared loss) we could hope to achieve would be equal $\sigma^2$, the inherent error/irreducible noise associated with the data generation itself

In practice, computing the irreducible error is difficult (impossible?), since it requires knowledge of the true process for generating the data. But, this critique is also applicable to the Bayes error, since that requires knowledge of the true class probabilities.


Having trouble figuring out how loss was calculated for SQuAD task in BERT paper

From your description it sounds like for every position $i$ in the input text the model predicts $$p_S(i) = \mathbb P(\text{correct start position is } i)$$ and $$p_E(i) = \mathbb P(\text{correct end position is } i).$$
Now let $\hat s = \arg\max_i p_S(i)$ and $\hat e = \arg\max_i p_E(i)$ be the most probable start and end positions (according to the model).

Then by "maximum scoring span is used as the prediction" they just mean that they output $(\hat e, \hat s)$ when predicting.

Then “The training objective is the loglikelihood of the correct start and end positions” means that if the correct start and end positions are $s^*$ and $e^*$, they try to maximize the predicted probability of $s^*$ and $e^*$. If the start and end positions are independent then this is equal to $p_S(s^*) p_E(e^*)$ and taking the negative log the loss becomes
$$L(e^*, s^*) = -\log p_S(s^*) -\log p_E(e^*).$$


It depends on how did you collect these MRI images from hospital and how do you want to use these MRI images.

If you want to use these data in your deep learning model for segmentation or detection, it is possible to skip preprocessing.
If you want to apply a classical data science approach, you can do some processing like feature extraction.
Lastly, from another perspective, if you want to use also patient history with images, preprocessing can be necessary.


Validation of a model generated by automated deep learning system

I would suggest Option E: Efron-Gong "optimism" bootstrap. It is similar to Cross-validation in spirit, but requires orders of magnitude less repetitions as CV and it exploits the whole dataset. The procedure is described in section 6 of this reference.


Imagine the matrix inputs as a 2D table. You have 100 rows and 32 columns. Then the for loop acts as an iterator which will return values along the first dimensional axis. This dimension thus disappears and returns the remaining dimensions. When there is a single dimension the default in Python is $(n,)$.

A matrix $(100, 32)$ iterates through $(32,)$

A matrix $(100,28,28)$ iterated through $(28,28)$

A matrix $(100,2,2,2)$ iterated through $(2,2,2)$


Finding parameters with extreme values (classification with scikit-learn)

If by "parameters" you mean features (called "Data Fields" at Kaggle), then, yes, you can log-scale those. To visualize them you can just use histograms.
To do it for all features in python, for example, you can put your data in pandas DataFrame (let us call it "data") and then use data.hist()
This has nothing to do with the regularization in any model.

If by "parameters" you mean the coefficients obtained after fitting the logistic regression, then one uses regularization. This has, however, is not directly related to log-transform. How you list/visualize your coefficients depends on the programming tool you use for logistic regression (or other model)


Forget that you are working with a neural network for a moment. Hopefully you are also taking into account time. If you were performing a ordinary regression and time was a component among others, you would have to apply an extrapolation penalty to your confidence intervals to penalize your model for deviated from the observed range.

Another possibility is that you have an intervention in your dataset. Meaning, it is possible that something occurred and what you need to truly do is a variance test between the training and the sample data.

Traditional regression handles minimization of variance around the mean, but there is something called median regression as well, which is design for this particular type of issue, when there are problems with dispersion in the model.


Using random forest to select important variables & then putting into logistic regression?

There are many factors which are underlying the 'importaces' of featues obtained from random forest.
For instance, features with more number of categories (unique values if its a numerical feature) would be more likely to get find splits; making it more important feature.
Having all features with same number of categories is not a common scenario.
So, even though the features identified are likey to be the best predictors (in logistic regressions etc.), one need to take the inherent biases in the randome forest algorithm into consideration while utilizing the 'important' features. Therefore whether it would make sense to use the important features from random forest model into logistic regression would depend on if any and how much the importances are biased.
Source for the information and further useful details on the biases in estimation of feature importances by random forest algorithm: Bias in random forest variable importance measures: Illustrations, sources and a solution Bias in random forest variable importance measures: Illustrations, sources and a solution


File format where column names are repeated on each row

Using Miller (https://github.com/johnkerl/miller)

mlr --ocsv unsparsify input.txt

you will have this CSV

col1,col2,col3
datac1r1,datac2r1,datac3r1
datac1r2,datac2r2,datac3r2
datac1r3,datac2r3,datac3r3
datac1r4,datac2r4,datac3r4


What does it mean for an image to have a distribution?

First, we must understand that an image can be represented as the numerical values of its pixels. For example, a 256x256 grayscale image can be represented by a vector of 65536 integer values between 0 and 255.
Now, let's consider that we are talking about images of human faces, like the CelebA dataset, but in grayscale.
And finally, let's imagine that we sample a vector of 65536 components from a uniform distribution between 0 and 255, that is, from $x \sim U[0, 255]^{65536}$. If we interpret such a randomly sampled vector as an image, like the interpretation we described in the first paragraph, do you think that is it probable that we see a face? Well, probably not. Probably we would see just noise, like this:

Why do you know that we won't see a face? Well, because you know that a face is not composed of uniformly random pixels, but has some structure in it. In other words, because you intuitively know that the distribution of pixel values of a face image does not match $U[0, 255]^{65536}$, but has a different distribution.
As a summary, the distribution of an image domain is the distribution the pixel values follow. Note that this is applicable not only to grayscale images, but to color images expressed in the RGB space or any other numeric space like HSB.
P.S.: this is how an image of Gaussian noise looks like (it was generated with Imagemagick, with convert -size 256x256 xc:gray +noise Gaussian out.png):


Python make_scorer giving incorrect outputs for Root Mean Square Logarithmic Error

The make_scorer function takes a greater_is_better parameter (bool, default=True) which defines whether 'high is good' or 'low is good'. Changing its value to False might solve the issue you are having.


How to feed LSTM with different input array sizes?

The easiest way is to use Padding and Masking.

There are three general ways to handle variable-length sequences:

Padding and masking (which can be used for (3)),
Batch size = 1, and
Batch size > 1, with equi-length samples in each batch.

Padding and masking

In this approach, we pad the shorter sequences with a special value to be masked (skipped) later. For example, suppose each timestamp has dimension 2, and -10 is the special value, then

X = [

  [[1,    1.1],
   [0.9, 0.95]],  # sequence 1 (2 timestamps)

  [[2,    2.2],
   [1.9, 1.95],
   [1.8, 1.85]],  # sequence 2 (3 timestamps)

]

will be converted to

X2 = [

  [[1,    1.1],
   [0.9, 0.95],
   [-10, -10]], # padded sequence 1 (3 timestamps)

  [[2,    2.2],
   [1.9, 1.95],
   [1.8, 1.85]], # sequence 2 (3 timestamps)
]

This way, all sequences would have the same length. Then, we use a Masking layer that skips those special timestamps like they don't exist. A complete example is given at the end.

For cases (2) and (3) you need to set the seq_len of LSTM to None, e.g.

model.add(LSTM(units, input_shape=(None, dimension)))

this way LSTM accepts batches with different lengths; although samples inside each batch must be the same length. Then, you need to feed a custom batch generator to model.fit_generator (instead of model.fit).

I have provided a complete example for simple case (2) (batch size = 1) at the end. Based on this example and the link, you should be able to build a generator for case (3) (batch size > 1). Specifically, we either (a) return batch_size sequences with the same length, or (b) select sequences with almost the same length, and pad the shorter ones the same as case (1), and use a Masking layer before LSTM layer to ignore the padded timestamps, e.g.

model.add(Masking(mask_value=special_value, input_shape=(None, dimension)))
model.add(LSTM(lstm_units))

where first dimension of input_shape in Masking is again None to allow batches with different lengths.

Here is the code for cases (1) and (2):

from keras import Sequential
from keras.utils import Sequence
from keras.layers import LSTM, Dense, Masking
import numpy as np

class MyBatchGenerator(Sequence):
    'Generates data for Keras'
    def __init__(self, X, y, batch_size=1, shuffle=True):
        'Initialization'
        self.X = X
        self.y = y
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(len(self.y)/self.batch_size))

    def __getitem__(self, index):
        return self.__data_generation(index)

    def on_epoch_end(self):
        'Shuffles indexes after each epoch'
        self.indexes = np.arange(len(self.y))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, index):
        Xb = np.empty((self.batch_size, *X[index].shape))
        yb = np.empty((self.batch_size, *y[index].shape))
        # naively use the same sample over and over again
        for s in range(0, self.batch_size):
            Xb[s] = X[index]
            yb[s] = y[index]
        return Xb, yb

# Parameters
N = 1000
halfN = int(N/2)
dimension = 2
lstm_units = 3

# Data
np.random.seed(123)  # to generate the same numbers
# create sequence lengths between 1 to 10
seq_lens = np.random.randint(1, 10, halfN)
X_zero = np.array([np.random.normal(0, 1, size=(seq_len, dimension)) for seq_len in seq_lens])
y_zero = np.zeros((halfN, 1))
X_one = np.array([np.random.normal(1, 1, size=(seq_len, dimension)) for seq_len in seq_lens])
y_one = np.ones((halfN, 1))
p = np.random.permutation(N)  # to shuffle zero and one classes
X = np.concatenate((X_zero, X_one))[p]
y = np.concatenate((y_zero, y_one))[p]

# Batch = 1
model = Sequential()
model.add(LSTM(lstm_units, input_shape=(None, dimension)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())
model.fit_generator(MyBatchGenerator(X, y, batch_size=1), epochs=2)

# Padding and Masking
special_value = -10.0
max_seq_len = max(seq_lens)
Xpad = np.full((N, max_seq_len, dimension), fill_value=special_value)
for s, x in enumerate(X):
    seq_len = x.shape[0]
    Xpad[s, 0:seq_len, :] = x
model2 = Sequential()
model2.add(Masking(mask_value=special_value, input_shape=(max_seq_len, dimension)))
model2.add(LSTM(lstm_units))
model2.add(Dense(1, activation='sigmoid'))
model2.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model2.summary())
model2.fit(Xpad, y, epochs=50, batch_size=32)

Extra notes

Note that if we pad without masking, padded value will be regarded as actual value, thus, it becomes noise in data. For example, a padded temperature sequence [20, 21, 22, -10, -10] will be the same as a sensor report with two noisy (wrong) measurements at the end. Model may learn to ignore this noise completely or at least partially, but it is reasonable to clean the data first, i.e. use a mask.


I think there is some NaN values in your data set.

First time it is printing the random values which are generated for the weights.  But after first step, that means, after the first update of weights, they are becoming NaN. That means there is some NaN values or some missing data in the training data.

Check your training data. I am not clear how tensorflow handles the NaN or missing data.


I keep getting an error message in R while using twitteR

See this stackoverflow answer, it seems you can request tweets older than 7 days from the official Twitter API. See also the documentation for the searchTwitter function:

Note that there are pagination restrictions as well as other limits on
what can be searched, so it is always possible to not retrieve as many
tweets as was requested with the n argument.

Try changing the since and until dates to something within the last 7 days to see if that does work.


You can also solve this problem by the following ways;

import re
regex = re.compile("(emp-.+)_(id-.+)_(sc-.+)")
strings = ["humanresourc-emp-001_id-01_sc-01","itoperation-emp-002_id-02_sc-12","Generalsection-emp-003_id-03_sc-10"]
print([regex.findall(s)[0] for s in strings])


pandas - under a column, count the total number of a specific value, instead of using value_counts()

You have plenty of ways to do it. You won't see a big difference in performance. My suggestion is to use whatever feels more convenient for you or your team.

import pandas as pd
import numpy as np

#Let's create a dataframe with 10 million integers from 0 to 100
df = pd.DataFrame(np.random.randint(0,100,size=(10000000, 1)), columns=list('A'))

#And now count the value 5 with 4 different ways

%timeit df[df.A == 5].shape[0]
10 loops, best of 3: 25.4 ms per loop

%timeit len(df[df.A == 5])
10 loops, best of 3: 25.4 ms per loop

%timeit len(df[df.A == 5].index)
10 loops, best of 3: 25.6 ms per loop

%timeit df['A'].value_counts()[5]
10 loops, best of 3: 149 ms per loop

As you can see, only the last one takes more time to run.

EDIT: Addition to your comment, you could try this

df = data.groupby('a_1').get_group(a_2)['suffix']
len(df[df.suffix == 'a_3'])


GAN: Discriminator converges, generator learns almost nothing

This is one of the bigger problems with GANs, they are usually hard to train.

This is an active area of research and it does not seem like this problem has been solved. Without additional information or code, it is hard to help, but usually different loss functions are one way to mitigate the problem.


R: I need to understand the mechanism behind reading a subset of a file in R (e.g., via sqldf) or other data-science-centered programming language

There are multiple aspects to performance when it comes to reading a portion of the file. I am going to discuss the two most common one based on functions that are available in the base R.

What is R doing at the time of reading?

Establish a connection with the file.
Open the file
Read the data sequentially
Load the data in memory
Close the connection
dump the object into a variable if assigned, else render the output in some gui.

While it is doing all these steps, it is also trying to:

Understand how big the data is so that it can allocate appropriate memory
What are the data types or classes so that it can (again) allocate appropriate memory.

Sub bullet 2 is important here because unless R reads the entire dataset, it cannot determine the datatype. Providing this information alone at the time of the read makes processing much faster.

By reading a subset of the data, R makes it easier to interpret the datatype and then assigns it to the subsequent sets without worrying about what the actual datatype is.

Does R really loads all the data every time?

Functions that allow filter at the time of read, really target the third step of reading data sequentially.

Each row is compared against the filter criteria and the result stored as a binary logical index. This index is then used to determine, what remains when the connection closes. I could not find specific reference in the documentation if the discard happens immediately or post entire read, but I will assume that the R collects the garbage automatically as it reads with a filter criteria and makes the memory available to a subsequent line of data.

May be there are other experts who can throw more light on the real inner workings here and either validate or correct my assumptions, but hope my context helps.


Having Trouble Using Orange Data Mining in Linear Progression in Continuous Data

Orange needs either a one-line or three-line header. Yours is two-line, hence it will cause confusion. Either add an extra blank line below the second or remove the second line altogether (Orange should recognize these values correctly).


Help with backpropagation equations for a simple neural network with Sigmoid activation

The gradient is actually calculated w.r.t to the weights so that each neuron would have a different 'change in the weight' according to its previous weight. In other words, not all the 3 neurons in the hidden layer will be penalized with the same value because the initial weights at t=0 are different for these neurone.

It is well explained here: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

I hope this short answer helped!


How do I find all the drastic changes in the stock and fetch news for those days of the stock?

I would say that it probably has a great deal to do with what you are planning on doing with the results of your model. For example, if you are looking for news that corresponds to a move of the stock price by more than 3% on that day, you would only want to look at those days. Alternatively, if you are looking for 'abnormally large' movements in the stock price, you might consider taking the mean and standard deviation of your day to day movements and only considering those days that are more than two or three standard deviations from the mean.

In terms of how to get the news for a particular day, here is a python tutorial that may be useful:
https://www.w3resource.com/python-exercises/basic/python-basic-1-exercise-8.php


Why would GradientBoostClassifier do better than XGBoostClassifier?

GradientBoostClassifier is slower but more precise. In your case, it could be finding a better model without suffering from overfitting.
Here are some of the main differences you are looking for.
XGBClassifier was designed to be faster. However, XGBClassifier takes a few shortcuts in order to run faster. For example, to save time, XGBClassifier will use an approximation on the splits, not spend so much time on calculating and evaluating the best splits. Usually XGB results will be close and you won't see much difference, but your case could be an exception.
If you are still curious, you can try changing some parameters in the XGBClassifier model, such as the tree_method='exact' or you can modify the sketch_eps parameter in XGBClassifier to more closely match the GradientBoostClassifier results. Of course, this will slow down the XGBClassifier model as a result.
sketch_eps [default=0.03]
Only used for tree_method=approx.
This roughly translates into O(1 / sketch_eps) number of bins. Compared to directly select number of bins, this comes with theoretical guarantee with sketch accuracy.
Usually user does not have to tune this. But consider setting to a lower number for more accurate enumeration of split candidates.
range: (0, 1)


Comparison between addition and multiplication function in deep neural network?

The observation is very interesting you report, since concatenation and addition are practically the same. A nice explanation can be found in https://distill.pub/2018/feature-wise-transformations/ .


Advantages of Recurrent Neural Networks over basic Artificial Neural Networks

okey, I will try to explain them as easy as possible.

  Regardless of the sequence length, the learned model always has the same input size, because it is specified in terms of transition from one state to another state, rather then specified in terms of variable-length history of states.

I will use a simple example for simplification but it does not lead to losing generalisation. Suppose that your task is to add to binary numbers. Numbers are stored bitwise in memory and usually for applying any arithmetic operations, they should have a same size. What you need for adding two numbers is to learn how to add zeros and ones and when to output carray to the next step if you use sequential learning approaches, like RNNs. In this case, it is not really important that the two numbers do not have the same shape due to the fact that they both can be resized to the biggest size by adding zeros to their most significant bits. After resizing the inputs to have the same shape you have two options, to use MLPs or RNNs. If you use MLPs what the network learns is entirely different from what an RNN learns. The former does not learn the transition of carray at least as the way RNN learns. Another difference is that your MLP always will be restricted to the size which it was trained while the RNN model will be able to add two numbers with even more bits. To explain it why, for MLPs, all inputs which are connected to the hidden layers or maybe output layers, have weight. Consequently, increasing the number of inputs will lead to more weights which are not trained yet. You are not allowed to input a signal which its size is not equal to the input size of the MLPs. On the contrary, RNNs are exploited in a different way.

First, you should know RNNs better. Try to think of hidden layers of anRNN. They are like usual MLPs. Their difference is that for each node, the inputs come from the previous step's outputs and the current time's inputs. Bear in mind that the outcomes of the previous time step are not coming from previous neurons. The reason and I guess the main answer to your question is that each RNN is repeated for each input in time $t$. It means you have just an MLP which is used for all time step. Suppose you are at the middle of the calculations. Two inputs are $1$ and $1$. You RNN takes them and the carray value and outputs $1$ as the result of time step $t$ and outputs $1$ to the next step as carray. For the next time step the same RNN is again used and takes the inputs alongside the carry which is coming from the previous step's outputs and outputs the corresponding outputs and carries.

Due to the nature of RNNs which just take the inputs of time step $t$, they are capable of dealing with signals with different lengths. The reason is that The RNN is used for each time step. This behaviour is usually called unfolding the network.


How to calculate the silhouette coefficient?

a(i) : the average distance between 'i' and all other data within the same cluster (source)

b(i) :  the lowest average distance of 'i' to all points in any other cluster, of which 'i' is not a member (source)

So, from the question, a(i) will be 24 as point 'Pi' belongs to cluster A and b(i) will be 48 as it is the least average distance that 'Pi' has from any other cluster than A (to which it belongs).

So, as a(i) < b(i),  silhouette coefficient s(i) = 1 - 24/48 = 0.5


welcome to the community
I guess there's a confusion in your understanding of the kernel we use in case of rgb data. We normally use a kernel of equal number of channels as the input coming in (in this case as you mentioned it's RGB, so my number of channels for convolution operation would be 3). So instead of using a 3 X 3 Kernel, we use a 3 X 3 X 3 kernel. Weight matrix multiplication of kernel and image pixels happen channel-wise.
However, having said this, you can use a kernel of size 3 X 3 when input image is rgb, by specifying the stride as 1 in the third dimension. What this will do is convolute the kernel not only horizontally and vertically but also through the depth or specifically through the channels as well. I don't exactly know why one would like to do it.
Apart from this I guess the course or the video you are referring to might have specified '2D convolution on a 3D image'. That doesn't mean using a 2D kernel. And a 2D convolution on a 3D image uses a 3D kernel and after weight matrix multiplication you get a 2D image hence justifying the 2D convolution name.


What do the symbols in this image mean in relation to autoencoding? (which concepts do they represent and why?)

The question seems more about notation than autoencoders:
$ a,b = argmin_{a,b}f(a,b)$ simply means that $a$ and $b$ are chosen to minimize $f$.
w.r.t. autoencoders, the argmin statement is saying that we want the decoding of the encoded data $X$ to be as similar as possible to the original $X$.
To represent "decoding of the encoding", we can use function composition: $\psi\circ\phi X$ which means $\psi(\phi(X))$, hence the decoding of the encoding of $X$.
So overall, we are saying that we choose the encoding and decoding functions to minimize the delta between input $X$ and encoded/decoded $X$.

Where the question asks:

what does the arg min section and beyond mean?

The latter half "and beyond" should be more specific.


Assuming you want to classify the images (and not use bounding boxes to locate classes within each image), a common way it to create a target vector for each image, which holds the information regarding all classes and is what the model would eventually predict.

If you have a dataset with, say 5 classes, and your first example image contains classes 1 and 4, you would create your target vector for that image to be:

example_sample = ... # your image array
example_sample_y = [1, 0, 0, 1, 0]

This is a kind of one-hot encoding, as the vector has a placeholder for each of the 5 classes, but only a 1 when the class is present.

Have a look at this high-level walkthrough.

I think your other suggestion of training an image

You want to learn some kind of joint probability between the classes, and in my opinion, training one the same image with different outcomes (e.g. the sample image above twice, producing either a 1 or a 4) will not only be very inefficient during training, but will also be mathematically confusing. The same input can give 2 possible outputs! This implies your underlying function that maps images to classes is not well-defined. That isn't usually a good thing!


This all comes down to your objectives that come to mind which you could use:

We care about the entire website.  The new recommender system will boost total payments either directly or indirectly.  We do not want the recommender to hurt total payments.
We care only about the recommender, perhaps we believe
this is the future of our business and we want to eventually replace
the entire website.  It's critical the recommender directly drive payments.

Either way, I would recommend tracking recommender-associated payments and non-associated payments.  It's important to understand direct and ecosystem effects.  It's possible the recommender is exceptionally successful at driving clicks and finding users great games, but also finding them games which do not monetize successfully.  It's also possible the recommender is so awful that it scares people away from the website.  You will want to measure and track both direct and indirect effects, but how you make your decision will depend on the objective you are trying to achieve.


Reinforcement learning for continuous state and action space

As you say, policy gradient methods work better than value-based methods (like DQN) with continuous action spaces.

Your problem seems very similar to the MountainCarContinuous environment. If you check the leaderboard, you can see how the top score uses A2C. You can find a clear implementation of the same algorithm (using the same environment) in the following notebook: Continuous MountainCar Actor Critic Solution.ipynb.

Here you can also find a collection of Policy Gradient Algorithms. For example, PPO has proven to be very successful in environments like the one that you describe.

TL;DR Look for Policy gradient methods.


Function growing faster for negative inputs than for positives

This should be possible using a piecewise exponential loss, something like this:
$
f(x) =
\begin{cases}
x^2 & x < 0\\
\lambda x^2 & x \ge 0
\end{cases}
$
with $0 < \lambda < 1$. A $\lambda$ of around 0.02 should roughly give you the scale you want.


how to check all values in particular column has same data type or not?

You can get the type of the entries of your column with map:

df['ABC'].map(type)

So to filter on all values, which are not stored as str, you can use:

df['ABC'].map(type) != str

If however you just want to check if some of the rows contain a string, that has a special format (like a date), you can check this with a regex like:

df['ABC'].str.match('[0-9]{4}-[0-9]{2}-[0-9]{2}')

But of course, that is no exact date check. E.g. it would also return True for values like 0000-13-91, but this was only meant to give you an idea anyways.


Bandwidth selection Kernel Density Estimation

There is vast literature for bandwidths. Different bandwidth perform good in different situations. Its not necessary to use Rule of Thumb bandwidth with normal data. For instance, see Silverman 1986. "Density Estimation".


Could anyone explain these terms, "input space", "feature space", "sample space", "hypothesis space", "parameter space" with a concrete example?

We'll discuss each of the terms.
Input Space
It contains all the possible inputs for a model. Suppose the model takes in a vector, $input = [ x_1 , x_2]$ , where $x_1 , x_2 \in [ 1 , 10 ]$, then we can have $10^{2}$ inputs. This constitutes the "input space". See here.
For the MNIST dataset, the dimensions of the image are 8 * 8 meaning 64 points. Now each point can have a value lying in the interval $[ 0 , 16 ]$, so it can have 16 values. So the input space has a size of $16^{64}$.
Feature Space
The multidimensional space in which are features is defined. Considering the above examples, we can have three samples,
$a_1 = [ 2 , 3 ] \\ a_2 = [ 7 , 4.5 ] \\ a_3 = [ 3.67 , 2 ]$
These vectors could be included in an n-dimensional space ( here n=2 for our case ). Hence, in our case, the 2D space where we can plot our features constitutes our "feature space".
For the MNIST dataset, the input vector has 64 elements which correspond to a 64-dimensional space ( feature space ).
See this answer.

Difference between input space and feature space.
Input spaces include all possible inputs for our model. Feature
spaces, on the other hand, include the feature vectors from a given
set of data. They may not contain all the possible inputs for a model.

Hypothesis Space
Space which contains all the functions produced by a model. The functions map the inputs to their respective outputs. A model can output various functions ( or rather relationships between the inputs and outputs ) based on its learning. If you have a larger hypothesis space, the model cannot find the "best" one. See this answer.
For the MNIST dataset, as we calculated earlier, the size of the input space is $16^{64}$. Each one of them can have any one of the 10 labels ( classes ). Hence, the size of the hypothesis space is $10^{16^{64}}.$
Parameter Space
For each model in ML, we have some parameters for the model. The space in which we can define these parameters ( or hyperparameters ) is our "parameter space". From Wikipedia's example, we can understand it,
The parameter space would differ for every model.

In a sine wave model ${\displaystyle y(t)=A\cdot \sin(\omega t+\phi
> ),}y(t)=A\cdot \sin(\omega t+\phi )$, the parameters are amplitude A >
0, angular frequency ω > 0, and phase φ ∈ S1. Thus the parameter space
is ${\displaystyle R^{+}\times R^{+}\times S^{1}}$.


By ‘design document’ I assume you are authoring a technical doc in an industry job.

I treat this similarly to an academic journal publication but with less formatting and page restrictions. Ultimately, you want to provide sufficient information such that if Apple hired you tomorrow your replacement could reproduce the work. Further, your colleagues should be able to grasp the problem attacked, solution proposed, and evidence of success. I usually put the yawn-inducing deep stuff in appendix,  but suggested sections might include:
0) executive summary
1) background/motivation
2) data sources
3) analysis/algorithm/model
4) validation performance
5) [optional] real world test results/pilot
6) [optional] estimated return on investment
7) [appendix] SQL source code, modeling heuristic, outlier treatment, etc.


The simple option is to design your features so that they represent the distribution of the values: every feature $f_i$ represents a bin and its value for a particular instance is the frequency of the corresponding range for this instance.
Example: let's consider 10 bins between 0 and 1, i.e. $f_1=[0,0.1), f_2=[0.1,0.2),..., f_{10}=[0.9,1]$:

$x_1=[0.2, 0.25, 0.15, 0.22]$ is represented as $[0,1,3,0,0,0,0,0,0,0]$
$x_2 = [0.124, 0.224, 0.215, 0.095]$ is represented as $[1,1,2,0,0,0,0,0,0,0]$
...


Elements of Statistical Learning - question on p. 18

Maybe this helps:

On YouTube, there's a video with explanations for the main topics in Introduction to Statistical Learning (sorry, not for ELS).

https://www.youtube.com/watch?v=3jQs02dbfrI


Does autoencoder with few images not work?

I'm doing the same thing actually. I had to find tools because I was getting black images too.
This one worked for me:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.figure import Figure
from matplotlib.backends.backend_agg import FigureCanvas
from PIL import Image
import tensorflow as tf

keras_imager = tf.keras.preprocessing.image

# creating a figure to plot data
fig = Figure(figsize=(3, 2), dpi=96) # adjust your params
canvas = FigureCanvas(fig)
ax = fig.add_subplot(111)

# plot the data
ax.plot(your_data)

# draw the canvas, cache the renderer
canvas.draw()
s, (width, height) = canvas.print_to_buffer()

# converting to a NumPy array.
X = np.frombuffer(s, np.uint8).reshape((height, width, 4))

# converting to 3D rgb array data
img = keras_imager.array_to_img(X)
rgb = img.convert('RGB')
arr = keras_imager.img_to_array(rgb)
# standardize each image
arr = tf.image.per_image_standardization(arr)

# If you want to resize
arr = tf.image.resize(arr, (32, 32), # <- adjust your param
                     preserve_aspect_ratio=True)
# you can view the keras transformed plot
keras_imager.array_to_img(rgb)

Honestly, I don't quite understand all the parameters, but it's the simplest way I could tweak it to work for my situation. Good luck.


Visualizing categorical data in scatter plot with Orange (version > 3.18)

As a work around, you can change the data type in the file widget, from Categorical to Continuous. Then you can use scatter plot.


Learning rate in logistic regression with sklearn

sklearn.linear_model.LogisticRegression doesn't use SGD, so there's no learning rate.

I think sklearn.linear_model.SGDClassifier is what you need, which is a linear classifier with SGD training.

References

http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html


Why is eulers number used as a constant in sigmoid

Euler's number pops up in a lot of places naturally; not quite something to do with growth rates but arises easily in common limits.

The form of the sigmoid function wasn't chosen because its derivative has a nice property, although that's true. It also wasn't chosen because it's a function with range (0,1); many functions do that.

The sigmoid function arises because it's the correct answer to a common type of problem. We think of logistic regression as a common classifier predicting a class probability, but it really is regression under the hood. It's not regressing the probability though, but log-odds of the probability (logit function). This is the right way to apply regression, given assumptions about the distribution of errors in a classification problem, which aren't the same as in a simple linear regression.

The sigmoid function is the inverse of the logit link function. That's why it's there. It gets from the regression output to the actual desired output, a probability. The logit function is there because it is implied by the assumption about the distribution of the 0/1 dependent variable.

That's actually it. It has to be there in certain types of common problems because it's the answer to those problems, not because it was chosen for nice properties.


How to combine two different machine learning models, to get the combined result?

Train another XGB model on second layer. Your first layer contains all the individual model and in the second layer you will train another model on top of all of them. XGB is good for this task.


$f =X^T Y$ looks like this
\begin{equation}
 f= \begin{pmatrix}
  \sum_i x_iy_{i1}\\
  \sum_i x_iy_{i2}\\
  \vdots\\
  \sum_i x_iy_{in}\\
 \end{pmatrix}
\end{equation}
where $x_i$ is the $i^{th}$ element of $X$ and $y_{ij}$ is the $(i,j)^{th}$ element of $Y$.
Now, to get the gradient w.r.t $X$, is equivalent to deriving each element of $f$ w.r.t each element of $X$. This will lead to a 2D matrix:
\begin{equation}
 \nabla_X f =
 \begin{pmatrix}
  \frac{\partial}{\partial x_1} f_1 & \ldots & \frac{\partial}{\partial x_n} f_1\\
  \vdots & \vdots & \vdots \\
  \frac{\partial}{\partial x_1} f_n & \ldots & \frac{\partial}{\partial x_n} f_n
 \end{pmatrix}
\end{equation}
We can therefore easily see that
\begin{equation}
\frac{\partial}{\partial x_j} f_i
=
y_{ij}
\end{equation}
Therefore $\nabla_X f = Y$.