This is relatively simple to do mathematically.  First, fit a regression line to the scatter plot of "total graduates" (y) vs. "total students" (x).  You will probably see a downward sloping line if your assertion is correct (smaller schools graduate a higher %).

You can identify the slope and y-intercept for this line to convert it into an equation y = mx + b, and then do a little algebra to convert the equation  into normalized form: "y / x = m + b / x"

Then, with all the ratios in your data , you should subtract this RHS:

normalized ratio = (total grads / total students) - (m + b / total students)

If the result is postive, then the ratio is above normal for that size (i.e. above the regression line) and if it is negative it is below the regression line.  If you want all positive numbers, you can add a positive constant to move all results above zero.

This is how to do it mathematically, but I suggest that you consider whether it is wise, from a data analysis point of view, to normalize by school size.  This depends on the purpose of your analysis and specifically how this ratio is being analyzed in relation to other data.


Advantages of monotonic activation functions over non-monotonic functions in neural networks?

I don't know of any papers about this topic, but intuitively it makes a lot of sense to use monotonic activation functions. Let's say we have a non-monotonic activation function, maybe a Gaussian kernel, symmetric around $x=0$ but slides off towards $f(x)=0$ if x strays away from 0 on either side. If we have a sample that we feed into our network that performs poorly when our activation is high, we want to change the input of our node to give a lower activation. In case of a non-monotonic activation, whether we want to decrease or increase the input depends on whether the input was positive or negative, and is mostly dependent on our weight initialization.

This makes learning more difficult, because if another sample also  needs it to be lower but was on the other side of the top, backpropogation will attempt to map the input to the other side. Most of the time the best solution will be to put everything on one side of the top, making it monotonic again. Another way of looking at it is that monotonic are somewhat one-to-one (not entirely true, for example ReLU). This means that two very different inputs don't map to the same output unless everything in between also maps there.

Here was a similar question with some links: (Why) do activation functions have to be monotonic?


Why do we need for Shortcut Connections to build Residual Networks?

The short answer is that when a net is very deep it becomes very difficult for gradients to propagate backwards all the way. Skip connections offer "short cuts" for gradients to propagate further and allow for efficient training of very deep nets.


Reward is converging but actions taken by trained agent are illogical in reinforcement learning

Without seeing the rest of your code, this is a bit tricky to answer, but you have to make sure that reward = -temp_penalty, i.e. the negtive penalty, otherwise you would learn exactly the behaviour you described. Here I'm assuming you excluded all other potential sources of error.

Further, I think it might be helpful to issue a reward if the agent stays within the limits you defined, i.e. in the else-clause set temp_penalty = -1. or something like that. I personally found this type of tweaking to be very helpful for DQN.


As @NoahWeber mentioned, one solution is to:

split by customer ids (A)
do the time series split on all dataset (B)
keep in the training (resp. testing) dataset only the data from training (resp. testing) customers split (A) and from training (resp. testing) time series split (B).

Below is a code sample I was writing at the same time he answered.
import pandas as pd
import numpy as np
from sklearn.model_selection import RepeatedKFold
from sklearn.model_selection import TimeSeriesSplit

# Generating dates
def pp(start, end, n):
    start_u = start.value//10**9
    end_u = end.value//10**9

    return pd.DatetimeIndex((10**9*np.random.randint(start_u, end_u, n, dtype=np.int64)).view('M8[ns]'))

start = pd.to_datetime('2015-01-01')
end = pd.to_datetime('2018-01-01')
fake_date = pp(start, end, 500)

# Fake dataframe
df = pd.DataFrame(data=np.random.random((500,5)), index=fake_date, columns=['feat'+str(i) for i in range(5)])
df['customer_id'] = np.random.randint(0, 5, 500)
df['label'] = np.random.randint(0, 3, 500)

# First split by customer
rkf = RepeatedKFold(n_splits=2, n_repeats=5, random_state=42)
for train_cust, test_cust in rkf.split(df['customer_id'].unique()):
    print("training/testing with customers : " + str(train_cust)+"/"+str(test_cust))

    # Then sort all the data (if not already sorted)
    sorted_df = df.sort_index()

    # Then do the time series split
    tscv = TimeSeriesSplit(max_train_size=None, n_splits=5)
    for train_index, test_index in tscv.split(sorted_df.values):
        df_train, df_test = sorted_df.iloc[train_index], sorted_df.iloc[test_index]

        # Keep the right customers for training/testing
        df_train_final = pd.concat( [ df_train.groupby('customer_id').get_group(i) for i in train_cust ])
        df_test_final = pd.concat( [ df_test.groupby('customer_id').get_group(i) for i in test_cust ])

Note: Generating random dates is based on this post
Note bis: I tested the generated training/testing dataframes ready for cross-val with this sample code that you can add right after the line df_test_final:
# Test condition 1: temporality
for i in range(len(df_test_final)):
    for j in range(len(df_train_final)):
        if df_test_final.index[i] < df_train_final.index[j]:
            print("Error with " + str(i) + "/" + str(j))

# Test condition 2: training customers are not in testing final df
for i in train_cust:
    if i in df_test_final['customer_id'].values:
        print("Error in df_train with " + str(i) + "th customer")

# Test condition 2: testing customers are not in training final df
for i in test_cust:
    if i in df_train_final['customer_id'].values:
        print("Error in df_train with " + str(i) + "th customer")

Here is a pseudo-code implementation:
function keep_customer_ids( data, ids ):
    goal: this function returns a subset of data with only the events that have
          an id tag that is in ids
    data: labeled events containing features, date and a customer id tag
    ids: the customer ids you want to keep
    for event in data:
        if event has a customer id tag that is in ids, keep it
        else, drop it
    return data

algorithm:
    for the number of cross-val you want:
        customer_train_ids, customer_test_ids = split_by_customers( customer_ids )
        train_data, test_data = split_data_in_time_series_way( data )
        final_train_data = keep_customer_ids( train_data, customer_train_ids )
        final_test_data = keep_customer_ids( test_data, customer_test_ids )
        do_the_fit_predict_things( final_train_data, final_test_data )


But then again, the input clearly has the correct shape.

>>> import numpy as np
>>> X = np.arange(15)
>>> X.shape
(15,)

In Keras, input_dim represents the number of input parameters, in your case that would be the number of columns of X or its second dimension (sometimes also referred to as number of features). It is clearly not 15. It is the first dimension that is 15. That means: X consists of 15 rows, also called samples (of one and the same feature).

So in that case, input_dim=1.

However, you will then run into the problem of having specified Y = 0. First, Keras will throw an error because it is an integer. You could do Y = [0], but then you will get

ValueError: Input arrays should have the same number of samples as target arrays. Found 15 input samples and 1 target samples.

So you have to turn this into an array-like object containing 15 samples, e.g. a list of length 15.

However, in case you meant to feed one single sample X to your model, that maps to one single output Y = [0], then you need to reshape X accordingly, for example via

X = np.arange(15).reshape(n_samples, n_features)

Can you figure out now what n_samples, n_features needs to be?


As per the link you provided, conda install -c auto dlx is the correct way to install the package in Anaconda.


In supervised learning, how to get info from correlation?

To help you, that shows the correlations between features and each other feature. For example, the number one in the image that you gave, is shown all along the diagonal part of the matrix. The ones represent the 100% linear correlation with one of the features and one of the other features. This image might help:

As you can see negative correlations mean that as one feature value increases, the other feature value decreases. A correlation of 0 means that the features appear to have no linear correlation.

If you want to merely concentrate on the correlations between the label and the features, then here is some python code (the language I assume you are using) to help you:

# Your data should be a pandas core dataframe

yourdata = ...

# To find correlations, use the corr() function
corr_matrix = yourdata.corr()
corr_matrix["your label"].sort_values(ascending=False)
# This should print out a correlation list.
# If it doesn't then wrap the last line of code in print( )
# You are going to notice that some features will be missing from the list.
# That is because the corr() function does not return any discrete features.
# If you still have every feature, then every one of your features are continuous.


Can I use LSTM models to evaluate multiple, independent time series?

If you are looking to predict multiple time series (which would be similar in nature, since each weather station in the area would record similar temperatures, even if they are not identical), using a separate LSTM model for each may prove quite time-consuming.

One approach you could take is one suggested in an excellent answer for another question on Cross Validated. Essentially, the author is describing a means for forecasting sales with LSTM whereby the model is trained on a mini-batch (or subset) of one series, and then a new series is selected. In this case, I would understand this to mean that a subset of data is incorporated from weather station 1, then another batch from weather station 2, etc.

This would have the advantage of essentially creating a unified series that takes the characteristics of all weather stations into account - which allows for maximum utilisation of the data, as well as allowing the network to learn patterns from all stations - not just one or a select few.

One option is to try this method using a subset of data for select weather stations at first, and then compare the prediction accuracy to those that use separate LSTM models for each. If you find that the predictions for the former method are more accurate, then it would make sense to go with that.

If you are using R, then there is also the option of using a package like forecastML to do so - if you are not tied down to using an LSTM model then this could also be another consideration.


Understanding pseudocode for co-democratic learning

Let's try to make some sense of what is going on here:

In the first 3 lines as you say, we build $n$ models, being $m$ a single model.

The second loop starts iterating over the unlabelled sample set $U$. For each unlabelled sample $x$, we iterate over the possible labels (e.g $\{0,1\}$ for binary classification, or $dog\space vs.\space cat$, etc...) from the set $C$ and we call it $j$.

$M$ thus is the set of models $m$, that predict that sample $x$ is of the class $j$

After that, if the number of models $|M|$ (where $0\leq |M|\leq n$) that agree that label is $j$ are more than half of the models available ($|M|>n/2$) and the weighted mean ($w$ is the confidence intervals
of the models) of the models that agree on this prediction is higher than the models that disagree (line 7) → then the sample $x$ is included in $L$ with label $j$ (line 8).


It seems you are doing it right. The difference between case 1) and 2) should be mainly the loss function (and maybe the activation of the predictions layer).

For 1) you would use e.g. mean_squared_error, while for 2) you would use the binary cross entropy from logits (logits are what you would feed into a sigmoid or softmax layer). In the keras-documentation it seems like keras does not provide cross entropy from logits as a loss, so you have to set the activation of predictions to sigmoid and use binary_crossentropy. I guess then you could do the If result >0 and <= 0.5 then classify as 0 ..., the result >0 would be taken care ouf by definition of the sigmoid, however you might want optimize the threshold w.r.t. the precision and recall you need.


How to deal with errors of defining data types in pandas' read_csv ()?

If you see the warning that your column has mixed types, but you only see numbers there, it could be that missing values are causing the problem.

In Pandas 1.0.0, a new function has been introduced to try to solve that problem. Namely, the Dataframe.convert_dtypes (docs).

You can use it like this:

df = pd.read_csv(filename, header=None, sep='|', usecols=[1,3,4,5,37,40,51,76])
df = df.convert_dtypes()

then check the type of the columns

print(df.dtypes)


How to get phrase tables from word alignments?

This isn't my area of specialty and I'm not familiar with Moses, but I found this after some searching.

I think you are looking for GIZA++.  You'll see GIZA++ listed in the "Training" section (left menu) on the Moses home page, as the second step. GIZA++ is briefly described in tutorial fashion here. Here are a few tutorial PowerPoint slides: http://www.tc.umn.edu/~bthomson/wordalignment/GIZA.ppt


Why is scale parameter on batch normalization not needed on ReLU?

Scaling in batch normalization is said not to be required because of the nature of ReLU. Although it is a non-linear operation, a multiplication with a positive scalar $\gamma$ over the values of a vector $x$ which are already normalized and centered to the trained bias $\beta$ makes no influence on its non-linear outcome: for $\gamma > 0$, $max(0, \gamma \times x) = \gamma \times max(0, x)$. In fact, the extra parameter adds no representative power in the model, for the same reason that we must include non-linearities between other neuron layers.

And so, the weights of the following layer would scale the values to fit accordingly. We could have $z = Wgx$, but that would be the same thing as $z = Wx$ if $W$ is scaled by $g$. The same reasoning can't be made for other non-linearities, such as the sigmoid or tanh, which have a saturating effect that is clearly influenced by a scalar.

Curiously, the official documentation on TensorFlow's tf.contrib.layers.batch_norm makes the same recommendation:

  scale: If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. nn.relu), this can be disabled since the scaling can be done by the next layer.


What does the activation of a neuron mean?

The activation of a neuron is mathematically nothing but a function of its input. Consider a neural network with one hidden layer and one input vector $\mathbf{x}$. The input $a_j$ of neuron $j$ can be written as:

$a_j = w_j^T \mathbf{x} + b_j$.

The activation of neuron $j$ is then a transformation $g: \mathbf{R} \rightarrow \mathbf{R}$ of the input. For exmaple, one can use the sigmoid activiation function

$sig(a_j) = \dfrac{1}{1 + \exp(-a_j)} $

The weights $w$ are chosen to minimize a loss function and not for the sake of their interpretation. Nevertheless, one can try to interpret the activation of a neuron as internal representation of the input. One of the nicest examples I've seen comes from the Rummelhart et al (1986), Figure 4. In that paper, two family trees were fed into a neural network. The family trees represented an Italian and an English speaking family comprising three generations. Among other things, when activating one name of the tree, the neuronal activities represented whether the person was from the English or Italian tree and which generation the person embodied.


How to Find the Average of the Input Vectors

Does this work for you? After the Bi-Directional LSTM layer, I sum the tensor over the batch axis and then divide by the batch size.

from tensorflow.keras import layers, Sequential
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
from tensorflow.keras.utils import plot_model

inputs = layers.Input(batch_shape=(4, 500), name="Input")
embed = layers.Embedding(input_dim=500, output_dim=30, name="Embedding")(inputs)
encoder = layers.Bidirectional(
                  layers.LSTM(units=20,return_sequences=True,dropout=0.3,name="LSTM")
                  ,name="Bi-LSTM")

hidden_states = encoder(embed)
total = layers.Lambda(lambda x: K.sum(x, axis=0)/K.cast(K.shape(x)[0],"float32"))(hidden_states)

model = Model(inputs, total)
model.summary()


Storing Sensor Data for Analysis of the Office

I have been doing similar project in my college. I have classroom and I'm supposed to collect data like temp, humidity, light, occupancy, etc. Assuming that you have worked with sensors and motes to use, I'm going to explain rest of the structure.

You need sensor network setup and like you said you have done it. These sensor networks generally do not send data directly over internet so you need a Gateway that can collect data from sensors and send it over internet to local server.

On server side you need REST API and you could use any language to develop it and I use PHP. I find it very easy to use and develop using PHP. This REST API shall receive data from Gateway and store it into database. I use mysql database because amount of data is not so big for us. But if your data is big enough you can use big data Nosql tool like mongoDB or so. Whatever type of database you use structure remains same.

For sending data from Gateway to server you can use protocols like HTTP or MQTT whichever you feel comfortable. What I do is I have WSN controller that sends data over USB to Gateway then Gateway sends data to server over Ethernet. So I had to develop USB to Ethernet Gateway. If you can just take two UART terminals out of your controller you can build UART to Ethernet Gateway using any microcontroller or even Arduino Ethernet shield would work in that case.

In my case data is sensed periodically but as you said you are sensing data when event of interest occurs then you can use poisson distribution method over periodically collected data to predict what is average number of events per day and then you can decide if your data is big or not.


Neural Networks - Loss and Accuracy correlation

Log loss has the nice property that it is a differentiable function. Accuracy might be more important and is definitely more interpretable but is not directly usable in the training of the network due to the backpropagation algorithm that requires the loss function to be differentiable. When your preferred loss is not directly optimizable (like the accuracy) you use a loss function that behaves similarly to proxy the true metric. In case of binary classification you would use a sigmoid at the end and a log loss to approximate accuracy. They are highly correlated.


Softmax vs Sigmoid in RBM/Auto Encoder final layer

This choice mainly depends on what your output represents. Given a vector $\mathbf{x}$, the sigmoid function is given by
$$ \sigma(x_i) = \frac{\exp(x_i)}{1 + \exp(x_i)} $$
while the softmax is given by
$$ \mathrm{softmax}(x_i) = \frac{\exp(x_i)}{\sum_{j=0}^N \exp(x_j)}$$
A key difference is that the output of the sigmoid function applied to $x_i$ only depends on $x_i$. The output of the softmax function depends on all elements of the vector $\mathbf{x}$.

The sigmoid will squash each $x_i$ into the range $(0, 1)$, which enables you to interpret $\sigma(x_i)$ as the probability of $x_i$.
But, if you have multiple classes, e.g. 0-9 in MNIST, each probability is independent, that means you could have probability $p=0.9$ for the digit 1 and $p=0.5$ for the digit 3. This is undesirable if you want to distinguish between multiple classes. However, if multiple classes can appear at the same time, then sigmoid is well suited.

Now, the softmax is basically a sigmoid function which is normalized such that $\sum_{j=0}^N \mathrm{softmax}(x_j) = 1$. This means that the output of a softmax layer is a valid probability mass function, i.e. the output of your neural network is the probability of the input belonging to a certain class.

So, if you want to classify between e.g. the 10 digits as in MNIST, then softmax is the way to go. If the classes are independent, i.e. it is possible that an image shows a cat and a dog at the same time, then sigmoid is the way to go.


Are there any rules for choosing the size of a mini-batch?

In On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima there are a couple of intersting statements:

  It has been observed in practice that
  when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize [...]

  large-batch methods tend to converge to sharp minimizers of the
  training and testing functions—and as is well known, sharp minima lead to poorer
  generalization. n. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation.

From my masters thesis: Hence the choice of the mini-batch size influences:

Training time until convergence: There seems to be a sweet spot. If the batch size is very small (e.g. 8), this time goes up. If the batch size is huge, it is also higher than the minimum.
Training time per epoch: Bigger computes faster (is efficient)
Resulting model quality: The lower the better due to better generalization (?)

It is important to note hyper-parameter interactions: Batch size may interact with other hyper-parameters, most notably learning rate. In some experiments this interaction may make it hard to isolate the effect of batch size alone on model quality. Another strong interaction is with early stopping for regularisation.

See also

this nice answer / related question
Efficient Mini-batch Training for Stochastic Optimization
this RNN study


Visualising feature selection results for multiple classifiers and feature subset sizes

You will have to study and understand Matplotlib and its tweaks.
This code will do the basic work. You can extend it. Also, please go through the references that are added at the end.
import matplotlib.pyplot as plt

data = {'AUC':{'RF':[0.7,0.2,0.5,0.9,0.4], 'LR':[0.9,0.25,0.35,0.99,0.55], 'SVM':[0.3,0.5,0.8,0.6,0.7] } }
x = ['S1','S2','S3','S4','S5']

plt.plot(x, data['AUC']['RF'], marker='^', linestyle='solid')
plt.plot(x,data['AUC']['LR'], marker='o', color='r',linestyle='dashed')
plt.plot(x,data['AUC']['SVM'], marker='s', color='b',linestyle='dashdot')

$\hspace{3cm}$
References -
Matplotlib Official tutorial
Python Data Science Handbook by Jake VanderPlas
Matplotlib Lines
Matplotlib Markers
Matplotlib Colors


Can we calculate AUC for deep learning based regression task

AUC is for binary classification tasks only, it's not possible to calculate it on a regression model. However it's plenty of other metrics one can use to check a regressor performance, such as MAE and MSE.


Precision or Recall when dealing with critical cases?

Let us say your target variable "1" represents "Harmful Object".
Precision answers the question - Of all objects predicted as Harmful - How many are really harmful ?
Recall answers the question - Of all the Harmful objects out there - How many were correctly identified ?
You want to increase on Recall in your case - since that reduces the overall risk to the machine.


Python - Appending a new Dataframe column that is a function of two separate numerical columns

Use the distance matrix to know the distance from one point to all other corresponding points.

Here is an example:

def dist_PQ(P,Q):
    PP=P*P
    QQ=Q*Q
    PSquared_Sum = np.sum(PP,axis=1,keepdims=True)
    QSquared_Sum = np.sum(QQ,axis=1,keepdims=True)

    squared_distance = PSquared_Sum + QSquared_Sum.T - 2*np.dot(P,Q.T)

    return(squared_distance)

Use the above function to find distances between two sets of points say (P and Q)
Say P has 10 points the values are as numpy array [[x11,y11],[x12,y12],[x13,y13]...] and Q has 3 points as [[x21,y21],[x22,y22],[x23,y23]].
pass P and Q to the dist_PR function to get the output as (10,3) matrix which gives the distances from all points in array P to all points of Q.

Now we can use the matrix of distance in our further calculation.
Suppose we want to find the place which is nearest from P1 (a point in P), we can simply find the minimum distance value in row number 1 of that distance matrix.


How pre-trained BERT model generates word embeddings for out of vocabulary words?

BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, BERT only gives you the subword vectors.
Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it will be sliced into multiple subwords, even reaching character subwords if needed. That is how it deals with unseen words.
ELMo is very different: it ingests characters and generate word-level representations. The fact that it ingests the characters of each word instead of a single token for representing the whole word is what grants ELMo the ability to handle unseen words.


NLP SBert (Bert) for answer comparison STS

I have used Siamese Bert and I can say it does a pretty good job. However, the issue is that the data that it has been fine-tuned atop of Bert may not necessarily, entirely represent the same semantic distance as with the answers between the true and the student's one. For instance, if there is a question about engineering, where a small change of word may mean a totally different thing; SBert would still find them quite similar cause they are related to the topic.  Unless it's fine-tuned.
Moreover, you will not be able to interpret the similarity. Should a student ask you why my peer's answer is better you won't be able to explain.
My opinion: I believe you could use this tool as a way to reduce totally incoherent answers, but at some point, human evaluation will be needed. And maybe use interpretable metrics such as ROUGE or BLEU. I aware as well, that this topic is quite trendy in NLP, I won't be surprised if there is or will be good off the shelf tool for that, but I am not aware of one currently.


Learning a simple sequence with RNN (Keras)

Using the raw integers as inputs and targets will make this a very difficult task. A better approach would be to come up with a vector for each number. You can simply encode each number directly as a vector, use a "onehot" representation, or use an Embedding layer. You can see an example of embeddings in conx that is built on Keras here:

http://conx.readthedocs.io/en/latest/Predicting%20and%20Generating%20Texts.html?highlight=embeddinglayer

and and example without embeddings using onehot representations here:

http://conx.readthedocs.io/en/latest/AliceInWonderland.html


How to order the data with respect to data type

Let your data frame be df. First get the numeric columns:

num_col = df.select_dtypes('number').columns

Then get the remaining columns.

non_num_col = set(df.columns) - set(df.select_dtypes('number').columns)

Merge as required.

df = pd.concat([df[num_col], df[list(non_num_col)]], axis=1)

The columns are now in the desired sequence.


Can we use a predictive model of a device to another device of similar specification?

The answer to your question is yes, essentially while building a model what we are doing in reality is devising a mathematical function of a physical object (here car). So essentially what we are building is a mathematical model of a car.

What you should be aware of is ?
1. Say for e.g. if you are building a model for a car A and Car B, and if car A is a gasoline(internal combustion car) and car B is an electric car, then things become tricky if you want to build a mathematical function of internals of a car.
2. This does not affect if the model does not consider the internals of the car meaning the outcome (prediction) is not dependent on the internals of the car, which may not be the case always.

but in general you can use the model, it is always preferable to do a validation on the new dataset.


Dynamic learning rates in XGBoost cross-validation

There isn't as far as I can tell by looking at the source code of the version I'm working with (April 2016). But it's definitely worth adding and it isn't impossible. xgboost.train loops over the iterations extracts corresponding learning rate, sets it on booster via eta parameter and calls booster update. xgboost.cv calls fold update (ie CVPack.update) which calls booster update. Just need to add a learning rate (aka eta) parameter to CVPack.update


Categorical, nominal or continuous variable?

I don’t know where you’re getting this particular classification scheme from, (continuous, categorical, nominal), but it’s worth mentioning that this isn’t a great scheme for classifying data types. You’ve already encountered an example of a data type (count data/natural numbers) that doesn’t fit neatly into them.

It is also worth mentioning that a lot of people will use “nominal” and “categorical” interchangeably to describe a discrete data type with no natural ordering. Probably the most widely-accepted term to describe discrete data with a natural ordering is ordinal data.

In this case, "number of rooms in a flat" has an unambiguous ordering. In a very real and intuitive sense, a five-room flat has more rooms than a two-room flat. A one-room flat has fewer rooms than a three-room flat. It makes sense to talk about one flat having a greater or lesser number of rooms than another, and if we want to know if that ordering matters in some way, we need to preserve the ordering in our analysis.


sklearn - How to create a sequential pipeline

When you want to do sequential transformations, you should use Pipeline.
imp_std = Pipeline(
    steps=[
        ('impute', SimpleImputer(strategy='median')),
        ('scale', StandardScaler()),
    ]
)

ColumnTransformer(
    remainder='passthrough',
    transformers=[
        ('imp_std', imp_std, ['feat_1', 'feat_2']),
        ('std', StandardScaler(), ['feat_3']),
    ]
)

or
imp = ColumnTransformer(
    remainder='passthrough',
    transformers=[
        ('imp', SimpleImputer(strategy='median'), ['feat_1', 'feat_2']),
    ]
)

Pipeline(
    steps=[
        ('imp', imp),
        ('std', StandardScaler()),
    ]
)


What is the problem with this architecture ? I am getting error negative dimensions. I want to avoid dense layers and dropouts

You are using too many layers and you run out of spatial space.

Most of your convolutional layers use "valid" padding, meaning that the convolution is performed only on actual "pixels" without any padding and as a result the spatial dimensions of the output are smaller than the input.

I've marked down where it happens in your script:

model = Sequential()
model.add(Conv2D(64, (1, 1), activation='relu', input_shape=(64,64,3)))
model.add(Conv2D(64, (5,5), strides=(2,2), padding='same'))
model.add(MaxPooling2D(pool_size=3, strides=(2,2)))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(64, (3, 3),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(128, (1,1),
                strides=(2,2),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1, 1),
                          strides=(1,1),
                          activation='relu',
                          padding='valid' ))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1,1),
                strides=(1,1),
                activation='relu',
                padding='valid'))

model.summary() # This is where it happens - The output of this layer is of shape (1,1,128)

model.add(Conv2D(128, (3,3),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (1,1),
                strides=(1,1),
                activation='relu',
                padding='valid'))
model.add(Conv2D(128, (3,3),
                strides=(1,1),
                padding='valid'))

model.add(AveragePooling2D(pool_size=2, strides=(2,2), padding='valid'))
model.add(Conv2D(1,1,200))
model.add(Flatten())
model.add(Activation('softmax'))

You can use the Keras "summary" method to investigate your model. For example, the output from the script I've written here is:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 64, 64, 64)        256
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 32, 32, 64)        102464
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 15, 15, 64)        0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 15, 15, 64)        4160
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 13, 13, 64)        36928
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 13, 13, 64)        4160
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 11, 11, 64)        36928
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 11, 11, 64)        4160
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 9, 9, 64)          36928
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 5, 5, 128)         8320
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 3, 3, 128)         147584
_________________________________________________________________
conv2d_11 (Conv2D)           (None, 3, 3, 128)         16512
_________________________________________________________________
conv2d_12 (Conv2D)           (None, 1, 1, 128)         147584
_________________________________________________________________
conv2d_13 (Conv2D)           (None, 1, 1, 128)         16512
=================================================================
Total params: 562,496
Trainable params: 562,496
Non-trainable params: 0


Multidimensional Scaling with Categorical Data

MDS only requires a distance matrix which stores the distance between each pair of data examples. How to compute that distance depends on the kind of data you are dealing with. If you only have numerical (real valued) features, you can use the Euclidean distance, but that is not always the case.

For instance, if you have both numerical and categorical variables, you may apply any metric for mixed data, like for instance this one.

Regarding your second question, I need further clarification. What do you mean by "converging? Are you asking if both algorithms converge to the same solution? Is that is your question, the answer is no, because the aim of both algorithms is different. MDS projects your data onto a 2D plane by trying to keep relative distances, whether PCA projects data by focusing on those "directions" in which there is more variability.


Is it possible to measure the object using deep learning

Not without some kind of reference object. I also don’t see a reason to use deep learning for such a task.
Sure - you could create a dataset with all kinds of objects and their typical dimensions. But the results for inference would probably be pretty bad and the effort to create the dataset ridiculously huge.


Try Resampling Your Dataset

One option is to reduce de occurance of the majority class in your training set.

One other option is to over-sampling the minority class. (you may need to ad a bit of noise )

Other idea:
Try Changing Your Performance Metric

The page helped me a lot with unbalance data:
Combat Imbalanced Classes in Your Machine Learning Dataset


There is an extensive literate and established software to normalize CT scans. It is required for medical research to conduct group level analysis. The most common process involves transforming to standard template. The paper "Age-specific CT and MRI templates for spatial normalization " goes into greater detail.


Implementation of Gaussian Mixture Model for clustering when dealing with multidimensional hyperspectral data in python

If I understand your type of data correctly, what you have is essentially an image with $800\times800 =640000$ pixels and each pixel has a vector of 4 values.

If this stands, I suppose you could then transform your data to a $640000\times4$ matrix, so as to conform with scikit-learn's data representation schema of inputting matrices of shape ($\#samples\times\#features$) and then you could use the GMM class implemented by the package.

And you could transform the data back if needed.

P.S.: On a second thought, I don't know if the spatial distribution of pixels plays a role on the generation of the GMM components. Maybe try it out and let us know?

Ok, after you have trained your model you can ask for it to predict the mixture model from which each pixel in your dataset has been generated.

That is if you run again:
$clusters=gmm.predict(newdata)$ you will get an array with values $0-2$, denoting the cluster that each pixel belongs to. You can, then map these values in the image at hand, applying a mask of different color for each pixel in order to see if your model correctly identifies the three classes.

Here is an example of what I mean:

from scipy.misc import imread, imshow
from sklearn.datasets import load_sample_image

X = load_sample_image('china.jpg')
old_shape = X.shape
X = X.reshape(-1,3)
gmm = mixture.GMM(covariance_type='full', n_components=5)
gmm.fit(X)
clusters = gmm.predict(X)
clusters = clusters.reshape(old_shape[0], old_shape[1])
imshow(clusters)


Method for predicting price based on Geographical market, Product, and Company

It may be usefull to put what you are doing in a formal context, that way you can look at some standard solutions to your problem.

You have obsvations of a (random) variable X (price), together with some explanatory variables L, C, P (L="location" = city, C=company, P=price). So your data consists of cuadruples $(C_i, L_i, P_i, X_i)$.

You are assuming a $\mathbf{multiplicative}$ model:
$$ E(X|C,L,P) = \lambda \space f_C \space g_L \space h_P .$$

This model fits nicely in the context of generalized linear models, since:
$$ \ln (E(X|C,L,P)) = \ln(\lambda) + \ln( f_C) + \ln( g_L) + \ln( h_P) .$$

You are fitting the model by a variation of iterative scaling. I don't know if there is a probabilistic model that corresponds to that way of fitting the models. Both the multiplicative model and iterative scaling have long traditions, so the answer might be known.

In any case, once in the context of generalized linear models, you can fit the parameters making some assumptions on the error. Since $X$ counts the number of monetary units per item, you can start by trying a Poisson model (and a Poisson model with overdisperssion). It is also worth trying other GLM's.


How to properly predict date using Orange 3

So you have two independent variables. iOS version and release name, which is actually a type. Both are categorical and only one has any predictive power, because iOS version you are trying to predict does not appear in your training set. Thus, effectively, you are predicting based on the release type only and only the "release" category is actually meaningful because this is what you are trying to predict.

I can hardly think of a model that would perform ok with such limited information. Anyway, regression models will always give you some answer, even if your input is unsound.

I guess the better features would be:

month
day of month
days since last release
months since last release
years since last release
days since beta
months since beta
years since beta

This type of thinking. Then just by looking at the data you would clearly see that prod releases of iOS happen always in mid Sept each year. No need for model to figure it out :) Any method would suffice, especially kNN or least squares.


How to deal with a constant value as an output from neural network?

Lowering the initial learning rate from 0.1 to 0.005 helped the problem.

adam = optimizers.Adam(lr=0.005)


Aha! Figured it out!

Here is the code I have:

install.packages('gtools')
library(gtools)

x <- c(0, 1) # heads = 0, tails = 1

p = permutations(n = 2, r = 10, v = x, repeats.allowed = T)

p.df = as.data.frame(t(p))

q = apply(p.df, 2, function(x){
max(rle((x == 0))$lengths)
}
)

length(which(q == 10)) #where q == the length of the longest run
```


Replace Values with corresponding numbers in the same dataframe

I'm not sure if this is most efficient way but it may help you.

df = read_csv('filename.csv')
grouped_df = df.groupby('City')
temp_df = []
for key, item in grouped_df:
    if 'NR' in item['Route'].values.tolist():
        values = item['Route'].values.tolist()
        to_change = [x for x in values if x != 'NR'][0]
        item = item.replace('NR', to_change)
        temp_df.append(item)
    else:
        temp_df.append(item)
final_df = pd.concat(temp_df, axis=0)

Output:

        City Route
4          A     3
5          B     4
6          C     5
7          D     7
8          D     7
9          D     7
10         D     7
2   Kolkatta     2
3   Kolkatta     2
0    Manipur    10
1    Manipur    10


Removing outliers with orders of magnitude differences

I managed to do it. I finally used three outlier filters.

Filter extreme cheaters: Per [level, currency source or sink, type (earn/spend), payer/non-payer, month of start (if game changes over time, new people might earn or spend in different ways than before)] I calculated the median and median absolute deviation (MAD) of the amount of currency earned/spend. Filter user if: amount earned/spend > 100 * median + 100 * MAD
Filter users that spend more of a currency than they have earned.
Filter users that earn a currency by means of an in app purchase (IAP) but didn't generate any real money revenue in that level (i.e. they did an IAP but didn't pay for it).


Vertical lines of overlapping points in Seaborn Swarmplot

As per the Sea born documentation, -

https://seaborn.pydata.org/generated/seaborn.swarmplot.html

"This function is similar to stripplot(), but the points are adjusted (only along the categorical axis) so that they don’t overlap. This gives a better representation of the distribution of values,

  but it does not scale well to large numbers of observations.

This style of plot is sometimes called a “beeswarm”."

A workaround may be adjusting the figsize.


Decision tree classifier prediction changes from one run of the model to the next

While this is a duplicate and the suggested link answers your question, for learning purposes I would like to suggest that you plot your DecisionTree every time you have a new run to see by yourself what happens behind the scenes:

from sklearn.tree import DecisionTreeClassifier
from sklearn.externals.six import StringIO

from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus
dot_data = StringIO()

X = [[190, 90, 43], [165, 65, 38], [170, 70, 39], [160, 56, 36], [190, 88, 45], [164, 63, 37]]
Y = ['male', 'female', 'male', 'female', 'male', 'female']

clf = DecisionTreeClassifier()
clf.fit(X, Y)

export_graphviz(clf, out_file=dot_data,
                filled=True, rounded=True,
                special_characters=True)

graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())

Once you may get:

where splitting is done on X0 feature, where as next run you may get:

where splitting is done on X1 feature.

If you wanna reproduce your result at every run, you can use random_state = a_random_number and then you should expect to get the same result every time as the same tree has been constructed every time!


Should I keep common stop-words when preprocessing for word embedding?

In general stop-words can be omitted since they do not contain any useful information about the content of your sentence or document.
The intuition behind that is that stop-words are the most common words in a language and occur in every document independent of the context. Therefore they contain no valuable information which could hint to the content of the document.


What does it mean when we say an algorithm/metric is agnostic

I know of the model-agnostic term, and a close meaning would be model-independent. Basically, when you are studying a machine learning problem, the underlying structure in the data may or may not be described by one type of model or the other. The model-agnostic approach consists in using machine learning models to study the underlying structure without assuming that it can be accurately described by the model because of its nature. This avoids introducing a potential bias in the interpretation.

A model-agnostic approach pretty much requires that several different techniques are used for the same task.


Image segmentation - handcrafted features vs DNN?

You have an object vs background task, i.e. two classes and instance (or m.b. class segmentation problem). If you have a labelled dataset, you can finetune one of the publicly available networks (e.g. FCN, MaskRCNN) to it to produce an image mask (i.e. an array size of the input image where every pixel is labelled as either object or background). So yeah, in 2018 feature engineering can't beat a (fully convolutional, 140M parameter, 16-layer) ConvNN.


Is the result of feature extraction a feature representation?

Yes, you are right. As a result of the extraction of features, new feature representations are created. For example, SIFT and HARR are algorithms to extract the crucial features from the images and create feature representation. CNN does it automatically with the help of convolution layers. In an image, smooth area, edge, corners are considered features that may represent it best. The importance of the smooth area, edge, and corners (there are others too) are normally accepted in ascending order.
In a simple CNN, if you can visualize the layers you can see that what it extracts from the image are mainly, corners and edge. It is also able to learn more complex features than we know (or defined). So, as you said, the convolution layers try to create a representation of features by extracting them. Then, this features representations (important information from the image) are used in the let's say prediction process.


It depends…

The general rule of thumb is that there should be at least 40 occurrences of an item to train an embedding model to find a robust representation. If most follower IDs repeat then an embedding model can learn which ones co-occur. If follower IDs are sparse then hashing (which randomly assigns numbers) is a better choice.

Which method is better is an empirical question. You can create both models, benchmark, and then choose the data processing pipeline that is best for your task.


Trying to return more than just the top result from sklearn NearestNeighbors

My guess would be that in the following piece of code:

for i,j in enumerate(indices):
  temp = [round(distances[i][0],2), clean_org_names.values[j][0][0],unique_org[i]]
  matches.append(temp)

The variable (array element):

distances[i][0]

contains the top element match for the $i^{th}$ line. Replace the second array index so it becomes:

distances[i][1]

You may want to increase one of the two zero indices in the following line as well to have a distance like value:

clean_org_names.values[j][0][0]


Convert Lasagne to Keras code (CNN -> LSTM)

You need to reshape the output of the Conv2d before you feed into the lstm. The output of Conv2d is 4d whereas the input for LSTM required is 3d.

Just print the output of Conv2d first, that will give you an idea on how to reshape the tensor as input to LSTM.


Syncfusion provides a Big data platform which is an easy to use Hadoop distribution for Windows. It can help you get started quickly. Syncfusion also provides a PMML processing library using which you can execute predictive analytic models. There is a Dashboard platform also available that can help visualize the data.

All of the above are available for free through the community license if you qualify.

Note: I work for Syncfusion.


Looking for other opinions on approach to classification problem

You should use text classification techniques. The most basic one is multinomial naive Bayes classifier with tf-idf features. for this method, take a look at this:
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html

If you don’t get enough accuracy (or maybe precision, recall or f-score), you could test more complex techniques e.g. using deep LSTM networks with word embedding. For this method, take a look at this:
https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/


build biased image dataset for emotion analysis

There is both subjective and objective approaches to removing bias ( de-baising techniques ) from the training datasets. It is observed that the sources of bias generally arise from the following data quality issues.

Over and under-sampling
Skewed samples
Feature choice/limited features
Proxies/redundant encodings
Biases and injustice in the primary data source

The essential steps for de-biasing require identifying the subspace of bias within the dataset. Then the key task is to neutralise and soften the bias if not completely equalising them.

There are interesting approaches for de-biasing image datasets. Deb-face is trying to address the problem in automated face recognition algorithms by constructing a de-biasing adversarial neural network that learns to extract disentangled feature representations for both unbiased face recognition and demographics estimation. Adversarial learning is adopted to minimise the correlation among feature factors so as to reduce the bias influence. Please refer to the approach paper in the following site.

DebFace - Debiasing Face Recognition


Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?

If your objective is to convert non-normal data into something that looks more normal / gaussian - try the Box-Cox Transform here.
It is a family of transforms that looks at your data - and provides the best possible transformation.


This should do the trick. You have to melt your data frame to use x,y and hue in your seaborn barplot.

yfreq['type'] = yfreq.index
yfreq = yfreq.melt(id_vars = 'type')
sns.barplot(x = 'variable', y = 'value', hue = 'type', data = yfreq)


Clustering (unsupervised learning) for uneven classes

You can try with DBSCAN. It's a clustering algorithm that is meant to isolate outliers. You can read more about it here. It's already available in sklearn (documentation here).

If you tweak its parameters correctly, those "extreme" observations should be identified and separated from actual clusters.


What is the difference between data-driven methods and machine learning?

Based on the quotation you have added in your comments, data-driven approaches are approaches where you use data that describes past states ("historical data") to get a (not defined) system to give a desired output.

To understand whether this definition includes machine learning or not we will have to define "machine learning", and while there could be plenty of ways to define it I expect that it will be quite difficult to come up with a definition that does not include within it "Using a system that, based on given states will give a desired output".

Note that in this last definition I use "given states" and not "past states" as to include approaches such as online learning.

Bottom line is that unless you really want to hold to a narrow definition of "past states" it seems that machine learning approaches are a subset of data-driven approaches.


Since apparently each feature is encoding something about two different categories, I would suggest that you should replace that with two features.  Your two features would be $(x,y)$ where $x$ is 0 or 1 according to whether it exists in the first category, and $y$ is 0 or 1 according to whether it exists in the second category.  In other words, instead of -1, 0, and 1, you would use $(1,0)$, $(0,0)$, and $(0,1)$, respectively.  I think that is closer to the true data and might give better results.  It might also make your results easier to interpret.

Then you can try both distance metrics and see which you find more helpful -- but it might not make a large difference.


Random forest classifier is predicting only one class even when the dataset is not imbalanced

It is not unusual that some method (given some data) predict only one class (meaning probabilities are all larger or smaller than 0.5). I had this case recently with a logistic regression application. The reason usually is that you don‘t have good features (x) to predict y. So first thing: have a look at correlation of X,y.

What you can do to improve your fit is basically to use boosting. Boosting is also tree based and similar to RF it is also an ensemble technique. But boosting gives higher weight to observations which are hard to predict. LightGBM or Catboost are good routines for boosting.

In my case, Logit predicted only one class with an AUC of about 0.3. LightGBM was much better and much more balanced in terms of prediction with an AUC of about 0.7.

You could also try Logit with L1 regulation (Lasso). Maybe some of your features are not very helpful in making predictions. Lasso shrinks these features, which helps to make okay predictions.


There really is nothing special about LSTMs when it comes to classification and metrics. So your question should be what metrics are good for multi-class classification. Both accuracy and auc works. Another metric that is good here is Matthew's Correlation Coefficient (MCC) which is similar to F1 score for a binary problem.


There are ways of understanding which of these is the right one to go with. Unforunately 10 views means no calculation is going to be great for you, but i'll try to explain both.

T-tests and Z-tests are used by a lot of AB Testing tools, as described here, to calculate winners. You'll be able to input views and clicks, and receieve back stats that describe if Group B's treatment is to win. You get back a yes/no answer, based on tolerances provided.

Bayesian calculations, on the other hand, rely on a percentage likelihood of being a winner, after which you can make a judgement.

If possible, you should run this for longer and try to drive more views to Group B. If the interaction rate of 20% persists, you'll see an eventual winner there (100% lift over the 10% rate for Group A).


Categorical data for sklearns Isolation Forrest

I coded isolation forest with dataset containing both categorical and numeric features, and it is working properly. How is it possible.?


I guess you are trying to use pandas, if so. Don't because you can't do that for what you want. I guess your operation needs all data to be loaded to the memory which is not possible. Try to use dask or a kind of sql. I'm not sure whether you want to do operations such as group by which needs all data to be loaded simultaneously or not. If you can do extra coding and your operations don't imply to load all data, such as finding the minimum of a column, you can use generators and specify the chunk_size of the read_csv method in pandas. But you have to do extra coding. You can also take a look at here. You can also take a look at here to figure out why sql and dask are more better for large operations.


How to Split And Resample Imbalanced Dataset Into Train, Validation and Test

Grzegorz's answer gets to the root of the problem - if the future data this model will be used to make predictions on will have the same distribution, then stratification by class % makes sense.

One thing I wanted to add is I typically use the normal train_test_split function and just pass the class labels to its stratify parameter like so:

train_test_split(X, y, random_state=0, stratify=y, shuffle=True)

This will both shuffle the dataset and match the %s of classes in the result of train_test_split.


Cleaning the univariate dataset with high noise

There are multiple algorithms for this. On one of the kaggle competitions I saw applying several algorithms and then throwing the data away by the number of votes from all algorithms. The algorithms that I remember:

Remove all points above or below 2 standard deviations. X>mean+2*std dev, X < mean-2*std dev. 2 is actually a parameter you need to optimize.
Remove the 5% of the highest and lowest values. X>95% quantile, X<5% quantile. 5% is a parameter you need to optimize.
X - median > 2*1.4826*MAD (median absolute deviation. 1.4826*MAD is an estimator for standard deviation under some assumptions). And the same for lower bound. https://eurekastatistics.com/using-the-median-absolute-deviation-to-find-outliers/

Repeat several times until no points are thrown away.

Example of this method with voting from some other kaggle winners: https://nycdatascience.com/blog/student-works/kaggle-predict-consumer-credit-default/


Benefits of using Deep Learning-specific hyperparameter optimization tools vs. sklearn?

In my case with hyperas I noticed one of the distinct advantage over gridsearch, that is, gridsearch function takes only one array as input. My requirement was two be able to send two array as input as I am working with siamese network. I could do it with hyperas out of the box. So hyperas is more flexible than gridsearchcv.  Check this example


Understanding the softmax output in Youtube's recommender

Part of the confusion comes from different prediction processes used for training and serving.

Softmax is used during training to make predictions for the 1 million video categories. The softmax makes the training of the neural networks gradients easier and works well within deep learning libraries.

Those 1 million video categories also happen to be embedded into 256-dimensional space. During serving, prediction is now a nearest neighbor search. Nearest neighbor search is faster, since it is a dot product operation and can use general purpose libraries.


Estimate the normal distribution of the mean of a normal distribution given a set of samples?

The same can be done through any of the statistical estimation technique such as maximum likelihood or Minimum variance unbiased estimator(MVUE). Generally maximum likelihood is the easiest way to evaluate the mean and variance of the distribution. If you perform Maximum likelihood method the estimated mean of the normal distribution is the sample mean. Given x1,x2.....xn are the samples

Estimated mean u = (x1 + x2 + ... + xn)/n
Similarly the estimated variance is  Σ(xi - u)^2/n

Both of these are obtained by setting the partial derivative of the likelihood function with respect to the mean and the standard deviation to zero.

Hope this helps.


Why not train the final model on the entire data after doing hyper-paramaeter tuning basis test data and model selection basis validation data?

The question is under a wrong assumption. Many people do what you say they "cannot" do.

In fact, the grid search implementation in the widely deployed sklearn package does just that. Unless refit=False, it will retrain the final model using the entire data.

I think for some hyperparameters this might not be very desirable, because they are relative to the volume of data. For instance, consider the min_samples_leaf pre-pruning tactic for a decision tree. If you have more data, the pre-pruning may not perform as you want.

But again, most people do in fact retrain using the entire data after cross-validation, so that they end up with the best model possible.

Addendum: @NeilSlater says below that some people perform hold-out on top of CV. In other words, they have a train-test split and then perform model selection on the training. According to him, they re-train using the original training set split, but not the testing set. The testing set is then used to perform a final model estimation. Personally, I see three flaws on this: (a) it does not solve the problem I mentioned with some hyperparameters being dependent on the volume of training since you are re-training anyway, (b) when testing many models, I prefer more sophisticated methods such as nested cross validation so that no data goes to waste, and (c) hold-out is an awful method to infer how a model will generalize when you have little data.


In Conditional Random Fields, is mandatory to use features related to following and preceeding tokens?

In other words, do we need these features related to other tokens?

No, these features are not needed. But they are often useful: CRFs handle sequential dependencies between the labels, however it's up to you to provide the relevant features, in particular some to represent the dependencies between (certain) features if needed.

With text, this relation between consecutive tokens is very often a relevant indicator. I'd suggest that you try both version, without and with this feature, and you're likely to observe a higher performance in the latter case. In my experience it's often worth trying different combinations, including trying features which go two or three steps back.


Estimating test AUC using k-fold CV for imbalanced classification problem

The test AUC is the AUC you find after predicting the held out test set (dataset B).

You have split your dataet into A = training set and B = test set. You then used downsampling (which is questionable, but I won't get into this) on dataset A to get what you refer to as dataset C. You should then train your classifier on dataset C and evaluate its performance on dataset B.

If what you are referring to is a validation set (i.e. not the test set = dataset B) then what you need to do is as follows: split your dataset into A = outer train, and B = test. Then, split A again into C = inner train, D = validation. Downsample dataset C, train your classifier on C only, then validate on D. Once you have optimized to D, then downsample A, retrain your classifier on A (using the exact same model building process as you did with C and D), and report your final, unbiased performance measure on B. Repeat this entire process if AUC scores are volatile to get more estimates of final model performance.

Since you have unbalanced data, I strongly recommend stratified sampling when creating your validation and test sets.


First you need to learn about Logistic Regression, it is an algorithm that will assign weights to different features given some training data. Read the wiki intro, is quite helpful, basically the Betas there are the same as the Ws in the paper.

The formula you have is correct, and those value do seem off. It also depends on the number of significant figures you have, perhaps they are making their calculations with more than the ones they are reporting.

But honestly, you can't understand much of the paper unless you understand LR


How to utilize user survey answers and the actual usage in forecasting power usage using LSTM?

Check out the example "Multi-input and multi-output models" here in the documentation. It gives an example how to combine an LSTM time series model with additional input. You would use separate input layers for the power usage and for the survey data. After running the power usage data through the LSTM you combine the results using "concatenate".

(You can ignore the Embedding layer in the example since your power usage data isn't text data.)


LabelEncoder with a Multi-Layer Perceptron?

Your colleague was correct that LabelEncoder is meant for the target variable (a.k.a. labels); more-or-less equivalent OrdinalEncoder is meant for the independent variables.

IMO, your colleague is partially correct about an ordinal encoding "not making much of a difference" for a neural network.  A neural net can contort its way into a correct understanding of your labels anyway, although I think an ordinal encoding will make its job substantially more difficult, so I still wouldn't advocate for it.  The intervening steps, scaling and PCA, make it less clear exactly what the neural network will actually see, and I don't know if that alleviate or worsen the issue of an ordinal encoding.  (Given your good score, perhaps alleviate.)


A simple group-by sum will solve your query:
df.groupby(['Genre'])['Global_Sales'].sum()


The goal of decision trees is to partition the feature space into successively smaller regions where each region is best characterized by a single label or value. Adding cyclical components would not help accomplish the goal, it would unnecessarily repeat the modeling fitting for a given region.


Does tensorflow pick samples from a dataset randomly or sequentially when training?

Welcome to the site! My understanding is that it does it sequentially. It's usually a good idea to shuffle your dataframe as a pre-processing step and then train on that.


Data Visualisation Techniques for Multi Labelled data

Are you looking for a Python or R package? Or would a tool like Tableau suffice?

If you feel comfortable using a Python package, seaborn has the ability to visualize multi-label classes.

Take a look at this link: https://towardsdatascience.com/journey-to-the-center-of-multi-label-classification-384c40229bff

Here, there's a good explanation of how to get this done.

However, if you want to use a tool to get your data visualized, look no further than Tableau. However, if you wish to do a lot of machine learning, this wouldn't be a right fit. Else, Tableau can visualize multi-label classification data. All you have to do is plug in your dataset as a data source and navigate to whichever chart preview suits your dataset. Also, Tableau neatly sorts your attributes into dimensions and facts; sometimes erroneously  might I add. In that case, all you have to do is a simple right-click and convert it to a dimension or vice versa.


How to increase the weight when it comes to outlier detection

Please look at my answer at Identfying spikes in data
.

A general method could be to calculate a (moving) average and standard deviation.
Outliers are points that differ more than 3 times the standard deviation.

In your case, B will probably have a standard deviation of 3%, because 10% is considered an outlier. For A, C, D, E, the standard deviation may be 7%, because 20% is not considered an outlier.

Relative importance does not make something an outlier. A large deviation does.


Image clustering by similarity measurement (CW-SSIM)

I would use a regular clustering algorithm and replace the objective function, which is usually the MSE, with a differentiable loss function of your choice. Another way is to learn an embedding that optimizes your similarity metric using a neural network and just cluster that.

If you would rather do similarity-based clustering, here are some papers:

A Similarity-Based Robust Clustering Method
A Discriminative Framework for Clustering via Similarity Functions
Similarity-Based Clustering by Left-Stochastic Matrix Factorization

sklearn implements two similarity clustering methods: Affinity propagation, and spectral clustering.


Finding orthogonal input patterns associated with logistic function output

All you have to do is complete $\omega$ to an orthogonal $\mathbb{R}^3$ basis, as $\omega^T V = 0$ and $v_1$, $v_2$ are orthogonal. You can do this using Gram Schmidt orthogonalization. The two other vectors of the basis will be $v_1$ and $v_2$.

Edit:

The initial vectors to do the Gram Schmidt process can be any pair of independent vectors, for instance $(1, 0, 0)$ and $(0, 1, 0)$.


You can think of binary encoding as a compromise between label encoding and one-hot encoding. For distinct categories, label encoding introduces a false linear order that brings a lot of noise into the model (category 1 < category 2 < category 3....) . Binary encoding introduces false additive relationships between the categories (e.g. category 4 + category 1 = category 5 or 100 + 001 = 101) but fewer of them.

Therefore, binary will usually work better than label encoding, however only one-hot encoding will usually preserve the full information in the data.

Unless your algorithm (or computing power) is limited in the number of categories it can handle, one-hot encoding will be preferred over other encoding schemes. If you are limited, mean encoding is a powerful alternative because it transform a categorical feature into a numeric one (giving you the minimal number of inputs) while preserving the most important information in the data.

[Mean encoding replaces every category with its target mean. The mean encodings need to be constructed carefully on a separate dataset to avoid data leakage. If you want to reduce your input dimensions as much as possible, this can be helpful. Mean encoding can also be helpful as an additional feature and is very popular on Kaggle to squeeze out some extra performance. This is also known as target encoding or likelihood encoding.]


Is it good practice to convert columns with a number to a range between 0 and 1?

I think there si some confusion with the quantile transformer : https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html#sklearn.preprocessing.QuantileTransformer it actually scale the values to a 0-1 range. the goal is to get a uniform distribution. It serves a different purpose than minmax scaler imho. Notably it could be helpfull if you get outliers.

To get a bigger overview on what different scalers do :  https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py


data processing, correlation calculation

Pandas is the best thing since sliced bread (for data science, at least).

an example:

import pd
In [22]: df = pd.read_csv('yourexample.csv')

In [23]: df
Out[23]:
   user   item1   item2
0     a        2      4
1     b        1      3
2     c        5      6

In [24]: df.columns
Out[24]: Index([u'user ', u'item1 ', u'item2'], dtype='object')

In [25]: df.corr()
Out[25]:
          item1      item2
item1   1.000000  0.995871
item2   0.995871  1.000000

In [26]: df.cov()
Out[26]:
          item1      item2
item1   4.333333  3.166667
item2   3.166667  2.333333

Bingo!


How to select random data for two different recommender systems?

Welcome @Remus Raphael :) - Your approach is a sound option.

More specifically, if a density-based algorithm was already working for you, I'd recommend the HDBSCAN clustering algo, which should have a better performance and has a unique built-in cluster validation (based on the DBCV algorithm).

Then your general pipe line could be:

An optional pre-processing
An optional NLP / TFIDF for meaningful text features
An optional dimensionality reduction (I found TSNE and TruncatedSVD to
work nicely with HDBSCAN with textual processing)
HDBSCAN tuning for different params and distance metrics
Finally, when you're satisfied with the clustering quality - you can simply start with the 2 largest clusters for your AB testing

I'd love to hear you feedback on your actual data :)


In which cases are non-linear learning methods preferred than logistic regression in classification problems?

There are several assumptions that are needed to be satisfied in order for logistic regression to work. From personal experience the two most important assumptions:

Features need to be not (or a little) correlated with each other, because if they are correlated then the change of one feature indicates also the change of the other that's a problem when you try to solve the optimization problem
May some features not be ordinal

You can also check here for more information.

Other learning methods such as RandomForest require no such assumptions and may work better for some datasets. But keep in mind the no free lunch theorem, most of the times you need to get your hands dirty and test several learning models before you decide which works best for your data.


DIfferent learning rates converging to same minima

This is the expected behavior. Different learning rates should converge to the same minimum if you are starting at the same location.

If you're optimizing a neural network and you want to explore the loss surface, randomize the starting parameters. If you always start your optimization algorithm from the same initial value, you will reach the same local extremum unless you really increase the step size and overshoot.


Why does the listed order of features specified in the data set matter to the random forest classifier

Since your fixed the seed, with default option bootstrap=True each tree is build on a fixed subsample of your data. With the default option max_features='auto' each tree uses 2 features. Since your fixed the seed, it is always the same, let us say, "1st and 4th" features for the first tree. And they are different in each of your shuffles. So, the tree is different. The same applies to each tree.

By the way, with 4 features in total, only 6 (4*3/2=6) various trees are possible. With n_estimators=10 some of these trees are necessarily repeated.
And in each shuffle those are different trees.


try to add more conv layers, and also play a bit with hyperparameters to improve accuracy.
Currently, the learning rate is being used as default, try to tweak it from 0.01 to 0.0001.


How should I evaluate writing quality to compare two articles(which article is better suited/written for a topic ) according to their content?

"Source credibility" of Internet articles is best calculated through the Page Rank algorithm.

Algorithmically determining writing quality might be intractable. However Page Rank could be a proxy. If an article is a hub then it is the authority on the topic and can be assumed well written (or at least very useful).


Pros and Cons of Python and R for Data Science

Interaction - Random Facts

Both are good stable languages with interesting complementary qualities. You can get much better packages in one and then stitch them with some data from the other. An example is using time series forecasting and decision trees in R and doing data munging in Python.

Both languages borrow from each other. Even seasoned package developers like Hadley Wickham (Rstudio) borrows from Beautiful Soup (python) to make rvest for web scraping. In addition to that, Yhat borrows from sqldf to make pandasql and many other.

Rather than  reinvent the wheel in the other language developers can focus on innovation because, in the end, the customer does not care which language the code was written, the customer cares for insights.

Mixing Them Up
AM mentioning few approaches to mix them together-

Use a Python package rpy2 to use R within Python . [Demo]

Use Python from within R using the rPython package. [Demo]

Use Jupyter with the IR Kernel. Python and R and makes the interactivity of iPython available to other languages.

Use Beaker Notebook. It allows you to switch from one language in one code block to another language in another code block in a streamlined way to pass shared objects.

Python vs R
Python vs R - This section will answer:

Which will be better?
How to choose one over other?
Specialization

See as I said earlier both are stable and you can choose any or work with both. But when it comes to master one I'll suggest keep these 3-4 guidelines in mind-
Personal Preference
Choose the language to begin with based on your personal preference, on which comes more naturally to you, which is easier to grasp from the get-go. To give you a sense of what to expect, mathematicians and statisticians tend to prefer R, whereas computer scientists and software engineers tend to favor Python.
Project selection
You can also make the Python vs. R call based on a project you know you’ll be working on in your data studies. If you’re working with data that’s been gathered and cleaned for you, and your main focus is the analysis of that data, go with R. If you have to work with dirty or jumbled data, or to scrape data from websites, files, or other data sources, you should start learning, or advancing your studies in, Python.
Collaboration
Once you have the basics of data analysis under your belt, another criterion for evaluating which language to further your skills in is what language your teammates are using. If you’re all literally speaking the same language, it’ll make collaboration—as well as learning from each other—much easier.
Job market
Jobs calling for skill in Python compared to R have increased similarly over the last few years.

Note: Have a look at this infographic by DataCamp. For a better view on it.

My Rationale
In my case am doing both and using them interactively and Customizing them as per my use. You can get something really interesting in one (as I mentioned above) which will be hardly available in other, so it's better to use both together. This is the best way to bridge the gap between these two.
But in the last, it's your call keep the guidelines, your interest, and scenarios in mind and make a clear view on that.
Strength & Weaknesses
R
Strength

R is great for prototyping and for statistical analysis.
It has a huge set of libraries available for different statistical type analysis. Check The Comprehensive R Archive.
RStudio IDE is a definitely a big plus. It eases most of the tedious tasks and fastens your workflow.

Weaknesses

The syntax could be obscure sometimes.
It is harder to integrate to a production workflow.
In my opinion, it is better suited for consultancy-type tasks.
The libraries documentation isn't always user friendly.

Python
Strength

Python is great for scripting and automating your different data mining pipelines. It is the de facto scripting language nowadays.
It also integrates easily in a production workflow. Besides, it can be used across different parts of your software engineering team
(back-end, cloud architecture etc.).
The scikit-learn library is awesome for machine-learning tasks.
Ipython (and its notebook) is also a powerful tool for exploratory analysis and presentations.

Weaknesses

It isn't as thorough for statistical analysis as R, but it has come a long way these recent years
In my opinion, the learning curve is steeper than R, since you can do much more with Python.

The Conclusion
Use R and Python. Learn how they inter-operate together. Start with one and then add the other to your workflow. As I like to remind myself- "choosing the tools should never be the primary problem". When in doubt, use the one that is available and that gets the work done quickly.
Hope it helps!
Ref- Udacity, Quora, Letustweak, kD, DataCamp


Delete all rows between two values in a dataframe that repeat multiple times in a column

You can achieve this by forward filling the blank values, and then selecting only those with new_records:

df.fillna(method='ffill')

df = df[df['column_A'] == 'new_records']

Depending on the actual values in your data frame, you may need to first replace what appear to be empty/space strings with NaN's:

df['column_A'] = df['column_A'].replace(r'^\s*$', np.nan, regex=True)


What should I use if I have millions of possible values for a feature in a sklearn predictive model?

Unfortunately, AFAIK no sklearn model supports categorical variables.

For instance, sklearn decision trees only support rules such as X<n, not X==n which would be desirable here.

Also, the decision tree algorithm they implement only produces local one-look-ahead optimization. What this means is that, it may not produce rules such as X<n followed by X>n-1, even if such rule would be highly desirable.

In the end you will end up with non-sense things like: Car > 1 and then Car < 6 where 1 is Volkswagen and 6 is Ferrari.

The typical workaround is to use one-hot encoding, which might be a pain in the ass in your case. And then, it might not: sklearn decision tree supports sparse matrices, so the memory penalty would be low. You can use scipy for this. (Sparse matrices stores data differently than regular matrices. Instead of requiring $n\times m$ size, the memory requirement is proportional to the number of non-zeros in your matrix.) In terms of speed, it should not be any different than if the algorithm natively supported categorical variables.

This being said, your data may not allow for use of sparse, if the rest of the features are non-sparse. I don't think scipy has support for a sparse-dense hybrid matrix.

Another workaround I can think of would be to produce an Euclidean distance matrix between observations of the various categories (you may want to normalize first). Then group categories that are close together. Then build a hierarchical model, where you predict the final category for each category. In python, this is easier than it seems. You can create a class for your model using sklearn base classes.

I love Python and sklearn. But I believe in using the right tool for each job. I would use R which has native support for categorial variables (they call them factors) and has a plethora of decision tree packages. (Note: xgboost for R does not support categorial variables, it ignores the factor class-type.) Weka could also be a good tool, which also has very powerful decision tree algorithms.


Machine learning technique to calculate weighted average weights?

Yes it is definitely possible to calculate optimised weightings provided you have some training examples where you know the document fields, the query, and either the outcome (relevant/not-relevant) or the desired score.

I think your training feature set should be the query score in range [0.0,1.0] for each field of each example. The training label should be either relevance 0 or 1 for each example, or the relevance score that the example has.

If you have a target score for each example

You want to determine the weights $W_i$ to use for each field $i$. Your calculated relevance score would be $\hat{y} = \sum_{i=1}^{N_{fields}} W_i * X_i$ where the caret signifies this is the estimate from your function and $N_{fields}$ is the number of field. Note I am ignoring your original idea of dividing by the sum of all $W_i$, because it makes things more complex. You can either add that term or force the sum to be equal to 1.0 if you wish (I am not going to show you how though, as this answer would get too long, and it probably won't help you much)

With a target score and training data, the simplest approach is to find the weights which cause the lowest error when used with the training data. This is a very common goal in supervised learning. You need a loss function. Having a target scalar value means you can use difference from target and a very common loss function for this kind of regression problem is the mean squared error:

$$E = \frac{1}{N_{examples}} \sum_{j=1}^{N_{examples}} (\hat{y}_j - y_j)^2$$

Where $\hat{y}_j$ is your calculated relevance score for example $j$ and $y_j$ is your training label for the same example.

There are a few different ways to solve for lowest $E$ with this loss function, and it is one of the simplest to solve. If you express your weights as a vector $W$ length $N_{fields}$ your example features as a matrix $X$ size $N_{examples} \times N_{fields}$ and the labels as a vector $Y$ length $N_{examples}$ then you can get an exact solution to minimise loss using the linear least squares equation

$$W = (X^TX)^{-1}X^TY$$

There are other approaches that work too - gradient descent or other function optimisers. You can look these up and see which you would prefer to use for your problem. Most programming languages will have a library with this already implemented.

Note that you will likely get scores greater than 1.0 or less than 0.0 from some document/query pairs.

You will have to use a adjust the technique if you want to divide by total of all weights or want sum of all weights equal to 1 in your scoring system.

If you have a relevance 0 or 1 for each example

You have a classification problem, relevant or not are your two classes. This can still be made to work, but you will want to change how you calculate your weighted score and use logistic regression.

Your weighted score under logistic regression would be:

$$\hat{y} = \frac{1}{1 + e^{-(b + \sum_{i=1}^{N_{fields}} W_i * X_i)}}$$

Where $b$ is a bias term. This looks complicated, but really it is just the same as before but mapped by a sigmoid function to better represent class probabilities - the result is always between 0 and 1.

You can look up solvers for logistic regression, and most stats or ML libraries will have functions ready to use.

Caveats

You have made a starting assumption that a simple combined relevance score will lead to a useful end result for your users performing search. This has led to simple linear models looking like a good solution. However, this may not be the case in practice, and you may need to re-visit that assumption.


What can I do with this information?

You can do a lot of things with this data. You can visualize it, you can use the vectors for prediction or regression, whatever the task at hand. However, there are a few restrictions of PCA that you need to keep in mind. For eg. its very memory intensive, so you need to have a "lot" of RAM to use PCA on certain data-sets.

How will this help me understand the data?

You can visualize the data like this (image is taken from http://www.nlpca.org/pca_principal_component_analysis.html):

With reference to the above image, you can see that the data-points can be clearly separated into different clusters. Using this, you can apply K-Means and get different cluster centres. Using these cluster centres, you can further investigate and find additional insights.

Is it useful per see?

PCA is a dimensionality reduction technique which is very memory intensive.
If you have the required memory, you can easily reduce the number of features by 50-80 %, while still retaining a good amount of information. For eg, we can reduce 100 features to 20-30 features which contain maximum amount of information.
While performing PCA, it's important to check if the matrix computation can be done with the RAM that you have, otherwise, you can check out Iterative - PCA.

Is it useful as a preparation method for other methods?

It is very useful as a preparation method for clustering and visualization.
Please refer this link : https://qiita.com/bmj0114/items/db9145a707cb6ed13201

Which ones can I try?

You can try the example given in the link above.


You could break the heading ($\theta$) into components: $sin(\theta)$ and $cos(\theta)$, which would remove the modulo problem.


Analyze performance Poisson regression model on a time series(count forecasting)

I'm not sure what you mean by "performance", but if what you mean is fit the answer is clear. You need to be using the log-likelihood to differentiate between different models. Basically, when you are fitting the model you are trying to maximize the log-likelihood. Thus the log-likelihood is giving you some sense of how well the parameters of your model are doing at fitting the data.

In your case, you want to get log-likelihood to be as close to zero as possible. Now this is kind of terrible advice, because if you were clever enough to come up with a feature for every observation, you could get a perfect fit. That's bad because your model would be completely useless. There are functions that take your log-likelihood as an input and transform it to penalize you for adding more variables, etc. We won't worry about those right now. Just keep in mind not to mindlessly lower the log-likelihood.

Once you have a model that you can live with you should run some sort of cross-validation, and/or hold out set. Then you can use any number of metrics to validate the predictive performance of your model. I think that is the more important of the two issues. You could calculate the mean square error on your hold out set. $$MSE=\sum_{i=1}^n(\hat y_i - y_i)^2$$
This would give you a really basic metric to assess how well your model is predictive of the output.


How to make forecast in LSTM with specific/defined feature values

I'm not really sure if I understood you correctly, but let's talk about it.

You are trying to predict only one value, that is price, right?
And what is your feature set? is it only date, type and region?

Remember, to predict a value you should always have known feature set. If you have a feature that is not known, then this is not the right way to do it.

What you can also do is, predict multiple values. Not sure if this is helpful in your case, but you can use something like MultiOutputRegressor to predict multiple times based on a model. So you can have date, type and region as features and predict anything else that is missing. Be aware though that this would usually reduce the prediction accuracy.

See here: sklearn.MultiOutputRegressor

So if you want to predict for future dates, based on previous dates. You can either have feature sets of one month data, and always predict for the next one month. So you have a feature set of 30 days, and you predict 30 times, for each days in the future.

Or you can predict only for one day, and train with your predicted data, then predict the next day and so on. So one by one.

You should try both of these ways to see which one would give you a better results.

Hope these helps.


Should I scale high ranging ordinal fields?

We scale data because certain algorithm will not work optimally esp. Gradient descent.
You may check the internet for further detail.
Coming to the exact question -
I am assuming, you have done the analysis that the Oridinal feature will be used as a Continuous(Not Categorical) feature with its respective values. So, I will ignore this point
Models don't see the feature scale, it looks for the respective variance, interaction with other features, Correlation with the target etc.(in a very simple language) and scaling will not hamper these parameters.
So, you must scale if that is your only concern.
 You may ignore it for some Models e.g. Decision Tree/RF will work fine even without scaling.


What are good practices in reporting RMSE or MAPE estimates for a machine learning model?

In general you should not attemt to make your model look better than it actually is. Always stick to the truth. Some ideas:

Outliers: It is not okay to simply drop the outliers. If you want to make a prediction or do inference, you have to ask yourself how relevant such outliers are in practice. Where do they come from (are data reliable or is it an error)? If they are part of the true bigger picture, you cheat if you just drop them. If they are a very special part of the market with specific attributes, you could try to control for these special types (e.g. by adding indicators or changing the parametric form of your model). This question essentially is data and method driven.
Your metrics should of course reflect the final model!
By definition of the RMSE it is the mean (not median). Naturally, you can describe the situation by discussing the influence of the outliers in your own way. But this will not make your model/fit better after all.

Finally: I think you should NOT ask about a metric that suits your approach and make it look better. You should ask yourself if you can improve your model. You did not specify what type of method you use and what data you work with, but I'm quite sure there are ways to improve the fit. The scatter plot looks like an L-shaped function. So this might be the first thing to start to work on!?


Expectation number of points in initial clustering for LSH

If you want to compute (approximate) nearest neighbors in 10-dimensional data, I would recommend using a k-d tree.  A k-d tree can handle skewed/non-uniform data sets and adjusts naturally to that.

More generally, you could look at data structures for nearest neighbor search, including binary space partitioning trees and metric trees.  But k-d trees are a good starting point, and they support both exact nearest neighbor queries as well as approximate nearest neighbor queries.

I would expect this to work better than locality-sensitive hashing (LSH).


Saving the hidden layers of a trained network

There are a few mistakes. The following is the way u need to change your code. I am not sure what your model class consists  but I am hoping it is class function with .classifier consisting the network.

def load_checkpoint(model, path):
    checkpoint = torch.load(path)
    model.classifier.load_state_dict(checkpoint['state_dict'])
    epochs = checkpoint['epochs']
    model.class_to_idx = checkpoint['class_to_index']
    return model, epochs

classifier = nn.Sequential(nn.Linear(25088, 512),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(512,350),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(350,250),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(250,102),
                           nn.LogSoftmax(dim=1))

model.classifier = classifier

test_model, _ = load_checkpoint(model, 'trained_model.pth')
print(test_model)

Similarly you need to load optimizer, scheduler and other attributes if you want to retrain the network.

if your intention is to load only part of the network instead of the entire network. Something as below

classifier = nn.Sequential(nn.Linear(25088, 512),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(512,350),
                           nn.ReLU(),
                           nn.Dropout(p=0.2),
                           nn.Linear(350,250))

model.classifier = classifier
weights = torch.load("weight.pth")
state_dict = model.classifier.state_dict()

for name, param in model.classifier.named_parameters():
    if name in state_dict.keys():
        state_dict[name] = weights["state_dict"][name]

model.classifier.load_state_dict(state_dict)

In this way you can only load the part of the network required.


I would impressed if there is already a dedicated 9gag clustering :P

However, you can read this blog post about Hierarchical clustering in Python for images, which is close to what you want. The problem is that the author uses the average color of the image as a feature and it could proved crude and not inefficient. You may find something more interesting to use. But in the end, you need to experiment a lot with your own dataset.


Mathematically, it is not possible to have the area under the curve equal to 1 if you have a precision less than 1... If you have a doubt, you should check the confusion matrix and calcuate the scores yourself. You'll know directly if you did a mistake or not

A few other remarks:

Accuracy is not precision
The loss is of no help here


How to choose between classification Vs regression approach?

I would say that the main thing is what information you think is the most valuable. There are some differences.

The benefit with a regressor is that it can give you a number of how much profit you can expect. That is obviously useful when considering a new product. The drawback is that it cannot tell you how likely you are to see this profit. It will not tell you if a product is risky.

Many classifiers can tell you with what probability it thinks a product will be profitable. This can alert you to potentially risky products. But the drawback is that you won't get to know how big the potential profit is.

As I said, it depends what you value most. But nothing is stopping you from training two models and using information from both of them to make your decisions.


In your example, you should calculate the probability that the value goes up, and use that as your $p$.

That said, I think the calculation should be $\sigma = \sqrt{p(1-p)}$ otherwise, your standard deviation is inflated by a factor tied to $n$.  By using probabilities, you are essentially taking $n$ into account because:
$$p = {Number\ of\ "Up"\ instances \over n}$$
Where $n$ is the total number of observations.

Given the nature of the data, that is, it is binary, we would expect the $\sigma$ to be between 0 and 1 and not particularly sensitive to the number of observations.

That said, if we look at a few calculations, the value of $n$ in your formula significantly impacts the $\sigma$.

Assume for simplicity that $p$ is 0.5, that is an equal chance of going up or down.  If you have three different values of $n$ you get three vastly different values for $\sigma$.

Example 1: $n$ = 1 (Or not included)
$$\sigma = \sqrt{0.5 * (1-0.5)},\ \therefore \sigma = 0.5$$

Example 2: $n$ = 100
$$\sigma = \sqrt{0.5 * 100 * (1-0.5)},\ \therefore \sigma = 5$$

Example 3: $n$ = 1000
$$\sigma = \sqrt{0.5 * 1000 * (1-0.5)},\ \therefore \sigma = 15.8$$

Of the three, only example 1 really makes any sense intuitively. The mean of all instances would be 0.5 and you would expect the observed values to be 0.5 away from the mean.