This should work
def findUntilNotSame(array1,array2):
    resultPos = 0
    for i in range (len(array1)):
        if(i < len(array2)):
            if(array1[i] != array2[i]):
                break
            resultPos += 1
    return resultPos

# find the length of same value
def findStartSame(array1,array2):
    for i in range(len(array1)):
        if array1[i] in array2:
            return [i,array2.index(array1[i])]

# find the solution one time
def findStartAndLength(array1,array2):

    start = findStartSame(array1,array2)
    start1 = start[0]
    start2 = start[1]
    length = findUntilNotSame(array1[start1:],array2[start2:])
    return [[start1,length],[start2,length]]

def updateAnswer(finalOrigin):
    finalResult=finalOrigin
    for i in range (len(finalResult)):
        if i != 0:
            finalResult[i][0][0] += finalResult[i-1][0][0] + finalResult[i-1][0][1]
            finalResult[i][1][0] += finalResult[i-1][1][0] + finalResult[i-1][1][1]
    return finalResult

def algorithm(array1,array2):
    i, j = 0, 0
    finalResult = []
    while(i<len(array1) or j<len(array2)):
        array1now = array1[i:]
        array2now = array2[j:]
        if(any(item in array1now for item in array2now)):
            result = findStartAndLength(array1now,array2now)
            finalResult.append(result)
            nextStart1 = result[0][0] + result[0][1]
            i += nextStart1
            nextStart2 = result[1][0] + result[1][1]
            j += nextStart2
        else:
            i += 1
            j += 1
    return updateAnswer(finalResult)


For text match problem, what is the different between question-question match and question-answer match?

You can refer to the paper "A Deep Look into Neural Ranking Models for
Information Retrieval" to get more discussion about different matching tasks.


The first step would be to find how similar a candidate number is against any number in the reference list. I think this is a perfect case for a character-based string similarity measure, typically the Levenshtein edit distance.

In case it's possible to have several matches, there could be a second step which would predict the most likely match, maybe based on the frequencies of the number.


Can numerical discrete finite data be always treated also as categorical?

I would separate value with representation in this case.

Energy as you mentioned, in the real world, holds a very continuous value.  However, we may choose (for various reasons) to represent this value in different forms.

We can take values as they are (15.21252, 23.76535), we can round them into integers (15, 24), we can even decide to represent this data by clusters ("UNDER 20", "OVER 20").

Technically, all data can be represented as categorical data, we need to consider what value does this data represent, and what are we losing/gaining from using different representations.


Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

There is no free beer in such matters so "automatically takes care of the danger of overfitting the data" is not true. When you optimize for a loss error function and also for simplicity the immediate question is how do you trade off simplicity on loss error. There is no answer to that, other than in very specific cases (but those cases are so constraint so they can't provide a generic answer).
This is the reason why many models which allows you to do this trade off will give you a parameter or more to use as knobs for this trade off. Examples are logistic regression with regularization, neural nets with dropouts, decision trees with max tree levels, boosting models with learning rates, and how many more. Even if the model does this trade off automatically, it does not mean it is the best trade off, mostly it is some trade off incorporated into the model structure and its assumptions.
The second aspect of your question is that you do not consider the whole scope of cross validation and of similar medthods in general. CV is used to assess the performance of different models, either if they have the same structure but with different parameters, either they are from different families. CV allows you to have a reliable hind regarding which will be the error of a model if used on similarly distributed new data. This is not covered by regularization at all.


Confusion Matrix vs AUC Confusion with Random Forest

From your confusion matrix it actually looks like your class imbalance is over-favoring the positive class (90.04% to 9.96%). Because you have such an extreme class imbalance in your data your random forest is likely just classifying almost everything as positive class and calling it a day.
To solve this issue you need to either down-sample your larger class or up-sample your lesser class to reach closer to 50/50 split. Otherwise you will likely just end up with a classifier that grossly over-predicts the positive class.
When you are splitting your data you can also try stratifying by your label (so that the same proportion of positive to negative class exists in your training and testing sets).


How does combining two linear perceptrons create non-linear boundaries?

You are correct that stacking two layers with a linear activation function on top of each other does not do anything that a single layer could not do (i.e. it is still a linear combination of terms).

This changes, once you use other activation functions. Then, once you combine neurons from the previous layer in the next layer e.g. as $w_0 + w_1 f(e_1) +w_2 f(e_2)$ for some suitable activation function $f$ such as ReLU, the sigmoid function etc., and then stack several such layers on top of each other, you can get substantially more complex functional relationships.


System to provide guide to students about getting admissions to universities of their choice or some specific courses

First of all, interesting question!

Most important thing to recommend you a starting point is to know what kind of data you have in hand. So my answer is regardless of that fact.

If you have a road-map of studies of students who got into University XXX then you can use model it with an HMM (Hidden Markov Model).

In this case the observations are the sequence of courses one passed and you want to calculate the probability of landing in desired university. Transition matrix is the probability of getting course A after getting course B. Emission matrix might be something like how successful they were in the course. Transition and emission matrices can be scored using facts like dropping a course or failing or the education track they exhibit in.


You have to write a custom callback for this. Steps are :

Subclass ModelCheckpoint   (https://github.com/keras-team/keras/blob/master/keras/callbacks.py) or create new one if you do not need filename pattern etc.
Add method that would be called at the end of each batch

class BatchModelCheckpoint(keras.callbacks.Callback):
     def on_batch_end(self, batch, logs=None):
        self.model.save(filepath, overwrite=True)


LSTM layers require data of a different shape.

From your description, I understand the starting dataset to have 3125 rows and 1000 columns, where each row is one time-step. The target variable should then have 3125 rows and 1 column, where each value can be one of three possible values. So it sounds like you're doing a classification problem. To check this in code, I would do:

>>> X.shape
(3125, 1000)

>>> y.shape
(1000,)

The LSTM class requires each single sample to consist of a 'block' of time. Let's say you want to have a block of 100 time-steps. This means X[0:100] is a single input sample, which corresponds to the target variable at y[100]. this means your window size (a.k.a number of time-steps or number of lags) is equal to 100. As stated above, you have 3125 samples, so N = 3125. To form the first block, we unfortunately have to discard the first 100 samples of y, as we cannot form an entire block of 100 from the available data (we would end up needing the data points before X[0]).

Given all this, an LSTM requires you to deliver batches of shape (N - window_size, window_size, num_features), which translates into (3125 - 100, 100, 1000) == (3025, 100, 1000).

Creating these time-blocks is a bit of a hassle, but create a good function once, then save it  :)

There is more work to be done, perhaps look at more in depth examples of my explanation above here... or have a read of the LSTM documentation, (or better still, the source code!).

The final model would then be simple enough (based on your code):

#Create model
model = Sequential()
model.add(LSTM(units=32, activation='relu',
               input_shape=(100, 1000))    # the batch size is neglected!
model.add(Dense(3, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam',
              metrics=['accuracy'])

Have a look at the documentation regarding input shape for the Sequential model. It basically says that we don't need to specify the number of batches within input_shape. It can be done with e.g. batch_size=50, if you require it to be a fixed number.

I know the input_shape argument is not in the documentation for LSTM, but the class itself inherits from RNN, which in turn inherits from Layer - so it will be able to use the info you provide.

One last tip: if you plan on adding several LSTM layers ('stacking' them), then you shall need to add one more argument to all but the last LSTM, namely, the return_sequences=True.


Everybody understands how to perform $k$-fold cross-validation but there is often quite a lot of confusion about where/how to use it. So thanks for this good question :)
First, cross-validation is a statistical method for evaluation, not for training:

Of course training is performed during cross-validation, but it is performed $k$ times and therefore there are $k$ models produced during the process. Each of these models is meant to be applied only to its corresponding test set, and in general it would be a mistake to use one of these models after the CV process (it would be an even bigger mistake to select the best of these $k$ models). If a final model is needed, it should be trained on the whole training set independently from the CV process.
The purpose of CV is to obtain a reliable evaluation of the performance of a method by minimizing the chance factor caused by splitting the data between training and test set. The variations across folds can also be used to assess the stability of the method: if there are large variations in the performance, the method is not stable and the resulting model is likely to overfit since it depends too much on the training set.

When to use a separate test set?

If the purpose is only to evaluate a single method reliably, then there is no need to train a final model or to evaluate it on a separate test set.
However quite often one uses CV to evaluate not one but multiple methods: with different algorithms, different hyper-parameters values, different features, etc. These cases fall in the category of parameter tuning, which is a kind of training: the goal is indeed to find the optimal values for some parameters, even if the parameters appear to be part of the design (e.g. some preprocessing option). Like any training stage, there is a risk of overfitting: using CV makes the risk quite low, but even with CV the risk exists (especially with a high number of methods). This means that the selection of the best method based on CV performance might be due to chance, this is why the proper methodology in this setting is to (1) select the best method according to CV performance; (2) train the final model using only this method on the full training set; (3) evaluate on a separate test set. This ensures that the performance obtained on the test set is reliable.


Solution 1 and 3 look really similar but I would go for 1 when thinking about the programming paradigm.

Seeing the classification as a big pipeline using standardization, balancing, and classification, one would not like to re-standardize something in the balancing. Furthermore, the assumption made in 1 is correct.

In a practical way, you could use the pipeline offer by the imbalanced API to use SMOTE with a scikit-learn normalizer


Does shuffling data for time series forecasting help?

Shuffling data would not seem to make sense here, since your model has "memory". You're not predicting $y_i$ from only $x_i$, but also $x_{i-1}$ and $x_{i-2}$. If you shuffle the data and perform prediction, you are implying that $x_1, x_2, x_3$ should give the same value as $x_2, x_1, x_3$ or $x_9, x_5, x_3$, or any series of values that merely ends in $x_3$ (since the target value is always $y_3$, regardless of the other $x$). If your target value actually does depend on preceding variables, shuffling the data breaks that relationship. If it does not depend on preceding values, it's arguably not a time-series model, since the ordering of observations is irrelevant.


Tutorial for restricted Boltzmann machine using PyTorch or Tensorflow?

Try to look for implementations of deep belief networks.

https://github.com/albertbup/deep-belief-network
https://github.com/JosephGatto/Deep-Belief-Networks-Tensorflow
https://medium.com/analytics-army/deep-belief-networks-an-introduction-1d52bb867a25
https://skymind.ai/wiki/restricted-boltzmann-machine
https://www.csrc.ac.cn/upload/file/20170703/1499052743888438.pdf


How to convert DNA sequences in FASTA format to OneHot Encoded Pandas Dataframe for Neural Networks?

I am not sure about any python package but you can simply do integer encoding using labelencoder() and then do one-hot encoding.
label_encoder = LabelEncoder()
integer_encoded_seq = label_encoder.fit_transform(seq)
onehot_encoder = OneHotEncoder(sparse=False)
integer_encoded_seq = integer_encoded_seq.reshape(len(integer_encoded_seq), 1)
onehot_encoded_seq = onehot_encoder.fit_transform(integer_encoded_seq)

Refer this.


Is knn similar to this version of k-means?

Your case is where K=number of points in dataset :
K-means: Lets suppose, there are 10 data points and k=10, so you have 10 clusters. the new test point will be matched with the cluster nearest to it
KNN: If K=1, then the new test point classified will be same as in K-means.
So, if there is one data-point per cluster, then your given answer ie. knn is technically correct, if we take k=1.


What is meant by outliers in text data set. How to detect them?

I define an outlier in the following ways.

It can be a wrong data entry (Eg. human typing error)
It can be a data that has values that are not relevant (Eg. an entry of total which is calculated as the sum of the above columns. This data can be misleading at times so it should be removed)
It can be a data entry that is all or most fields blank (Eg. a row in the data where all fields are blank. This row maybe not contributing anything to the analysis)
It may be extreme values which fall way out of the range of the other data (Eg. when we are calculating the age of Humans, anyone with age (say) above 120 years is an extreme case and that can be ignored depending upon our analysis goal)


How can I measure if a population has the same distribution as other?

What I'm gonna say might seem too simple, but I assume, it might not be bad to fit a multivariate distribution (like Gaussian) to each distribution and then figure out what the mean and covariance matrix are. Mean might not depict so much information but the variance and correlation that are exposed in the covariance matrix might be helpful.

For the case of determining if a variable is useful or not, analyzing its correlation with other parameters might turn out to be useful.

(These are my own thoughts)


The function problem which Goodhart's Law described is the changes of the underlying model from which the data is generated.

One way to look at this is that the data generating process in the past is bias. Say we our try model is a product of two variables $z = xy$. In the past, $y$ has always been constant at $a$, so you come to the conclusion $z$ must equal to $ax$. But this is only true when $y=a$.

In Goodhart's setting (economic policy and social science in general), it is usually the prediction value of $z$ that causing the changes in $y$. It is a dynamic system of which we only have a bias static slice of data.


On the page you linked there is actually a Python example on how to get the data. It is in Python 2, but I will show you how to make it work in Python 3.

import urllib
import json                 # Used to load data into JSON format
from pprint import pprint   # pretty-print

url = "https://data.sa.gov.au/data/api/3/action/datastore_search?resource_id=fec742c1-c846-4343-a9f1-91c729acd097&limit=5&q=title:jones"
response = urllib.request.urlopen(url)
print(response)
# Just an object:   <http.client.HTTPResponse at 0x7f2618123e10>

We get the text data out by using the read() method:

text = response.read()

The response in this case is a raw string. We can use the json module's function loads to load a string):

json_data = json.loads(text)

pprint(json_data)

returns the following JSON data:

{'help': 'https://data.sa.gov.au/data/api/3/action/help_show?name=datastore_search',
 'result': {'_links': {'next': '/api/3/action/datastore_search?q=title%3Ajones&offset=5&limit=5&resource_id=fec742c1-c846-4343-a9f1-91c729acd097',
                       'start': '/api/3/action/datastore_search?q=title%3Ajones&limit=5&resource_id=fec742c1-c846-4343-a9f1-91c729acd097'},
            'fields': [{'id': '_id', 'type': 'int4'},
                       {'id': 'LGA Name', 'type': 'text'},
                       {'id': 'Tenure type', 'type': 'text'},
                       {'id': 'Very low income <$603 per wk', 'type': 'numeric'},
                       {'id': 'Low income $603-$964 per wk', 'type': 'numeric'},
                       {'id': 'Moderate income $965-$1446 per wk',                            'type': 'numeric'},
                       {'id': 'Total', 'type': 'numeric'},
                       {'id': '_full_count', 'type': 'int8'},
                       {'id': 'rank', 'type': 'float4'}],
            'limit': 5,
            'q': 'title:jones',
            'records': [],
            'resource_id': 'fec742c1-c846-4343-a9f1-91c729acd097'},
 'success': True}

I would suggest using Pandas, which can do a lot of the tedious work for you very easily. It can read straight from a JSON string (our text above). The issue is that is will parse it a little strangely.

There is no simple way to write this directly to a CSV file, because there are nested structures: e.g. under "result" there "fields" and then more values, and CSV files can't display that directly. You need to essentially flatten the structure yourself, perhaps decide what is important or what you want to can leave out.

You could take the JSON formatted json_data above and unpack it manually, removing nested parts, which means looking through the response and making your own Python dictionary with only single level i.e. no nesting. Let's say you do that and have a new flattened dictionary named r. Once you have done that, you can do the following using Pandas to write the CSV file:

import pandas as pd

df = pd.read_json(r)
df.to_csv("output.csv")


Doing some further searching, I found a nice post by Jason Brownlee that has helped me better understand the problem and one potential solution:

a multi-input model with an LSTM input for the historical data and a
vector input for expected conditions.

Furthermore, Keras' Functional API (https://keras.io/guides/functional_api/) will help me build such a model.


In order to "tap" intermediate layers of an existing model you could do the following:

# get your feature layer
cnn_model = load_model('weights/pre-trained_CNN.hdf5')
feature_layer = cnn_model.get_layer('fc7')

# stack your LSTM and other layers below, e.g.:
lstm_layer = TimeDistributed(LSTM(...), input_shape = ...)(feature_layer)
output = Dense(...)(lstm_layer)

# create a combined model
model = Model(inputs = cnn_model.input, outputs = [output])


How can I discover topics in a social media data-set?

You can take a look at Latent Dirichlet Allocation. In my experience this does very well without too much effort. You need to remove words that don't help like stopwords (and in your case Twitter handles and probably URLs) before feeding it to the algorithm. The only really important parameter that you need to give it is the number of topics. This will depend on your population (are these random tweets, or only tweets from a specific subgroup/hashtag?) and you need to compare some settings. What you can do is print the most important words per topic and see if they indeed do belong together.

If there are different languages in your tweets you need to deal with that beforehand, maybe classify them on language and only keep the English ones for example.


I know this question was asked sometime ago. I took a look at the diagram and the example code. First, the example code is way off. Here's a description to help those curious how to understand the diagram.

The diagram (architecture) shows only the frontend of a CNN. That's the convolutional layers. On the leftside are a series of VGG convolution blocks. The links to the right side are identity links between the VGG convolution blocks, which pass through their own convolution before being matrix added into the output of the convolution block. The addition of the convolution on the identity link makes it a variant of a ResNet architecture.

The UV 224x224x2 would be the output from the convolutional frontend. It would then be flattened and passed into the DNN layers.


I think your manual is for an older version of Weka.  I have version 3.9.3. In some version the Weka developers have perhaps changed some internal structures and class names.

In a terminal window with a command line prompt, go to the directory where you have Weka installed, that is where the weka.jar file is located. There should be a data/ subdirectory.

cd c:\.....

There, run this command.

java -cp weka.jar  weka.classifiers.trees.J48 -t data/weather.nominal.arff
(Your data file name could be different)

Then I get this output:

=== Classifier model (full training set) ===

J48 pruned tree
------------------

outlook = sunny
|   humidity = high: no (3.0)
|   humidity = normal: yes (2.0)
outlook = overcast: yes (4.0)
outlook = rainy
|   windy = TRUE: no (2.0)
|   windy = FALSE: yes (3.0)

Number of Leaves  :     5

Size of the tree :  8

Time taken to build model: 0.33 seconds

Time taken to test model on training data: 0 seconds

=== Error on training data ===

Correctly Classified Instances          14              100      %
Incorrectly Classified Instances         0                0      %
Kappa statistic                          1
Mean absolute error                      0
Root mean squared error                  0
Relative absolute error                  0      %
Root relative squared error              0      %
Total Number of Instances               14

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 1.000    0.000    1.000      1.000    1.000      1.000    1.000     1.000     yes
                 1.000    0.000    1.000      1.000    1.000      1.000    1.000     1.000     no
Weighted Avg.    1.000    0.000    1.000      1.000    1.000      1.000    1.000     1.000

=== Confusion Matrix ===

 a b   <-- classified as
 9 0 | a = yes
 0 5 | b = no

Time taken to perform cross-validation: 0.02 seconds

=== Stratified cross-validation ===

Correctly Classified Instances           7               50      %
Incorrectly Classified Instances         7               50      %
Kappa statistic                         -0.0426
Mean absolute error                      0.4167
Root mean squared error                  0.5984
Relative absolute error                 87.5    %
Root relative squared error            121.2987 %
Total Number of Instances               14

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0.556    0.600    0.625      0.556    0.588      -0.043   0.633     0.758     yes
                 0.400    0.444    0.333      0.400    0.364      -0.043   0.633     0.457     no
Weighted Avg.    0.500    0.544    0.521      0.500    0.508      -0.043   0.633     0.650

=== Confusion Matrix ===

 a b   <-- classified as
 5 4 | a = yes
 3 2 | b = no


Modern methods for reducing dimensions and feature engineering

I'm not really knowledgeable about the modern techniques but I can tell you about the old ones: ;)
First there are two main approaches to dimensionality reduction: feature selection and feature extraction. You're using the former, which consists in discarding some of the original features. The latter consists in some kind of "merging" of similar variables, it can be worth trying especially if you have redundant features.
As you rightly noticed, feature selection based on individual features is rarely optimal. There are methods which can take the full set of features as a basis for selection, in particular genetic feature selection.


Debugging a simple 1-D CNN for solving a simple classification problem

you can add dropout to the model It will change your model accuracy.
from keras.layers import Dropout

model = models.Sequential()
model.add(layers.Conv1D(filters=1, kernel_size=2, activation='relu', input_shape=(1024, 1)))
model.add(layers.MaxPooling1D(pool_size=64))
model.add(layers.Flatten())
model.add(Dropout(0.5))
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
      loss='binary_crossentropy',
      metrics=['accuracy'])


In survival analysis, which is the correct way to introduce a variable which changes the survival rate but occurs at different times?

The problem you'll run into if you are not careful is the "immortal time bias". In short, the problem is that a subject isn't "in" the  "1-2 years" group until they atleast 1 year under observation. This 1 year period is called immortal because patients can't die then. More concretely, if I naively partition my population into "First 0-6 months" vs "1-2 years" and measure survival, the latter group is going to look like they have much better survival in the first year because in order to qualify for the latter group, you need to live longer.

So what do you do? You need to model the time-varying nature of your data. Check out the "long" format for survival data. Below I have a Python example that uses lifelines. There are four individuals, each will be treated in a different treatment period. We use multiple lines (but same id) to denote different time periods (note the mutually exclusive start/stop). A dummy variable of the treatment period is provided.

import pandas as pd
from lifelines import CoxTimeVaryingFitter

df = pd.DataFrame([
    {'id': 1, 'start': 0, 'stop': 12, 'E': 1, 't1': 0, 't2': 0, 't3': 0},  # never received treatment, died at t=12

    {'id': 2, 'start': 0, 'stop': 10, 'E': 1, 't1': 1, 't2': 0, 't3': 0},  # received treatment at very start

    {'id': 3, 'start': 0, 'stop': 3,  'E': 0, 't1': 0, 't2': 0, 't3': 0},   # will received treatment in second "period"
    {'id': 3, 'start': 3, 'stop': 9,  'E': 1, 't1': 0, 't2': 1, 't3': 0},  # received treatment in second "period"

    {'id': 4, 'start': 0, 'stop': 6,  'E': 0, 't1': 0, 't2': 0, 't3': 0},   # will received treatment in third "period"
    {'id': 4, 'start': 6, 'stop': 11, 'E': 1, 't1': 0, 't2': 0, 't3': 1},  # received treatment in third "period"

])

ctv = CoxTimeVaryingFitter().fit(df, id_col='id', start_col='start', stop_col='stop', event_col='E')

Another solution instead of using dummy variables is to create an interaction with the time variable above, but that will be harder to interpret I believe.


SRCNN - how to get a color image output training only with luminance

The authors use the Y-channel as luminance channel since their images are in the YCbCr color space. The other two channels are upscaled by bicubic interpolation. Later in the paper, they write:

  Specifically, we first transform the color images into the YCbCr space. The SR algorithms are only applied on the Y channel, while the Cb, Cr channels are upscaled by bicubic interpolation.

In their paper, the authors also compare this method to other learning strategies, e.g. training on all three channels or first training the Y-channel and then fine-tune the network with all three channels. The results are on page 11. The section ends with:

  It is also worth noting that the improvement compared with the single-channel network is not that significant (i.e., 0.07 dB). This indicates that the Cb, Cr channels barely help in improving the performance.


As far as I understand, you have factors ("bad, "good" etc) and continuous "invitations". If you want to compare two groups you could use a t-test (e.g. Wilcoxon). If you want to compare all of the groups, you could use a simple linear regression of form:

$$ invitations = \beta_0 satisfaction_1 + \beta_1 satisfaction_2 + ... + u.$$

R example:

library("e1071")
iris = iris

table(iris$Species)
#iris = iris[!(iris$Species=="versicolor"),]

library(dplyr)

iris %>%
  group_by(Species) %>%
  summarise_at(vars(Sepal.Length), funs(mean(., na.rm=TRUE)))

Result (means):

# A tibble: 3 x 2
  Species    Sepal.Length
  <fct>             <dbl>
1 setosa             5.01
2 versicolor         5.94
3 virginica          6.59

Compare two groups:

# Two-samples Wilcoxon test
wilcox.test(iris$Sepal.Length[iris$Species=="setosa"], iris$Sepal.Length[iris$Species=="virginica"])
# The p-value is less than the significance level alpha = 0.05. We can conclude that Sepal Length is significantly different

Result:

    Wilcoxon rank sum test with continuity correction

data:  iris$Sepal.Length[iris$Species == "setosa"] and iris$Sepal.Length[iris$Species == "virginica"]
W = 38.5, p-value < 2.2e-16
alternative hypothesis: true location shift is not equal to 0

Regression:

# Simple linear regression
summary(lm(Sepal.Length~Species, data=iris))
# p-values are smaller than 0.05 which means each factor's contribution is statistically different from the intercept

Result:

Call:
lm(formula = Sepal.Length ~ Species, data = iris)

Residuals:
    Min      1Q  Median      3Q     Max
-1.6880 -0.3285 -0.0060  0.3120  1.3120

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)
(Intercept)         5.0060     0.0728  68.762  < 2e-16 ***
Speciesversicolor   0.9300     0.1030   9.033 8.77e-16 ***
Speciesvirginica    1.5820     0.1030  15.366  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.5148 on 147 degrees of freedom
Multiple R-squared:  0.6187,    Adjusted R-squared:  0.6135
F-statistic: 119.3 on 2 and 147 DF,  p-value: < 2.2e-16

The interesting bit here is Pr(>|t|). If the number in this column is smaller 0.05, you can say that the factor is significantly different from the intercept (which is the base category, in this case "setosa").

In this application, the column Estimate directly gives you the mean of "setosa" for the intercept. The effect for "versicolor" is 0.9300, where 5.0060+0.9300=5.936, which is the mean for "versicolor" and so on.


Predefined Neural Networks instead of fine tuning?

It depends. If you are doing a task for which the weights of a very large network, trained on the same type of problem, are present, then it's better to use the weights of that pre-trained network. You can also fine tune the layers later on. One such example is the VGG16 Net. This NN was used in the ImageNet challenge in 2014, hence if you are trying to do image classification tasks where your images are the subset of ImageNet, then you should use the pre-trained VGG16 weights. This is a great tutorial if you want to go in details.

But the above situation might not be true in every case. For example, the above case is not true if you reverse the situation.


This is a problem of identifying a good regular expression to parse the strings into their component parts. I made an assumption that this data is sitting in a DataFrame and you are trying to add a new column with the extracted string.

Code

import pandas as pd
import re

def extract_second_group(full_string):
    pattern = '^\S*:50.*:\W*(\w*):\S*$'

    match = re.search(pattern, full_string)
    if match is not None:
        return match.groups(0)[0]
    else:
        return None

df = pd.DataFrame(columns=['col1'])

df['col1'] = [
            '{kjkjkjdd:50:/kjhjhjhsa:ddd',
              'kjkjytdvdjjh:50K:kkcknkknn:nbnbds',
              'mvmmbmnnm{:50B:iut6565vdjbj:kjkjkj',
              'not a matching string'
]

df['col2'] = df['col1'].apply(lambda col1:extract_second_group(col1))
df

Explanation

The regex pattern is ^\S*:50.*:\W*(\w*):\S*$. Let's break it down to see what it is doing.

^   : Start of string
\S  : Matches any character which is not a whitespace character
:50 : Match this exact set of characters (in that sequence)
.  : matches any character except a newline
\W : Matches any character which is not a word character (matches the (/) character, if any)
(\w*): This is the most important part of the expression. The \w* matches Unicode word characters, while the parentheses tell us to create a group. We are interested in capturing this group
:   : Another colon separates the group from the rest
\S*  : Another non-whitespace sequence of characters
$   : End of string

The regex is enclosed in a function so that we can check when a regex search doesn't return a match. Call the apply() function to run each row through the function and return the matched group.

Edit Jul 24 2019

This edit is to reflect the change in the data from OP.
The problem remains a question about regular expressions. Here's the code for the updated dataset

Updated Code

import pandas as pd
import re

# https://docs.python.org/3.6/library/re.html
def extract_second_group(full_string):
    pattern = '\W*:50\w{0,1}:(/*[\w]*):\S*$'

    match = re.search(pattern, full_string)
    if match is not None:
        return match.groups(0)[0]
    else:
        return None

df = pd.DataFrame(columns=['col1'])

df['col1'] = [
            '{hjjlhllllh:59:kjkjkjdd:50:/kjhjhjhsa:ddd:70:jnjjh',
              'kjkjytdvdjjh:fhghghghg:52:jkjkt:50K:kkcknkknn:72:nbnbds',
              'mvmmbmnnm{hghghgsl:65A:jjhjhjh:50B:iut6565vdjbj:kjkjkj',
              'not a matching string'
]

df['col2'] = df['col1'].apply(lambda col1:extract_second_group(col1))
df

Output

$$\begin{array}{c|c|}
  & col1 & col2\\
\hline
0& \{hjjlhllllh:59:kjkjkjdd:50:/kjhjhjhsa:ddd:70:j... & /kjhjhjhsa\\
\hline
1& kjkjytdvdjjh:fhghghghg:52:jkjkt:50K:kkcknkknn:... & kkcknkknn\\
\hline
2& mvmmbmnnm\{hghghgsl:65A:jjhjhjh:50B:iut6565vdjb... & iut6565vdjbj\\
\hline
3& not a matching string & None\\
\end{array}$$

Updated Regex
The updated pattern is \W*:50\w{0,1}:(/*[\w]*):\S*$.
I updated it as follows

\W* Start with some non-word characters
:50\w{0,1} Followed by :50 and 0 or 1 Unicode word characters
:  Followed by a colon
(/[\w]) Followed by the group we want. I included one or more / and any number of Unicode word characters.
:  Followed by a colon
\S*$ Followed by any number of non-space characters to end of string

If there are other patterns you need to match, you can use the resources from https://docs.python.org/3.6/library/re.html to locate the different options available in regex patterns.


Yes, it is possible to train an RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn word embeddings.

The word embeddings of the corpus words can be learned while training a neural network on some task e.g. sentiment classification. Before it can be presented to the RNN, each word is first encoded so that it is represented by a unique integer e.g. using a tokenizer. We add a padding token to make all the sentences of the same length.

Doing this is referred to as building an embedding layer "in front of" your LSTM/RNN/GRU network model. For the embedding layer you need to specify:

maximum length of a sequence
embedding size for each token.

The drawback is that using this technique, training will take more time because you have to not only train your prediction model but also your word embeddings. This is because the total amount of trainable parameters will be larger as compared to a model where we use a pre-trained word embedding with frozen parameters.

Finally, depending on your dataset size using pre-trained embeddings trained on some larger corpus will lead to better embeddings than training your own word embeddings on your smaller dataset.

Here is an experiment using both approaches:
https://towardsdatascience.com/machine-learning-word-embedding-sentiment-classification-using-keras-b83c28087456


Universal function approximation with fixed values (as vector or matrix)

By "plain fixed numbers" do you mean a piecewise-constant function?  Sure, those are universal approximators (with some natural hypotheses*).  Indeed, decision trees generate a special kind of piecewise-constant function, and they are universal approximators*.

*I've actually been unable to find references for these, and am not sure how to work out reasonable hypotheses.  Using the examples $\sin(x)$ and $\sin(1/x)$ for $x>0$ indicates that we need a compact domain, and surely some sort of continuity is required.  That's probably enough, but my analysis classes are too long forgotten to know how to produce a proof; you might need some fancier stuff like Stone-Weierstrass.  If you take Lipschitz continuity as a hypothesis, it seems like there'd be an easy(/ier) direct argument.


Which is more important - stable training results or good test results?

I would personally prefer the scenario where the training/testing accuracy is high/low respectively, because it would mean that your model was at least well-fit in the training data but unfortunately performed poorly on unseen data. It is also a conservative scenario, because it can't get worse than that, so you have a good feeling that your model needs improvement.

On the other hand, if your training accuracy is low, you shouldn't even test the model on unseen data, because you are certain that it is not trained well enough. However, if you insist on testing it, even though the training accuracy is low, and the testing results are better than the training results, then this probably means that you were lucky during testing. Under no circumstances should you make any inference of the model's quality in such a scenario.

The solution to your problem would be to use k-fold validation, meaning that you should randomly split your dataset multiple times into different training/test sets and evaluate its training/testing accuracy over all the splits to have a better feeling of how the model behaves on your dataset.


LSTM is a neural network architecture used for sequence prediction, whereas NER is name of NLP task.

The two are incomparable, as one is a method and the other type of problem.

The relationship between them is that LSTMs can can be used for sequence tagging, which can be used for NER.


Dealing with training set of questionable quality

Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online.  Those are super polished and refined.  But, the real world data is not so.

The step where one does the cleaning and scrubbing is called the data pre-processing step.

So, some nice data cleaning techniques, in addition to @jknappen's excellent answer are:

Elimination of zero variance columns/predictors: These columns are not important, and they cause the model and the fit to crash and leak errors. So, eliminating them would make complete sense.
Correlated Predictors: Reducing the level of correlation between the predictors would be a very nice step in the pre-processing process.
Scaling: You must be knowing why scaling is important during pre-process.
Predictor Transformations

A nice reference from Kaggle forums where the pre-processing and cleaning of data sets is discussed.


Training accuracy is ~97% but validation accuracy is stuck at ~40%. What does it imply?

A little more context about the data set and type of model would help, but most likely your model is overfitting to the training data. This means that your model is picking up noise from the training data and has basically "memorized" the data it has seen.

Therefore, the model is not generalizable, and this results in relatively poor performance on a data set it hasn't seen, explaining the 40% validation accuracy.


Statistical distances for time series of distributions

This is similar to the fundamental information theory problem that Shannon explored.  In that domain, it is framed this way: given two rvs, X and Y, what information does X convey about Y?

An example would be that I create sequence of numbers/bit/letters from a known PDF (X) and you receive a distorted version of those values for which you also know the PDF (Y).  The mutual information is the number of bits that Y communicates about X which can be thought of as a type of correlation.

In the case of time varying PDFs (ie stochastic processes), information theory would just treat the ensemble of rvs as one joint rv and then calculate the mutual information of the joint PDFs.  If the PDFs are iid, then significant simplifications are possible.  Joint Gaussian PDFs also make things easier.

Another information theory concept that might be useful is the entropy rate of a process.  ER is a quantification of the amount of information contained in a process.  Depending on your problem, you might be able to compute the ER for each process and then use those values as realizations of a feature that can be grouped with a clustering algorithm.


Over fitting in Transfer Learning with small dataset

First of all:

I think you should reduce the number of FC layers and number of nodes of FC layers, for example, one FC with 256 or 512, or 2 FC with 256 and 512. Try this.

Try to make your batch size 30, and decrease number of epochs to nearly 10 or 20. 100 epochs are too many for your small size dataset.

Secondly, there is more than one way to reduce overfitting:
1- Enlarge your data set by using augmentation techniques such as flip, scale, etc.
2- Using regularization techniques like dropout (you already did it), but you can play with dropout rate. Try more or less than 0.5.
3- One of the good techniques in your case is to do early stopping. In any epoch when you see that the model goes to overfit, stop it.
4- Using cross-validation to train/test your model.
and many more.

For more, see here, or here, or here.

Feel free to ask any further questions.


Correlations - Get values in the way we want

Several kernel functions can serve as similarity functions (=scores). See a list, for example, here. You can try several of them and see which suits you the best.

You need something that drops fast at low distances. You can try
$$ score = 1/(1+distance)^2$$
and adjust coefficient in front of distance so that the score fits between 0 and 1

About your picture: what are axis labels? and what are x-ticks?


How much can bias decrease performance of the network at the beginnng of the training?

Bias takes care of variables which are latent for i.e you did not include it in the training set.

So you are trying to overfitt on training set, but now you introduce pertrubance. Ofcourse it will have worse performance.


How do you define the steps to explore the data?

That is a very good framework of solving a question you have. According to me, it has multiple answers. I'll give you the one which I relate to.

After cleaning the data or rather while cleaning it, we have to be clear about the task ahead and about our results.
The work on the data follows the below steps mostly:

Feature Detection
Training using the above features(there are many Machine Learning/Deep learning models to do this) like Classification (this depends on the task)
Check the trained model on validation test if needed and later on test set
The features depend on the dataset. Standard deviation or to find central tendency isn’t always a criteria. Machine Learning is needed to train on the dataset in most of the cases.


Machine learning algorithm to classify matrices

You can indeed use a simple neural network for this task, flatten your matrix and feed the matrix entries as inputs to your network. However, I would suggest using a convolutional neural network, these have had very good performance on image data, which is essentially a matrix. This class of NN is powerful because it uses a neighborhood around each point as an input and thus significantly reduces the number of inputs to the network, which in turn reduces the model complexity.

I would also suggest using some matrix transformations to get better features. For example you can perform eigenvector decomposition of the matrix to ensure that the features you are feeding to your network are orthogonal. This may increase performance of your classifier.

If you give us some examples of your dataset we can help you build a network to start testing ideas.


When to use deep learning for java as opposed to python

When it comes to picking a language for a certain application, many factors may play a role:

Who will be building/maintaining the application and are they familiar with that language?
What other systems will the application need to communicate with and is one language easier to do that with than the other?
Is this project experimental or is it supposed to be a cornerstone for the next few years?
Are there existing implementations/frameworks of what needs to be done in that language? Does it fit the need or does it need to be altered?
Etc.

If we look at your list, here is my take:

ETL

ETL is quite a language agnostic, both Java and Python have good bindings to tools necessary for ETL.

Data pre-processing

Similar to ETL, although Python may be easier to do simple things.

Making use of pre-trained models / transfer learning

Python is much more convenient as most existing frameworks and pre-trained models are compatible with Python, and likely not with Java.

Distributed computing

The Hadoop stack is very Java friendly. Good to note that Python is increasingly become easier to use with Hadoop products (but Java is still running underneath)

Processing large volume of data (images,time series data,sensor data..etc)

Although Python has a reputation of being slow, if you know what you're doing, most Python libraries you'll use will ultimately run C/C++ code anyway, so speed shouldn't be a huge concern.

Production implementation (batch or real time / online)

Java is statically typed which offers a lot of advantages for production, it's becoming steadily easier to do this with Python as well (i.e. pydantic, mypy, etc.)

Post production training (bacth or real time / online)

This is independent of the language, this has more to do with your MLOps and your pipelines in place.

Mobile app development

Java definitely has the edge, although in a world of microservices, Python can prove very helpful.

IoT edge device computing

When it comes to device computing, Java probably has the edge, although, C/C++ are likely your best allies.


Sports prediction using machine learning

You have two categorical features (team names) and two continuous features (goals scored). The continuous features will probably remain bounded at a lower value since soccer isn't typically a high scoring game, so no normalization will be necessary.

You may need dummy variables for the team names, as the text names will likely not play well as a feature for most classification libraries. Research the sklearn Ordinal Encoder for this task.

Look at the sklearn classification algorithms that are available and try implementing a few of those.

Your response variables will be the two predicted scores. One for home and one for away. Or you may decide that the response variable could simply be one 'spread
' feature for the expected spread of the score.


You can simply make your training data a shape that is divisible by 50.
    In [1]: sample_size = 63648

In [2]: to_remove = sample_size % 50

In [3]: end_index = sample_size - to_remove

In [4]: model.fit(trainX[:end_index], trainY[:end_index], ...

What are the dimensions of trainX and trainY? You are not specifying a validation dataset, is that intentional?

You say you want to reset the state after a mini-batch, but you are resetting the state after each epoch (one loop through your entire dataset).

One more comment: the n_epochs = 10000 seems excessive.


How do I predict a set of frequently bought items?

The simplest way is to build a matrix representing how often two items are bough together: $M[i,j]$ is the count of how many times items $i$ and $j$ are bought together (note that you need to fill only a diagonal matrix, since it's symmetric).

From there you can calculate whatever you need: most frequent $B$ item bought with a given item $A$, top most frequent pairs of items in general, etc.

You could also consider the matrix as a graph representation, and use algorithms specific to graphs.


Keras categorical_crossentropy loss (and accuracy)

I am using keras with tensorflow backend. I checked and the categorical_crossentropy loss in keras is defined as you have defined. This is the part of code (not the whole function definition)-

def categorical_crossentropy(target, output, from_logits=False, axis=-1):
    if not from_logits:
        # scale preds so that the class probas of each sample sum to 1
        output /= tf.reduce_sum(output, axis, True)
        # manual computation of crossentropy
        _epsilon = _to_tensor(epsilon(), output.dtype.base_dtype)
        output = tf.clip_by_value(output, _epsilon, 1. - _epsilon)
    return - tf.reduce_sum(target * tf.log(output), axis)

As you can see in last line, it is returning the sum of product of true values and log of output values for each observation. You can find complete function definition here in line 3176.

For theano backend, it should be same. You can check here in line 1622.


At the beginning of the algorithm, you are taking an initial guess for the relevant parameters. Then in each iteration, in the E-step you are computing the responsibilities for all the given datapoints and in the M-step you are computing the weighted means and variances.

Since you are clustering the datapoints there is not training or test set. You just want to divide your dataset to two (or more) clusters.


You need to convert the elements of corpus to character:

myTokenizer <- function(x, n, n_min) {
  tok <- unlist(tokenize_ngrams(as.character(x), n = n, n_min = n_min))
  M <- matrix(nrow=length(tok), ncol=3,
              dimnames=list(NULL, c( "gram" , "num.words", "words")))
  M[, 3] <- tok
  M[, 2] <- lengths(strsplit(M[, 3], "\\W+"))  # counts the words
  M[, 1] <- 1:length(tok)
  return(M)
}
res <- tm_map(corp, function (x) myTokenizer(x, n=3, n_min=1))

or apply your function directly without using Corpus:

res <- myTokenizer(x = list("A Sample Text ...", "Lorup ipsum x 12"), n=3, n_min=1))


Ways to convert textual data to numerical data

A renown method is using two layers Neural Network which transform each word to a vector. you can use word2vec library also to do this. To know more about this you can find this online article useful.


One simple example of an application of citation analysis is ranking. Imagine you do a keyword search over a collection of scientific articles, How do you rank the result set from most relevant to least? A measure one could use is the amount of citations an article has or the quality of the citations it has and also the quality of the articles the paper refers to. This problem is closely related to the problem of ranking web pages according to its hyperlinks.


Train classifier on balanced dataset and apply on imbalanced dataset?

Is this a problem?

No. not at all.

  Will the model reproduce the trained target distribution from DS1 when
  applied on DS2?

No, not necessarily. If

Balanced set DS1 is a good representative of imbalanced (target) set DS2, and
Classes are well-separated (pointed out by @BenReiniger), which holds easier in higher dimensions,

then model will generate labels with a ratio close to imbalanced DS2 not balanced DS1. Here is a visual example (drawn by myself):

As you see, in the good training set, predictions resemble the real-world ratio, even though model is trained on a balanced set, of course, if classifier does a good job of finding the decision boundaries.


Is there any clustering algorithm to find longest continuous subsequences?

What might help is a custom distance computation as input to the clustering algorithm. These algorithms usually take Euclidean distance as a measure of dissimilarity.

You can try DBSCAN (in Python scikit-learn), with metric='precomputed' and 'X' as a custom distance matrix. You can construct this distance matrix to conform to your requirement. Eg: specify that nodes 3 and 4 have a large distance, even though they are equal.


First Approach:

You can probably train a CNN ( Convolutional Neural Network ) on the images you have. If the number of images is less, I prefer you to use Transfer Learning.

  In Transfer learning, we have a highly trained model ( like Inception, ResNet ) and we modify it so that it can classify images which we need.

Second Approach:

Use object detection. You need to localize an object ( card ) from the image and then classify that cropped image. This could be tedious if you're a beginner.

You can refer to these links:

https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html
https://towardsdatascience.com/creating-your-own-object-detector-ad69dda69c85
https://pythonprogramming.net/training-custom-objects-tensorflow-object-detection-api-tutorial/

  You need to train a localization model which can track and crop your images ( of cards ).


Scalable open source machine learning library written in python

Is there a specific reason beside the fact that you would like to contribute? I am asking because there is always pyspark that you can use, the Spark python API that exposes the Spark programming model to Python.

For deep learning specifically, there are a lot of frameworks built on top of Theano -which is a python library for mathematical expressions involving multi-dimensional arrays-, like Lasagne, so they are able to use GPU for intense training. Getting an EC2 instance with GPU on AWS is always an option.


What is the difference between 8 filters twice and one 16 filters in convolutional layers at CNN?

Each layer has a limited amount that it can transform the layer below it. There is one linear component (weighted sum of output of layer beneath it), and one non-linear component (typically ReLU).

It is in theory possible to approximate any function with a large enough single layer in a fully-connected network. However, a stack of similar smaller layers is more expressive using less resources. That means for the same number of parameters you have access to a more flexible function approximator. At some level of complexity for your target function, the cost (in terms of CPU time, data required and effort in training) of making a single layer wider is higher than the cost of stacking more, similar layers.

In addition, for a CNN, you have to worry about receptive field. Any feature map can only express values that the filter can "see" due to width of the kernel. As you add more layers, each kernel applied extends the width and height of the base image that the features in the last layer effectively calculate over. If you also have a fully-connected layer after the convolutional layer, then you can in theory compensate for a poor receptive field with a very large fully-connected layer - but then you are back to the first problem of wide network with more parameters than strictly necessary to learn the function.


Are there any machine learning algorithms that focus on comparing items, rather than classification or regression?

There are ranking algorithms based on machine learning that are aimed to build ranking models. Training data for these models is given in the form of partial ordering between each pair of elements in a sample. A brief description, together with a list of useful references, is given in the corresponding Wikipedia page.


If I understand correctly (not sure), it looks to me like you don't need a model which can predict at any time, you just need:

a model which predicts the ETA at any stage given information about the past stages. The easiest way to do that is probably to just train a different model for each stage, since the number of stages is fixed.
Then between two stages the ETA can be updated in a deterministic way: if the last stage was passed at time $t$ and the predicted ETA was say 10mn, then at time $t'$ the ETA is just 10mn - $(t'-t)$.


Is there a theorem of prominent Russian mathematicians that played an important role in the development of machine learning?

A search on Google Scholar yielded
NeuralComputation 1991 article
That is

Kolmogorov's Theorem Is Relevant
Abstract We show that Kolmogorov's theorem on representations of
continuous functions of n-variables by sums and superpositions of
continuous functions of one variable is relevant in the context of
neural networks. We give a version of this theorem with all of the
one-variable functions approximated arbitrarily well by linear
combinations of compositions of affine functions with some given
sigmoidal function. We derive an upper estimate of the number of
hidden units."

This theorem of Kolmogorov is surely the one referred to in the seminar I attended. But I somewhat erred I first thought in using the plural of mathematician.
But then I see the terminology
"Kolmogorov–Arnold representation theorem (or superposition theorem)" is also employed.
Wikipedia gives the following references:

Andrey Kolmogorov, "On the representation of continuous functions of
several variables by superpositions of continuous functions of a
smaller number of variables", Proceedings of the USSR Academy of
Sciences, 108 (1956), pp. 179–182; English translation: Amer. Math.
Soc. Transl., 17 (1961), pp. 369–373.
Vladimir Arnold, "On functions
of three variables", Proceedings of the USSR Academy of Sciences, 114
(1957), pp. 679–681; English translation: Amer. Math. Soc. Transl., 28
(1963), pp. 51–54.


You can likely use pandas.date_range as follows assuming your columns are called start and end:
import pandas as pd

df.apply(lambda x: pd.date_range(x["start"], x["end"]), axis=1)


Using predictive modelling for temperature data set

I am not able to understand how and what modelling techniques do we
  use?

Every data science workflow has the follwing steps:

Pre-processing (data cleaning and wrangling)
Exploratory analytics
Model selection
Prediction and testing.  (And re-iteration)
(Optional) Reporting the workflow

  Does it depend on the data type?

Yes, the entire workflow is dependent on the type and features of the data.

  Does it depend on size of data?

Size of data makes a difference in the tools and sometimes(very rarely) the algorithms used.

  I have to predict missing values in a given temperature data set and i
  am unaware of anything that i can use

There is a lot of material and algorithms on how to impute missing data, which you can refer to and use them accordingly depending on the type of data and the problem statement.


How to create a training set for sequence labelling

There are several tools for this you could check the Stanford Simple Annotation Tool, also Brat has online demos. For a comprehensive list you could check this question on Quora


What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

Adagrad is completely unrelated to ridge regression.  There is no reason to expect that there should be any similarity in their methods.

Adagrad divides by the L2 norm of the gradient.  Ridge regression modifies the objective function by adding the L2 norm of the parameters.  So even though it looks like the same thing, it's really not; those are two unrelated values.

There's no principle in mathematics that the L2 norm always has to be added or divided.  It's just a number.  It's like seeing one equation where you multiply by 2 and another equation where you add 2 and asking "why does the second one add 2 rather than multiplying by 2?" -- the answer is "why not? there's no reason to expect them to have to use the number 2 in the same way".


You are missing one little step here. At the output node the backpropagation algorithm starts with the loss function $L$. You want to find out how much the weight $w_1$ impacts our loss function thus
$\frac{\partial L}{\partial w_1}$.
We can break this down with the chain rule as
$\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial w_1} = \frac{\partial L}{\partial f} \frac{\partial f}{\partial \sigma} \frac{\partial \sigma}{\partial w_1}$.
Why do we do this?
We do this to simplify the math. Because the $L$ is a function of $f$ and $f$ is a function of $\sigma$ and $\sigma$ is a function of the weights. But if we ignore the loss function for now.
How do we calculate $\frac{\partial f}{\partial w^2_{i, j}}$?
$\frac{\partial f}{\partial w^2_{i,j}} = \frac{\partial f}{\partial h^2_i} \frac{\partial h^2_i}{\partial w^2_1}$
$f = h_1^2 w_{1,1}^3 + h_2^2 w_{2,1}^3$
$\frac{\partial f}{\partial h^2_i} = w_{i, 1}^3$
$\frac{\partial h^2_i}{\partial w^2_1} = \sigma(W^2h^1) (1 - \sigma(W^2h^1))$
Backpropagation
Based on this answer we can see that $h_i^2$ is a function of the inputs from the previous layer and the activation at the neuron. Thus, we can substitute this function into our equation and continue taking the derivatives and we can see how the error propagates backwards through the network.
$\frac{\partial f}{\partial w^1_{i,j}} = \frac{\partial f}{\partial h^2_i} \frac{\partial h^2_i}{\partial h^1_i} \frac{\partial h^1_i}{\partial w^1_{i,j}}$

(Update)
The answer
This may not be correct, the work needs to be checked.
$f = \begin{bmatrix} w_{11}^3 & w_{21}^3 \end{bmatrix} \begin{bmatrix} h_{1}^2 \\ h_{2}^2 \end{bmatrix} = \textbf{W}^3\textbf{h}^2$
$\textbf{h}^2 = \begin{bmatrix} h_{1}^2 \\ h_{2}^2 \end{bmatrix} = \sigma(\begin{bmatrix} w_{11}^2 & w_{21}^2 \\ w_{21}^2 & w_{22}^2\end{bmatrix} \begin{bmatrix} h_{1}^1 \\ h_{2}^1 \end{bmatrix}) = \sigma(\textbf{W}^2\textbf{h}^1)$
$\textbf{h}^1 = \begin{bmatrix} h_{1}^1 \\ h_{2}^1 \end{bmatrix} = \sigma(\begin{bmatrix} w_{11}^1 & w_{21}^1 \\ w_{21}^1 & w_{22}^1\end{bmatrix} \begin{bmatrix} x_{1} \\ x_{2} \end{bmatrix}) = \sigma(\textbf{W}^1\textbf{X})$
Then using the chain rule
$\frac{\partial f}{\partial \textbf{W}^1} = \frac{\partial f}{\partial \textbf{h}^2} \frac{\partial \textbf{h}^2}{\partial \textbf{h}^1} \frac{\partial \textbf{h}^1}{\partial \textbf{W}^1} = \textbf{W}^3 * \sigma(\textbf{W}^2\textbf{h}^1)(1-\sigma(\textbf{W}^2\textbf{h}^1)) * \sigma(\textbf{W}^1\textbf{X})(1-\sigma(\textbf{W}^1\textbf{X}))$


Where can we find the application of bayes's theorem in Bayesian optimiation with gaussian processing

It is a 49 page long paper, so following observations are based only on a cursory reading.

The optimisation is for finding best value of parameters for cost function of machine learning models.
Rather than finding a fixed value of the parameters, it is assumed that the parameters come from statistical distribution and the task is to find the nature/shape of this statistical distribution.
Bayes theorem tells you that if you have prior beliefs and evidence (data), you can go to posterior.
They start with the assumption that prior distribution of the parameters is Gaussian. The task is to then find the posterior.
Since there are multiple parameters and not single variable, Gaussian process comes into picture rather than single Gaussian distributed random variable.
Bayes theorem comes into picture while going from priors to posteriors. The optimisation is solved using sampling technique like Monte Carlo sampling. Reading up about MCMC sampling method will help you see the connection.


Unsupervised feature reduction for anomaly detection with autoencoders

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this on original (assumed to have no outliers) gives you a network which has learnt an abstract representation of the 40 features with 5 features.

When outliers show up these values are different and the network was unable to construct these particular data points back and the error scores for these values should be typically higher.

I used this with Deeplearning4j with SparkDl4jMultiLayer with KLDivergence loss functions on all layers except output layer with MSE loss functions. with Sigmoid activation function.


Trying to understand free-energy equations in a Karl Friston neuroscience article

I'm not qualified to understand almost all of that paper, but, I might be able to give some intuitions from information theory that help you parse the paper.

|| denotes the Kullback-Leibler divergence. It measures an information gain between two distributions. I suppose you could say it indicates the information in the real distribution of data that a model fails to capture.

When you see "negative log" think "entropy".

In the first equation, think of it as "-ln(...) - -ln(...)". This may help think of it as the difference of entropies. Likewise in the second, read it as "D(...) + -ln(...)". This may help think of it as "plus entropy".

If you look at the divergence definition, you'll see it is defined as the log of the ratio of the PDFs. This may help connect it to logs and negative logs. Look at the definition that writes it as cross-entropy minus entropy. Then this is all a question of differences of entropies of things which may be clearer.


Autoencoder feature extraction without validation set?

To the best of my knowledge, if the performance is higher on the training set than on the validation set, you should probably be worrying about over-fitting.

I am assuming that your validation set, for the moment, is just a fraction of the data of the training set.

Notice that if the autoencoder is not performing well on your validation set, it is unlikely that you will reach good performance on new data "since new data will be very similar to current data", as already on current data you don't have good performance.

If you have to retrain the autoencoder every time you want to encode something, it may be not worth using the autoencoder in the first place.


GBM: small change in the trainset causes radical change in predictions

A good way to avoid this problem is to add noise to your training dataset this will made your model more robust and less versatile.

there is a different way of adding noise, often a gaussian noise is added it depends to the kind of data you have.


CV(Curriculum vitae) Recommendation System guidance

Try word embeddings and unsupervised learning, for example:

  medium

main idea is that similiar users will be in similiar clusters


storing a huge dataset in h5py file format

You can use h5py files but due to the fact that they store the dimensions, they take even more disk to be stored and while fetching you have to supply enough memory which is not possible in your case, so you have to read them chunk by chunk. There is a simpler approach. One of pandas's inputs is chunk. try to use it.


Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?

There is a slight problem with the semantics that you have used.
RNN is a 'recurring' neural net. LSTM is a type of RNN. So when you say "number of LSTMs in the encoder would have to be the same as number of words in the text", I suppose what you mean to say is that the number of time-steps or number of NN's in the (single) LSTM layer is the same as the number of words in the sentence. If this is what you actually meant, then you are correct. For e.g. If you have a 20 word tweet and a set of 10000 such tweets. You can use a single LSTM layer with num of timesteps=20 to process this. You do NOT use 20 LSTM layers.
This should answer your second question also. The decoder too does not have 20 LSTMs. It has probably 1 LSTM layer with 20 time-steps.
The decoder output length need not match exactly with the encoder output length. In other words it need not be a word for word translation. The models typically use BOS and EOS markers to start and stop processing.


Behavioural data required to predict churn

You are not being unreasonable but it is hard to know if they are being reasonable or not as well. If they feel the existing amount of information is enough, from which one can derive many other things, and what the definition of churn is, then it is also fine to abide by them. Of course, it is better to have the data you are looking for in terms of tracking user behavior on the website but that's not all.

Definition of Churn:

This definition can vary and you need to ask yourself how you are going to define it.

Do you consider someone to have churned once they stopped buying after X days, in your case 60, irrespective of what they are doing on the website? Does a failed transaction or a pending transaction such as added to cart but not yet ordered count as churned?

If you only consider someone to have churned if there has been no successful transaction within 60 days, then you can look into many other factors.

May be you have considered these things already but for future viewers, I will still add how I approached problems like these in the past.

Given that you have data of the customers' orders, you can now derive the following:

1) Average sales value of the customer

2) Count of the number of orders

3) What category of products they have bought

4) Age of the customer since first order or account creation or both

5) When was the last placed order

The above points already show you have more or less tried to get the Recency-Frequency-Monetary aspects of a customer.

Using this, you can now create extra columns of features, if required and helpful, to study how these customers buying products from particular Category Z behave.

1) What is their average

2) How does it fare with an individual customer's buying behavior

3) How often do these group of customers buy products

4) What are the other related products they have been buying

5) The shortest and longest time period for a customer belonging to a particular category to be back again and place an order on the shop

So, there you go, you have just extracted so much information from the most basic order information. You can of course extract more and more out of these data as well. However, whether this is enough or not is a matter of the problem statement, amount of data, and of course opinion.


Naive Bayes Multinomial, independence assumption misunderstood

Their outcomes X are dependent because they must be summed to n

If you are really want to model multiple experiments probabilistically, please do not forget the normalization constant that make the unnormalized quantity normalized by integrating/summing them to one.

Multinomial Naive Bayes model, as described by Manning et al (2008), estimates the conditional probability of a particular word/term/token given a class as the relative frequency of term t in documents belonging to class c:

Thus this variation takes into account the number of occurrences of term t in training documents from class c, including multiple occurrences Not multiple experiments.

see The Naive Bayes Text Classifier


Coloring clusters so that nearby clusters have different colors

matplotlib already takes care of coloring adjancent clusters with different colors. But, I believe it uses a unique color for each cluster. If that's the case, 400 colors would be too much.
There might be better ways, but worst case, try this:
We want to color with minimum number of colors. Hence, the problem turns to a graph coloring problem in which, we don't want two connected adjacent nodes have the same color. Finding the minimum number of colors is an np-hard problem but we can use approximate algorithms.
One way could be defining your cluster centroids as graph nodes and storing their connections and then using a graph coloring algorithm.
Step 1: Store your clusters as an adjacency matrix and convert this matrix to a graph wherein each centroid is the representative of a cluster and the adjacency matrix contains the connectivity(neighborhood) between each centroid and the others.
Explanation
Code
Step 2: use a coloring algorithm to color the nodes of the graph.
This post contains a coloring algorithm that gets the graph and returns the coloring for that.
Step 3: having the color of each centroid, you can color the whole clusters with minimum number of colors.


Learning character embeddings with GenSim

I believe that you misunderstood the word2vec concept. Basically for words, the feature vector for a word is learnt from the surrounding words.

You shall know a word by the company it keeps- Firth.J.R

In your case characters have been used, so the feature vector for each character depends upon the adjacent characters present. your example might work, if you have the following training sentences.

ABCDEFGHI
abcdefghi
AbcDEFghi
aBcdEfgHI
abcDEFgHi
ABcdEFGHi

With these training sentences, the characters 'A','a','B','b' will preserve capital features and english alphabet order feature. But whereas, when trained with wikipedia sentences, the characters will preserve the probability of being present in a meaningful word. For instance, the closest letter to 'C' would be 'a','o','e' but hardly 'x' or 'd' because there would be words like 'covenant','country','cat' as no common words would be 'cx..'


Training Validation Testing set split for facial expression dataset

NB This advice assumes your goal is to recognise expression in pictures of any person, and not just people from your training data.

  Should the validation set consists of unseen subjects as well?

Yes. This will give you the most accurate measure of performance in the task you want to use the network in, in order to choose the best generalisation and take it forward to testing.

You would only use a simple random split if the end goal of your trained network is to recognise expressions from images of the people in the training set.

  Or can I shuffle the whole training set and use a part of it (10-20%) as validation set?

No. If you take random samples where the same face appears in training and cv, you will get an over-estimate of generalisation. I have seen this effect first hand in the Kaggle State Farm Distracted Driver contest, and you should see it discussed in the forums there. It might be helpful for ideas to improve performance too.

  if I use 10-20% of the training set as my validation set, my model learns with reasonable accuracy (45-50%) using a 3-layer CNN

This is a data leak between train and cross validation - the network has learnt to correctly classify expressions in images of people it has already seen, and that is what you are measuring by taking a split in this way. It is not surprising that the test result does not match the promising values from cross validation.

  I'm just wondering which is correct since using a validation set of unseen subjects, my model starts to overfit after 3-5 epochs and doesn't learn at all

You are over-fitting regardless of how you split train,CV. When you split the CV incorrectly, then in addition to over-fitting, the data leak gives you bad guidance.

It looks like that you have very little diversity in the training data, whilst training an image classifier from scratch requires a lot of data. Consider:

Collecting more labelled data, perhaps some other dataset you can download.
Adding a lot of regularisation to your model (e.g. multiple dropout layers)
Using a pre-trained image classifier network (e.g. VGG-19 or Inception) as a starting point and fine-tuning it for your classification task.
Use full k-fold cross-validation regardless of the computational cost, to mitigate problems of using small training set. This won't help solve your training problems directly, but will give you a better shot at tuning your network hyper-parameters once you solve that issue.


Training vs test data set for supervised learning in real life scenario

One important assumption of data for a lot of Machine Learning algorithms is that the data from the training set and test set are i.i.d (independent and identically distributed) and come from the same probability distribution. More explanation here. That said, you cannot train a model for movies recommendation and use for house rentals. Or not that I'm aware of.

However, it's possible to train a model for house rentals recommendation if you have a lot of data for movies recommendation and a small amount of data for house rentals recommendation, using a technique called Transfer Learning. The idea is simple: If two tasks are similar, we should be able to use what we learn from one task to learn the other task faster and better.

Following is the general process (for neural networks):

Using movies data, train a model for movies recommendation
Fine-tune this recommendation system using house rentals data: Freeze the most part of the movies recommender, retrain a few last layers with house rentals data.

This is just a direction about how it can be done, there's quite a lot of related literature for you to learn more. I would say this is a not very uncommon scenario.


What is the positional encoding in the transformer model?

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be
$$\begin{align*}e_{w}' &= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),cos\left(\frac{pos}{10000^{2/4}}\right)\right]\\
&=e_{w} + \left[sin\left(pos\right), cos\left(pos\right),sin\left(\frac{pos}{100}\right),cos\left(\frac{pos}{100}\right)\right]\\
\end{align*}$$
where the formula for positional encoding is as follows
$$\text{PE}(pos,2i)=sin\left(\frac{pos}{10000^{2i/d_{model}}}\right),$$
$$\text{PE}(pos,2i+1)=cos\left(\frac{pos}{10000^{2i/d_{model}}}\right).$$
with $d_{model}=512$ (thus $i \in [0, 255]$) in the original paper.
This technique is used because there is no notion of word order (1st word, 2nd word, ..) in the proposed architecture. All words of input sequence are fed to the network with no special order or position; in contrast, in RNN architecture, $n$-th word is fed at step $n$, and in ConvNet, it is fed to specific input indices. Therefore, proposed model has no idea how the words are ordered. Consequently, a position-dependent signal is added to each word-embedding to help the model incorporate the order of words. Based on experiments, this addition not only avoids destroying the embedding information but also adds the vital position information.
This blog by Kazemnejad explains that the specific choice of ($sin$, $cos$) pair helps the model in learning patterns that rely on relative positions. As an example, consider a pattern like
if 'are' comes after 'they', then 'playing' is more likely than 'play'

which relies on relative position "$pos(\text{are}) - pos(\text{they})$" being 1, independent of absolute positions $pos(\text{are})$ and $pos(\text{they})$. To learn this pattern, any positional encoding should make it easy for the model to arrive at an encoding for "they are" that (a) is different from "are they" (considers relative position), and (b) is independent of where "they are" occurs in a given sequence (ignores absolute positions), which is what $\text{PE}$ manages to achieve.
This article by Jay Alammar explains the paper with excellent visualizations. The example on positional encoding calculates $\text{PE}(.)$ the same, with the only difference that it puts $sin$ in the first half of embedding dimensions (as opposed to even indices) and $cos$ in the second half (as opposed to odd indices). As pointed out by ShaohuaLi, this difference does not matter since vector operations would be invariant to the permutation of dimensions.


What are the performance measures in the neural networks field?

Typical predictive performance measures used to compare accuracy of ANN models are:

RMSE - (root mean squared error) measures the distance between estimated and actual outcomes. Other metrics that measure the same concept are MSE, MAE or MPE.
R square - (R^2 or coefficient of determination) measures the reduction of variance when using the model.

When comparing two different ANN models for performance, metrics that take into account the complexity of the model may be used, such as AIC or BIC.


With today's standard amount of RAM for a personal desktop or laptop computer (usually 8Gb or more), there should be little problem in handling 4Gb in memory. If your hardware setup is short in memory and your ML algorithms make your SO start paging, the best hint is to use algorithms and settings that do not copy the data around, but do everything in place.

In pandas, many operations provide a boolean parameter precisely called inplace that controls this behaviour, e.g. DataFrame.reset_index. Numpy also has support for in-place operations, e.g. sort.

Take into account that:

inplace=True does not guarantee that the operation is actually performed in place, it is just a hint that may or may not be honored.
for some operations. inplace=True can worsen performance, like dropna.

All that said, in my experience inplace=True actually makes a difference in memory consumption and can be used to avoid paging and therefore enabling computations to take place completely in RAM for datasets like yours.

Finally, be careful with inplace default values. As noted in this warning in the pandas documentation:

  Warning: For backwards compatability, inplace defaults to True if not specified. This will change in a future version of pandas - if your code depends on an inplace assignment you should update to explicitly set inplace=True


Dataset map function error : TypeError: Expected list for 'input' argument to 'EagerPyFunc' Op, not Tensor

Okay, it was a very stupid mistake.
As the error message states, an input list is required.
So replacing this part of the code:
tf_string = tf.py_function(serialize_one_image, img, tf.string)

to this
tf_string = tf.py_function(serialize_one_image, [img], tf.string)

i.e. wrapping the imgobject into a list solved the problem.
Now it works as expected.
Thanks for reading anyways.


Let's take an example : "I went to the shop."

Let's say you want to predict "to" and "the". With bidirectionality, you will predict :

p(to | I, went, the, shop) : No problem here.

p(the | I, went, to, shop) : Here we have a problem, because we already saw the word 'the' while predicting 'to'. It's trivial for the model to predict 'the'.

With MLM, if we take the same example :

p(to | I, went, [MASK], [MASK], shop)
p(the | I, went, [MASK], [MASK], shop)

There is no more problem because the word cannot see itself in other predictions. It's more difficult to predict, but anyway only 15% of the time the word is masked.

Edit

In MLM, BERT will see all the words of the sentence, including the word to predict itself.
This is because BERT will create representations not only for the [MASK] token, but also for the other tokens of the sentence (but in this case we are interested only in [MASK]).

Note : It is also because BERT will create representations of all tokens that we need to replace [MASK] sometimes by the original token, sometimes by a random token.

You can find more detailed information in this blog in the "Pretraining" section.


LSTM model prediction scaling with loaded model

Answering to my own question:
One way to do it was to use joblib dumping the scalers that were used in the training process.
More about it is discussed in here: https://stackoverflow.com/questions/41993565/save-minmaxscaler-model-in-sklearn
Works the same way for LSTM.


Which error metric is good for measuring accuracy

Since you have a regression problem, you can use many possible evaluation metrics. Other regression evaluation metrics include mean squared error (MSE), mean absolute value (MAE), and mean absolute percentage error (MAPE).
However, it sounds like the issue is variance across different target values. If you switch to Bayesian Regression, it will better model variance and uncertainty across the range of target values.


I think I have found the answer, but I'd like to have some validation from the community. Could someone please let me know if this seems like a valid explanation?

The reason for the remaining 320M parameters

in the input of the encoder, computing the word embeddings requires taking a one-hot vector of 160K words and transforming it into a 1K dimensional word embedding. Thus this accounts for 160M parameters;
the softmax layer of the decoder takes a 1K dimensional input (the hidden state of the last layer of LSTM cells) and transforms it into a vector of 80K probabilities. This accounts for 80M parameters;
the input of the encoder is given not only by the hidden state of the decoder, but also by the previous output of the encoder. This requires taking a 80K dimensional vector of possible words and transforming it into a 1K dimensional word embedding. Thus this accounts for 80M parameters.

Does this explanation make sense?

Recurrent connections

As far as I understand, once the cell state of the LSTM is computed, it is transferred as is to the next timestep. Thus there are no additional parameters from the recurrent network (such as, e.g., was the case in this post).


Overfitting is when the test score is way below the training score, and when the latter is high. Here your test score is 0.92 which is not bat at all, so according to this score there is no overfitting here. To see that, you can dedicate a part of your dataset to be a validation set. So that, you will be able to evaluate training score, test score and validation score.

In the same idea you can use cross-validation (if you use scikit here), which will split you dataset into k-folds, k-1 of them will be used for the training and 1 for the test. This process in then repeated so that each of the folds is used as the test set.

If the score varies at each iteration, it could possibly mean that you overfit, if not then you can say that your model is robust

For the second question what do you mean by continuous feature ?


If you are looking for a way to compile code written for the Keras API to code only using the Tensorflow API, there is no ready-to-use solution for that. And I cannot see any good reason why someone would want to do that in the first place...

You can simply look at the Keras backend source code for Tensorflow and rewrite your entire implementation. But once again I don't see any valid reason that would make this worth it.


How can I compute the ideal variance threshold value for my data?

Variance thresholding is used to select those features with a variance above the suggested threshold. Ideally you would want to take in all features with a non-zero variance but I'm not sure of the data youre handling, it would be better to calculate the variance of the individual feature, arrange them in the increasing order of variance and then select that value where the variance sharply increases.
Or you could do PCA and find the order of the importance of features and then set the threshold.


You could also try tSNE, also available in scikit-learn. It is a probabilistic dimensionality reduction technique, specifically designed for plotting data in two or three dimensions, while preserving the original distances between the samples as much as possible.

You should be able to see how well your data is clustered.


Simple neural network implementation in keras

Considering the nature of your problem (disease classification?), probably you have imbalanced data: more negative/benign w.r.t. positive/malicious. In such cases accuracy is not a good metric to assess the quality of a model.

Consider, as example, a dataset with 99 "negative" cases and only one "positive" case, and consider a classifier which predict "negative" all the times. The accuracy of such model is, actually, 99%:

What you want to do in order to deal with unbalanced data is:

1) just use another metric, as ROC-AUC. This can be sufficient if you have a dataset not so much unbalanced, say up to 1:20. This metric is not officially supported in Keras, but you can find a custom callback which calculate the AUC score every end of epoch here.

2) if the imbalance is very big, say 1:1000, you need to balance the dataset. There are lots of techniques, the simplest one is to make copies of the minority class samples, as many as needed to reach the balance. Other smarter techniques exist, and some of then are included in imbalanced-learn for SKLearn.

If you want to have a better overview of techniques and metrics used in such cases, I suggest you this paper.


Your hypothesis about missing colours in your samples affecting results in production could be correct. However, it is trivial to convert images to greyscale as you load them from storage. So keep them in colour, and convert them as you load them if you need black and white.

That way you can try both with and without colour as input and you will have your answer.

This is very little effort to do in practice, and allows you to do the "science" part of data science by comparing two approaches and measuring the difference. This is standard practice, even if you are reasonably certain one approach or another is "the best", it is normal to explore a few variations. When you are not sure, then it is even more important to try it and see.

To test your hypothesis, you could put all the t-shirts of a particular colour in your test set. If that reduces the accuracy of your results with the colour model, it would back up your concern. One fix for that might be to remove colour information from the model, if it is not relevant to the task. An alternative is to collect more data so you have enough samples of different colour shirts. However, you might find if you are fine-tuning a neural network trained on many more images (e.g. Inception v5) that the impact of colour is less even though your samples do not cover all possible T-shirt colours.


Estimating time to travel between two lat/longs

This example is carried out using R.

Suppose there exists a data frame with latitude and longitude coordinates for two locations:

coors <- data.frame(
  lat = c(48.8566,52.5200),
  long = c(2.3522,13.4050),
  stringsAsFactors = FALSE
)

The distance between the two points is calculated as follows:

> p=0.017453292519943295
> a = 0.5 - cos((coors$lat[2] - coors$lat[1]) * p)/2 + cos(coors$lat[1] * p) * cos(coors$lat[2] * p) * (1 - cos((coors$long[2] - coors$long[1]) * p)) / 2
> distance <-12742*sin(sqrt(a))
> distance #Distance in kilometers
[1] 876.0783


How to measure execution time on distributed system

I have some thoughts about your question. I hope it may help you to solve your problem.

  I'm planning to run some experiments with very large data sets, and I'd like to distribute the computation.

In one of my posts, I have done research on topic of evaluation methods of Data Science. With Learning Curve, you can evaluate your experiments learning ability. To talk a bit more, you will fix commodity configuration, and then will run same experiment on the same number of machines with different size of data set you have (from starting from small chunk in size, incrementally increase the size until you reach the whole data set).

To point on, you should avoid having power distribution for result of performance test being run with different size of data sets. To avoid, you should carefully choose step size (step size = amount of increments).

  I have about ten machines available, each with 200 GB of free space on hard disk. However, I would like to perform experiments on a greater number of nodes, to measure scalability more precisely.

For this type question, I have intuitively searched and read materials; afterwards, published as a blog post. At the end of the post, I have briefly talked about how to test your hypothesis on real complex system. If you let me, I want to briefly talk about;

First of all, base requirement should be formed in order to run data set as a whole. The minimum requirement will build your baseline evaluation score which is calculated with one/combination of evaluation metrics you have chosen, or with one/combination of methods being used to calculate Running Time = Computation complexity + Communication cost + Synchronization cost.

After those steps, with an evaluation strategy, add new elements, e.g. new node, to the system you have doing scalability test; meanwhile, for each addition, measure performance w.r.t new system configuration.

Just to note, evaluation strategy must be planned along with considerations of default behavior of parallel and distributed systems. For example, what I mean by behavior is that adding just more cores will, after some point, automatically drop performance of the system not due to your algorithm characteristics. It is because more cores need more RAMs, more hard driver, or etc. In other words, there is a N-Way relationship between hardware components. As a second example, adding more nodes to the distributed system will punished you with more communication and synchronization costs.

As a last step, you will sketch two different graphs with your evaluation results via data analysis program or language (As a recommendation, use GNU Plot or R programming language). Print out and put those results at your desktop, and start to examine them, carefully. According to your investigation, modify/erase + rebuild evaluation strategy and re-do the performance test.

  Are there commodity services which would grant me that only my application would be running at a given time? Has anyone used such services yet?

I have no much experiment on commodity services, but I can easily say whether it grants or not depends on your configuration of services. If you configured say Hadoop to your node as an only service, Hadoop will grant your code will be only running at any time.


Semantic similarity between two or more sentences

Word2vec as the name suggests will create an embedding for each word in your sentence. In order to get a sentence level embedding you would need to average (or combine in some other way) the individual embeddings together.
An example of a model to generate sentence level embedding would be the Universal Sentence Encoder (USE). You may want to try it out and see if it can outperform a word-level model in your use-case.
The original paper can be found here: https://arxiv.org/abs/1803.11175
An example blog post leveraging USE: https://medium.com/@gaurav5430/universal-sentence-encoding-7d440fd3c7c7


How does time-series classification work?

To be clear, here, time-series classification refers to forecasting discrete values. In another context, time-series classification could refer to predicting a single class for the entire time-series (i.e. heart disease vs healthy heart). Thanks @mloning for pointing this out.

When it comes to forecasting discrete outputs, models are trained to predict the next value based on the previous ones, which means that

The input is the historical data up to timestamp t (in your scenario, the data up to week = 201904)
The output is the value at timestamp t + 1 (y when week = 201905)

If you want to predict more than 1 value into the future, you should perform predictions in a recurrent way, i.e.:

use data up to t to predict t + 1
add your prediction to your data
use data up to t + 1 (where t + 1 is your own prediction) to predict t + 2
and so on

How far into the future you want to look is called the horizon. And you are free to use as big of a horizon as you like. Of course, since every new step into the future is based on guesses and not actual data, it is expected that the further you look, the worse your predictions will become. Which makes perfect sense. That's why it's easier to predict what the weather will be like tomorrow than in 7 days.

Splitting data into training and testing is not very different than for a normal classification problem. The only constraint is that your test data should only contain data "in the future" compared to the training data. Here is a good blog about it: https://towardsdatascience.com/time-based-cross-validation-d259b13d42b8

should I train the model for each time series independently?

Short answer: try both approaches and see which one works best

Long answer:

All of this depends on whether you believe that each time-series represent a different "process" or "pattern". It could be that a single model could make good predictions for either time-series. It could also be that both time-series represent different processes and therefore, it might be wiser to train different models for each.

Think about it like a weather forecast. I could have time-series data for a bunch of cities. I could train a model specific to each city, but it might also help to train a model using data from all cities so that it can capture generic weather patterns. This can help if all of a sudden, I need to predict the weather for a new city I didn't know about. Since I didn't know about it, I don't have a model specifically for it, so I would need to reuse one, which ideally should be general enough.


about predict the class for a new datapoint

Since you have your clusters set up already you should be able to calculate cluster centroids and then determine the distance between a each cluster centroid and the new data point. You could follow the same process with random samples of each cluster.


NLP Feature creation from phrase matching

The problem with imbalance is that the optimizer can get a very good score by declaring everything 'not raised'. You need to cheat with your training data by removing that incentive. I would suggest a training set that is balanced 50/50 between the classes. Your evaluation set can still be representative, which will give you a sense of how it'll generalize.