Tell me some things a data scientist/statistician needs to do that can't be done in R/Stata/SAS/SPSS

Why do I need to learn to program in python or Java and learn data structures?

The simple answer - you dont need to lear python or Java or Scala or Julia or anything else if you dont wont to. If you dont know python it doesnt mean that you cant be a data scientist or use ML/DL.
The point is that some tools are better for some tasks, it might be benefitial to lear C++ or Java if you need code that runs faster in production, etc. But you might be totaly okay with well optimized R code. You should do what is best for you and your tasks.
About the things you cant do - I agree with you, you can do basically anything in R, it might be not efficient, but its possible.


XGBoost in final model what does "yes=3,no=4" mean?

I assume it should read as follows:

if condition is true, go to branch 3
otherwise go to branch 4

I am only guessing, but seems plausible


Is it possible to apply PCA on different subsets independently?

You can use mini batch PCA. One formulation is available in sklearn.

Alternatively, you can run PCA on a carefully selected subset of your data. It's very time consuming and I'm not sure it's possible, and its feasibility is task specific.


neural network probability output and loss function (example: dice loss)

I think there is a bit of confusion here. The dice coefficient is defined for binary classification. Softmax is used for multiclass classification.

Softmax and sigmoid are both interpreted as probabilities, the difference is in what these probabilities are. For binary classification they are basically equivalent, but for multiclass classification there is a difference.

When you apply sigmoid, you make sure that all output neurons are between 0 and 1.

When you apply softmax, you make sure that all output neurons are between 0 and 1 AND that they sum up to 1.

This means, when the output is sigmoid, the input data can be in several classes at the same time. For softmax, you force the network to pick one of the classes.

The code you posted here is correct for binary classification, but for multiclass, there is some intricacy when it comes to combining the classes.

Since in your example the target consists of two pixels, each labeled either 0 or 1, you are dealing with binary classification and should use pixel-wise sigmoid in the first place, i.e. the probabilities from your model should be e.g. [0.7, 0.8] or something like that.

Pixel-wise softmax should only be used if each pixel could be in only one of many classes and softmax over all pixels does not make much sense, as this would force the model to pick one out of many pixels to label as 1.


Use the output of 2 hidden neurons in the last hidden layer of a NN to visualize the result of a 4-class classification task

I'm somewhat new to the topic as well, but I think what you are looking for is an autoencoder.  I have only used it in the h2o deeplearning package, but it seems to work well.

The idea behind an auto-encoder is to begin with your inputs, encode down to less nodes (two is nice for visual representation), then decode back to your original outputs (ass accurately as possible).

If the auto-encoder can decode back to an accurate representation of your inputs, then there was sufficient important information withheld in those two nodes.  You can then plot the features that those two nodes held in a two-dimensional plot that would be analogous to a bounded PCA plot.

Also - the auto-encoder is not constrained to linearity like the PCA is.  Below is a quick snippet of some code, hope it helps.

library(h2o)
localH2O = h2o.init(ip = "localhost", port = 54321, startH2O = TRUE, min_mem_size = "3g", max_mem_size = "4g", nthreads = -1)
dat_h2o <- as.h2o(train.df)

unsupervised <-
  h2o.deeplearning(x = 24:356,  # column numbers to use
                   training_frame = dat_h2o, # data in H2O format
                   autoencoder = TRUE, ## unsupervised autoencoding
                   activation = "Tanh", # or 'Tanh' 'Rectifier' 'WithDropout' node activation function, Tanh seems to work best for autoencoding
                   hidden = c(5,2,5), # three layers of nodes, with 5/2/5 nodes, respectively
                   epochs = 1) # max. no. of epochs

## layer 1 corresponds to hidden[1], so it will reduce to 5 variables
## below is roughly similar to using predict(pca.object, newdata)
training_data <- h2o.deepfeatures(unsupervised, dat_h2o, layer = 1)
testing_data <- h2o.deepfeatures(unsupervised, test_h2o, layer = 1)

## explore the second layer with 2 nodes, similar to exploring PC1 vs PC2
train_supervised_features2 = h2o.deepfeatures(unsupervised, dat_h2o, layer=2)

plotdata2 = as.data.frame(train_supervised_features2)
plotdata2$label = as.character(as.vector(dat_h2o[,364]))

## L2 corresponds to layer 2, so use L2
qplot(DF.L2.C1, DF.L2.C2, data = plotdata2, color = label, main = 'Neural network: 5-2-5')


LightGBM results differently depending on the order of the data

A possible explanation is this:

When the order of the columns differ, there is a little difference in the procedure.

What LightGBM, XGBoost, CatBoost, amongst other do is to select different columns from the features in your dataset in every step in the training.

The selections of these columns is done randomly: Let's say your dataset has 20 columns. The root node selects the features 1st, 3rd and 18th, on both datasets the 1st, 3rd and 18th features are different in both possible datasets. This is repeatedly done and in every step there is randomness affecting your ultimate result.


In a pandas dataframe can I convert my column values into numbers?

Well, one way i like to handle this problem (which is a common problem, at least in daily job life) is to convert each possibility in a column with binary value. Let me elaborate a bit. Let's say you have your column animals with 3 possibilities : dog, cat, and horse. You explode your column in 3 differents columns : colDog, colCat and colHorse. And you fill your new columns based on the value of the column animals. For example : if you have dog in the first row, you put 1 in the column colDog, etc.
The problem with handling categorical data with numerical value instead of binary is that you create a hierarchical order between your values. If dog is 1, cat is 2 and horse is 3, then horse will have more impact than cat and dog. Or i think you just want to represent your categories.


Scikit-learn SelectKBest is picking up obviously unwanted Features

There seems to be two possible approaches to your problem :

If they are just identification features that you know aren't informative, you should remove them yourself. SelectKBest - like almost any other EDA tools - works on all the features you provide it, there is no way it knows what features are supposedly uninformative identification features and which are not.

It is possible that somehow the identification feature is informative. I can think of at least two reasons : correlation with time as the instance are entered in order and what you want to observe change with time. Or, if your identification feature is not unique (instances observed trough multiple different times), correlation between your observation. Depending on how your identification feature is built and what you want to achieve, you might want to keep this information or not.


Neural Network Initialization - Every layer?

Yes, every layer that contains weights needs to be initialised.

Every weight is just a number (usually a decimal, floating point number). It has to be initialised with some value so that the algorithm (e.g. backpropagation) has some values to work with.

Any learned parameters need to be initialised, as they will be adapted during training. This means you can inialised e.g. the parameters of momentum, and it is changed over time.

There are "layers" that don't need initialisation. I mean layers that perform some operations, but don't have dynamic weights; weights that are altered during training.

An example would be a standard dropout layer (with no learned parameters). This is called a layer, but doesn't require initialisation as it doesn't have any weights that are changed during training.


Is normalizing the validation set of time series a kind of look ahead bias?

To block the data leakage from the validation set to the training set in step (2),

We should first split the data to training and validation sets, then
Calculate the mean and standard deviation only using training set, and finally
Use this mean and std to normalize both the training and validation sets.

This way no information is leaked from validation set to training set, although validation set would be almost normalized. This deviation denotes an informative difference between distribution of validation and training sets which later [rightfully] leads to a validation error larger than training error.

We may think of normalizing the sets separately, however this way we may lose a potential discrepancy between the distribution of validation and training sets.

For example, in a 1D time-series of price, if the average price in training set is $1.0\$$, we expect that a validation set with average $2.0\$$ causes problem for a model that is trained on prices around $1.0\$$. However, by normalizing the validation set separately to $1.0\$$, we wrongly bring the validation set to a range that the trained model already expects, which leads to underestimation of validation error.

Edit

  in the training data itself - wouldn't every example be biased by the future? - Alon Gouldman

Correct. If training set is $[P_{t_1}, ..., P_{t_N}]$, then for $n \in [1, N)$ normalized point $\hat{P}_{t_n}$ would be biased toward (contains information of) points $[P_{t_{n+1}}, ..., P_{t_N}]$, but this is not a leakage since the entire training set is what we know, thus we can do whatever we want with it. From another point of view, this seemingly cheaty transformation would turn out OK since (1) the shortcoming of the normalization will be revealed in the validation error (more cheat, more validation error) thus we would not be misguided, (2) normalization could benefit us by lowering the validation error (otherwise, we do not normalize)


Meaning of "TRUE" column in R RandomForest output for Importance()?

I don't see any columns like that. Here's a simple example:

library(randomForest)
data(iris)
iris.rf <- randomForest(Species ~ ., data=iris, importance=TRUE)
colnames(importance(iris.rf))

# [1] "setosa"               "versicolor"
# [3] "virginica"            "MeanDecreaseAccuracy"
# [5] "MeanDecreaseGini"

If you're seeing columns named "TRUE" and "FALSE" in the matrix returned by importance, my best guess is that those are the names of columns in your training data.


How to do encode this target vector containing strings

To answer your first question, I think OneHotEncoder from category_encoders should be up to the job, just set return_df=False to get a numpy array which you could wrap around a sparse matrix. I don't see how that would lead to a memory error unless you were operating in a very limited space.
If that is the case maybe you could simply try LabelEncoder, though this may be a less optimal solution.


It depends on the type of dataset you have. For instance, if you are trying to classify different types of flowers, the order in which you train the model on the features is irrelevant.

However, that being said, if you are dealing with non-stationary data such as time series values i.e. stock market prediction, the order of the features relative to time, is very important.


What is the tag mapping for entity recognition in nltk?

Tag mapping according to nltk source

  'CC':   'Coordinating conjunction',   'PRP$': 'Possessive pronoun',
  'CD':   'Cardinal number',            'RB':   'Adverb',
  'DT':   'Determiner',                 'RBR':  'Adverb, comparative',
  'EX':   'Existential there',          'RBS':  'Adverb, superlative',
  'FW':   'Foreign word',               'RP':   'Particle',
  'JJ':   'Adjective',                  'TO':   'to',
  'JJR':  'Adjective, comparative',     'UH':   'Interjection',
  'JJS':  'Adjective, superlative',     'VB':   'Verb, base form',
  'LS':   'List item marker',           'VBD':  'Verb, past tense',
  'MD':   'Modal',                      'NNS':  'Noun, plural',
  'NN':   'Noun, singular or masps',    'VBN':  'Verb, past participle',
  'VBZ':  'Verb,3rd ps. sing. present', 'NNP':  'Proper noun, singular',
  'NNPS': 'Proper noun plural',         'WDT':  'wh-determiner',
  'PDT':  'Predeterminer',              'WP':   'wh-pronoun',
  'POS':  'Possessive ending',          'WP$':  'Possessive wh-pronoun',
  'PRP':  'Personal pronoun',           'WRB':  'wh-adverb',
  '(':    'open parenthesis',           ')':    'close parenthesis',
  '``':   'open quote',                 ',':    'comma',
  "''":   'close quote',                '.':    'period',
  '#':    'pound sign (currency marker)',
  '$':    'dollar sign (currency marker)',
  'IN':   'Preposition/subord. conjunction',
  'SYM':  'Symbol (mathematical or scientific)',
  'VBG':  'Verb, gerund/present participle',
  'VBP':  'Verb, non-3rd ps. sing. present',
  ':':    'colon',


Interpretation of the data through scatter plot

It is forming a triangle because you always have Cancelled items < Total items which is expected.
Using this representation of the data is not so informative as many points are clustered and we can't assess the distribution. You might want to consider a plot such as a 2D histogram of (x=Total items, y=Proportion of Cancelled Items) in order to assess if some relation exists. And if you do so you might want to normalize each X slice so as to visualize the distribution for each range [Xmin, Xmax]
If you are looking for a linear regression, you might want to use scikit-learn to fit a linearRegression model with X = total Items and Y=Cancelled Items and check the correlation coefficient.


Spacy tokenizer that strips HTML/XML keeping the positions

Generally, XML is first parsed. Then, the contents can be analyzed with something like spaCy.
xml.etree.ElementTree is the most common way to parse XML in Python.


KL divergence loss goes to zero while training VAE

take a look at the paper "Generating Sentences from a Continuous Space" by Bowman. In Section 3.1 it is explained why LSTM_VAE tend to this behaviour:
"This problematic tendency in learning is compounded by the lstm decoder’s sensitivity to subtle variation in the hidden states, such as that introduced by the posterior sampling process. This
causes the model to initially learn to ignore ~z and
go after low hanging fruit, explaining the data with
the more easily optimized decoder. Once this has
happened, the decoder ignores the encoder and little to no gradient signal passes between the two,
yielding an undesirable stable equilibrium with the
kl cost term at zero. We propose two techniques
to mitigate this issue."
....


Is stratified sampling necessary (random forest, Python)?

If the number of values belonging to each class are unbalanced, using stratified sampling is a good thing. You are basically asking the model to take the training and test set such that the class proportion is same as of the whole dataset, which is the right thing to do. If your classes are balanced then a shuffle (no stratification needed here) can basically guarantee a fair test and train split.

Now your model will be capable or at least enough equipped to predict the outnumbered class (class with lesser points in number). That is why instead of just calculating Accuracy, you have been given other metrics like Sensitivity and Specificity. Keep a watch on these, these are the guardians.

Hope this helps.


For every ML Scenario, there is a notebook section where you can upload/create files with code and/or Jupyter notebooks:


Why is pre and post silence important when collecting speech data?

Silence is often useful when segmenting the raw data into suitably sized samples for the machine learning methods. It is practical to run recording for several minutes at a time, but often the input to the machine learning model should for example be 1 second clips. Each clip should avoid cutting of a word midways.
Having silence allows to segment words or phrases using a simple threshold algorithm.


How does the validation set get used in the training phase?

The validation set is used just to give an indication about the network's performance. The hyperparameters are tuned based on the training data itself.
Based on the optimizer, the neural network calculates the loss for a training sample. Based on the computed loss, the weights of the network are updated. This is repeated for every sample in the training set.
The validation set just uses the latest weights of the network on unseen data to check the model's performance.


weight sharing among neurons at same depth

I agree that these pictures are a bit confusing. I think they are all hinting at the following:

As you know, CNNs involve splitting up the image into many small "blocks" and multiplying each block pairwise by a filter (actually, by several different filters). Weight sharing means that all the blocks use the same set of filters -- blocks in the upper-left corner, middle, and bottom of the image all use the same set of filters.
There is no weight sharing across the channels -- we don't use the same filter banks for the red channel and the blue channel. This makes sense because different classes have different signatures in each channel. There are various ways to think about this. Personally, I find it easiest to visualize that both the "block" and the filter are three-dimensional, and the pairwise multiplication is across all three dimensions.

So, to your pictures:

The top one shows that $x_{i-1}$, $x_i$, and $x_{i+1}$ are three different channels, and so use three different weights (colors). But when switching from block $i$ to block $j$, the same weights (colors) are reused.

The middle one has four columns:

The left column is the image itself -- three rows, one for each channel.
The second column is one filter -- also three dimensional. This also shows that each channel of the image is multiplied pairwise against each filter.
The third column I assume is a second filter bank, which works the same way as the original filter bank (CNNs usually apply many different filters at each stage of the network, so that multiple patterns can be learned).
Then the fourth column is the output, i.e., the next layer in the network.

The bottom one is not representing the filter bank, but actually represents the subsequent layer of the network (i.e., the output from the convolutional operation). In this subsequent layer, each "channel" refers to the output from a different filter in the preceding layer. Given this, the picture makes sense: everything within the same channel was formed by applying the same weights to different parts of the image.  And each channel is in a different color because they were formed using a different filter, and therefore different weights.


SMOTE for multi-instance learning i.e num_rows(x_train) > num_rows(y_train)

What you have here is called Multi-Instance Learning. From Wikipedia

  In machine learning, multiple-instance learning (MIL) is a type of supervised learning. Instead of receiving a set of instances which are individually labeled, the learner receives a set of labeled bags, each containing many instances.

Source: https://en.wikipedia.org/wiki/Multiple_instance_learning

The approach you take in this case is different. You need to bring the Multi-Instance Learning problem into a Single-Instance Learning one. One way you can do this is:

Perform for each bag of instances a K-Means clustering
Calculate the Hausdorff distance of each instance from each of the cluster
Use those distances as features and keep the label from the y_train set

Then apply SMOTE on the new dataset (where you have one row for features and label) and any kind of model of Single-Instance Learning.

You can find details in this Review of Multi-Instance Learning and Its applications


Why does my LSTM perform better when randomizing training subset vs. standard batch training?

Couple of points:

1) When you 'run this training ten times' does that mean you completely re-initialize the model and train from scratch?

2) I'm not entirely sure that you're loading the data correctly from the code you've supplied. What are the shapes of X_train and y_train?

3) It may depend on your data and your selection process. Time-series data is typically temporally correlated -- the closing value of the Dow Jones on Thursday is more likely to be close to the values on Wednesday and Friday than on Monday or in two weeks. It may be that if you sample 100 data points in a row and they're temporally correlated, you're giving your model an easier problem, because those values are more like each other than values elsewhere in the series.

Good luck!


Predict output sequence one at a time with feedback

I assume, that your data looks somehow like this (a pair of question and answer)

Question,Answer
How are you?,I'm fine.
How are you?,OK.
How are you?,Very well, thanks.

I'd suggest to transform it in three features: Question  (remains the same),
InputSeq is the prefix of the answer generated in all possible lengths between zero and the number of words minus one.

Finally the NextWord is the expected result.

So for your initial data you ends with the following setup.

Question, InputSeq, NextWord
How are you,,I'm
How are you,I'm,fine
How are you,,OK
How are you,,Very
How are you,Very,well
How are you,Very well,thanks

The model takes Question and InputSeq   as features and predicts the NextWord.


You can simply use
df.plot.area()

Found here after a quick google search.


It behaves like a fully connected network regarding the channels dimension. For each pixel location, the CNN takes the weighted sum of all pixels in that location on the channels dimension. The weights are of course trainable.

This is only for the first layer, though, because afterwards layers have an arbitrary number of channels that has nothing to do with the colors in the original image.

You could also read this answer where I explain how the operation works in a bit more detail.


How to fill missing value based on other columns in Pandas dataframe?

Assuming three columns of your dataframe is a, b and c. This is what you want:

df['c'] = df.apply(
    lambda row: row['a']*row['b'] if np.isnan(row['c']) else row['c'],
    axis=1
)

Full code:

df = pd.DataFrame(
    np.array([[1, 2, 3], [4, 5, np.nan], [7, 8, 9], [3, 2, np.nan], [5, 6, np.nan]]),
    columns=['a', 'b', 'c']
)
df['c'] = df.apply(
    lambda row: row['a']*row['b'] if np.isnan(row['c']) else row['c'],
    axis=1
)


What you need is perhaps Named Entity Recognition with custom entity dictionary. See this example:

Many packages like NLTK or Spacy have a large dictionary of such entities that enable models to identify them without using regular expressions. However, those pre-defined entities often is not applicable to one's application, thus won't recognize what you are looking for. That is why you need to spend time to update/add a list of such entities (products in your case). Also you mentioned that there are misspelling, variations etc. that make it harder. Here is where you need to expand your dictionary for such products with a fuzzy-matching algorithms to account for such edge cases programmatically.
You may start reading this article to the idea. More about Named Entity Recognition: a Blogpost, or the official Named Entity Recognition 101 by Spacy. Spacy supports adding products name in the Entity list, see page. Check out this question in stackoverflow too. But it is not easily available as you may have thought!!


Need a little help Understanding how to build model's in Keras

I have implemented your model to the astonishment, there is a very minute error that is hard to notice.

The way, I was able to get better accuracy is by changing the optimizer to "SGD" or "ADAM".

As you have used "ADADELTA" which is an extension of "ADAGRAD" optimizer. In "ADAGRAD" has good performs on sparse data & while training a large scale neural network. Its monotonic learning rate usually proves too aggressive, stops learning too early.

Refer to this link for understanding on optimizers


How do I deal with data that has only limited target values?

I would try two different approaches:

interpolate the missing values on a user level.
work with the sunset of rows for which we actually have the glucose level.

Then, I would compare the test accuracy of the model built with both methods. Remember that your test set has to be composed of rows for which you have the glucose level - you cannot build it with interpolated data, that would be cheating!


Classification of multidimensional data to multidimensional clusters with a varying subcluster structure

You can pose this as multi-class classification problem (all sub-clusters becomes classes). Since, your input length varies, you should pad your input to get the length equal for all inputs. You can then use neural networks (1D Conv layers followed by Dense and Softmax) to classify this.
An alternate approach to do this would be using tree-based approach where missing values can be handled. Look at CatBoost classifier. This would also require you to pose your problem as multi-class classification.


Confused between optimizer and loss function

Think of loss function what to minimize and optimizer how to minimize the loss.
loss could be mean absolute error and in order to reduce it, weights and biases are updated after each epoch. optimizer is used to calculate and update them.


In simple words, PCA (Principal Component Analysis) is a dimension-reduction method where you try to reduce the number of variables from a larger number to a smaller one without any loss of the important information available in the data.
Mathematically, it is a projection of a higher-dimensional object in a lower-dimensional vector space.

A great example in a real-life scenario is where we see 3D objects in a 2D Television. Reference

Another good explanation on the intuition on PCA

Hope it helps.


How can I do the correlation between two estimators?

There are probably many options, I'd look at inter-annotator agreement.

An alternative option would be to to evaluate one (e.g. by accuracy or f1-score) using the other as the gold standard.


Doing a fine tuning after a transfer learning

Generally, it will increase the performance of the model, but you'll have to experiment with it and check for your case. Try it, till you get a better result. If after many trials also you are not able to increase the performance of the model then it means that you are limited by the architecture of the pre-trained model.
Refer to this http://cs231n.github.io/transfer-learning/ for more details.


Using Cross Validation technique for a CNN model

Question 1: Why do most CNN models not apply the cross-validation technique?

$k$-fold cross-validation is often used for simple models with few parameters, models with simple hyperparameters and additionally the models are easy to optimize. Typical examples are linear regression, logistic regression, small neural networks and support vector machines.
For a convolutional neural network with many parameters (e.g. more than one million) we just have too many possible changes in the architecture. What you can do is to do some experiments with the learning rate, batch size, dropout (amount and position) and batch normalization (position). Training a convolutional neural network with a huge dataset takes quite a long time. Doing hyperparameter optimization would just be total overkill. Often in papers, they try to improve the results of other research papers. It is not the goal to get better results by improving the chosen hyperparameters but rather to come up with new ideas to solve the given task but with better accuracy or less computational effort.

  Question 2: If I use cross-validation how can I generate confusion
  matrix? can I split dataset to train/test then do cross-validation on
  train set as train/validation (i.e. doing cross-validation as
  train/validation except for the usual train/test) and at last use test
  set the same way? or how?

In order to do $k$-fold cross validation you will need to split your initial data set into two parts. One dataset for doing the hyperparameter optimization and one for the final validation. Then we take the dataset for the hyperparameter optimization and split it into $k$ (hopefully) equally sized data sets $\mathcal{D}_1,\mathcal{D}_2,\ldots,\mathcal{D}_k$. For the sake of clarity let us set $k=3$. Then for each possible hyperparameter combination that we want to test we use  $\mathcal{D}_1$ and $\mathcal{D}_2$ to fit our model and we use $\mathcal{D}_3$ to validate our model. Then we do the same with $\mathcal{D}_2$ and $\mathcal{D}_3$ and use $\mathcal{D}_1$ for validation. Then we do the same with $\mathcal{D}_1$ and $\mathcal{D}_3$ and use $\mathcal{D}_2$ for validation. We will get $3$ confusion matrices for every possible hyperparameter configuration. In order to derive a metric from these three results, we take the mean of these confusion matrices. Then we can scan through all averaged confusion matrices so select the hyperparameter configuration that was the best (you have to define what parts of the confusion matrix are important for your problem). Finally, we pick the 'best' hyperparameters and calculate the prediction performance on the final validation set. This performance metrics are the ones that you report.


How to extract entities from text using existing ontologies?

I think you are looking for Spacy, Polyglot and AllenNLP to find your NERs.


Why L2 norm in AdaGrad update equation not L1?

The reason is because the article introducing the method, which can be found here, has proved bounds on the regret function using inequalities about the L2 norm.
The square is also explained by the fact that is the diagonal of some positive quadratic form.
However I don't know if anyone has ever considered using AdaGrag with L1 norm instead of L2.


How to find possible subjects for given verb in everyday object domain

If you want something quick, I think pattern is the best tool for the job. It provides a ready-to-use multilingual parser that you can use in the following way:

import pattern
from pattern.en import parse
s = 'I put water in the vase'
s = parse(s)
print s
# output = I/PRP/B-NP/O put/VBP/B-VP/O water/NN/B-NP/O in/IN/B-PP/B-PNP the/DT/B-NP/I-PNP vase/NN/I-NP/I-PNP

Once you have a string like output above, you only need regex parsing to extract every sequence of tokens whose tags match the sequence [B-NP, B-VP, B-NP].

NP stands for "noun phrase" and VP stands for "verb phrase". In English, virtually every sequence consisting of a noun phrase, a verb phrase, and a second noun phrase, all in strict adjacency, is a subject-verb-object sequence, so this should give you what you're looking for.

pattern's parser will also be able to handle some non-strict adjacencies (e.g. intervening adverbs and adjectives between the three phrases in the subject-verb-object sequence).

However, pattern is not terribly sophisticated -this will give you some Precision and some Recall, but not terribly high numbers. If you need high-quality parsing, you should try the Stanford parser's Python implementation or spacy.

Hope this helps!


Binary Neural Network Classification or Multiclass Neural Network Classification?

I think you are making things more confusing then they are.

Binary

In this case you have two possible outputs:

Obama = 1.
Not-Obama (who in this case can only be Romney) = 0.

Multi-Class

In this case you have k possible outputs, for example when k = 4:

k = 0: Obama
k = 1: Romney
k = 2: Clinton
k = 3: Bush

There are approaches to tackle multi-class classification as binary classification which are called One-vs-rest classification and One-vs-one classification, other classifiers, such as Random Forests, are able to deal with a multiclass setting in a natural way. For a brief report, see here.


Loading Street View House Numbers Dataset in Python

If you already have a <HDF5 object reference>, and you want to know the real data, here is one approach:

import h5py
labels_file = './sv/train/digitStruct.mat'
f = h5py.File(labels_file)
ref = f.get('/digitStruct/name').value[0][0]  # <HDF5 object reference>
print(f[ref].value)

In my environment, f[ref].value shows:

In [84]: f[ref].value
Out[84]:
array([[ 49],
       [ 46],
       [112],
       [110],
       [103]], dtype=uint16)

For more information on object references, you can refer to http://docs.h5py.org/en/latest/refs.html


High accuracy on test-set, what could go wrong?

There are many possible ways to make errors giving you a really huge score on test in Data Science. Here are a few examples :

Your test set is also in your train set : Imagine you have data from 2010 to 2020, and use 2019 as test. If you trained on all 2010-2020 without excluding 2019, then you'll test on a data well known by the model, since it was used to train it. Moreover, if the model tends to overfit (so fits "too perfectly and precisely" with training set), you could achieve a 99% accuracy

Data leakage : This is a phenomenom in which your test set contains info that you shouldn't have in real new cases. Example : you're using Titanic dataset, predicting who dies and who survives. Imagine now the dataset has an attribute called "Hour of death", empty if the person survived, and filled with an hour if he died. Then your model will just learn "if this attribute is empty then the person survive, else he died". On your test set, you'll apply your model, knowing this info that you shouldn't know if you had to predict true new cases

Wathever happens, a 99% accuracy have to make you wonder and look for errors, this is almost impossible to achieve unless your problem is REALLY easy (and might not need a Data Science model at all)


How to club the orders in such a way that maximum number of items are common amongst them?

Have you tried using frequent itemset mining?

It finds the most frequent item combinations.

The key idea is that when item A is not frequent, you don't need to further explore any combinations including A.


welcome to the forum, I will try to clarify some things.

First of all, when talking about regression you do not calculate accuracy. Accuracy is a metric used in classification tasks, it basically measures how much labels your model got right divided by the total labels.

You are working in a regression task, that is, you are trying to predict not labels, but continuous values as your target. By default, scikit-learn estimators calculate as regression scores the R_squared metric. Intuitively, R_squared measures how much of variance your model explains. It is calculated by the below equation:

(Image taken from scikit-learn r2_score page: https://scikit-learn.org/stable/modules/model_evaluation.html#r2-score

Basically, I like to think that R2 is testing your model against the simplest possible one in a regression task, that is, just predicting the mean for your target variable. It makes sense talking about accuracy as an analogy, however I wanted to make clear that they are different things.

Second, talking about results, RMSE and how can you measure the effectiveness of your model, in linear regression context it will be highly dependent of the problem your are trying to solve. Maybe you are a social scientist trying to predict with some data the number of voters in certain candidate, an R2 of 0.4-0.5 would be amazing (?). Maybe you are sure that this is not sufficient because your variable are strongly related to your target variable (some problems and physics) and a R2 of 0.80 is unacceptable. Another thing is RMSE, it tries to measure the variance of your model that is, how far off you usually are when predicting your target. I had a problem where I was prediciting the number of days of a certain event to happen, and my variance was 15 days off, and this was not helpful at all.

A highly recommend you to read the Chapter 3 of introduction to Statistical Learning Book that will help you a lot (Actually, the whole book). They have a free version here: http://www-bcf.usc.edu/~gareth/ISL/

I hope this helps. Also, any problem to understand what I wrote, please let a comment. And if anyone spots something wrong in my answer, tell me please!


But the problem of this dataset is that we have unbalanced data

I think that the way to fix your problem is to use something like SMOTE or one of its variants. (My favorite is SMOTE-ENN, but go with what works the best). Here is an implementation that you can use to fix the class imbalances. I don't know if this will solve your specific problem, but it is worth a shot.


Short version: this is just terminology, but arguably K-NN doesn't actually fit a model.

Conceptually, K-NN is a lazy learning model. This means that, in a sense, there's no fitting until a new instance arrives to be classified or a value has to be predicted (depending if you're using K-NN for classifying or regression, both are possible). Even then, using the term "fitting the model" is a bit out-of-place, in my opinion.

I think your confusion is in the sentence "model fitting happens to reduce error". No, it does not. Model fitting is simply getting a model (for example, a family of data distributions) and fitting the model by finding the parameters that better describe the data (thus choosing a member from that family).

Maybe you're thinking of neural network's epochs as "fitting"? What happens there is that there's a family of possible models (all the possible values for each weight in the network), and fitting the model is just finding the best possible values. The fact that neural networks do so in an iterative manner does not mean that fitting is an iterative process, fitting is the end result.

  So training set and test set is only for do things below, right?

Well, that's true for every model, yes. Training and test set separation serves the only purpose of evaluating the model on the chosen metrics. At the end of the day, the final model you choose to deploy, be it lazy or eager, will use all the available data (or a portion/transformation of it, in some cases), not just the training set.


How Tensorflow text prediction predicts without softmax activation

The logits have the distribution according to the len of vocabulary and the model training. So, you can use np.argmax(logits) to get the prediction, but normally to the application of generating script is more interesting to take into account a factor of aleatory which in this case is the function "random_categorical" which is used to get the value according to the probability. The value 29 appears in the distribution, you didn't see, because you execute the function a lot of times and every execution, "random values" will appear. If you want the exact value, use np.argmax.


How to deal with class imbalance in a neural network?

This is for classification, and I am not sure if it is possible to extend them to reinforcement learning.

As you figured out, accuracy should not be used as a metric for a dataset as imbalanced as the one you have. Instead, you should look at a metric such as Area Under Curve(AUC). If you would have infinite data, then you could just rebalance and remove some of the data from the class that has the most samples. However, in many cases data is sparse and you want to use as much of it as possible. Removing data can have a disastrous effect on many applications.

So what are good and convenient ways of handling this?

Add weights to the loss function. One weight for class A and one for B. By increasing the magnitude of the loss for the B class the model should not get stuck in a suboptimal solution that just predicts one class.
Use another objective(loss) function. F1-score can, for example, be implemented and used as an objective(loss) function.

What is great with these approaches is that it will allow you to use all the data.


GridSearch mean_test_score vs mean_train_score

When you do k-fold cross-validation, you train k models, each one of them leaving the proportion $1/k$ of the data out. For each of the models, you can compute its train error and validation error. The train error will be the error on the data selected to train the model, and the validation error will be the data left out of the training.

For this reason, you have k training errors and k validation/test errors, and computing their averages will give you the quantities you are talking about.


If you're referring to the fact that your dataset is small:

You should use k-fold cross validation. This will let you evaluate your model on all 279 instances

If you're referring to the class imbalance being 31:202 in train and 8:48 in test:

Use AUROC and PRC to eliminate bias in thresholding
Also see MCC


100% Accuracy on test dataset using a previous developed model oputput

Welcome to the community. If I understood correctly:

you have is a dataset whose target labels (aka ground truth) you don't know, so you figured these out by assigning the output of another model from someone else --> I guess this pre-trained model was built with the same dataset right?
you built a new model on this dataset, using as target values the other model's output as your ground truth, giving you a 100% accuracy --> you convert the predicted probability to the corresponding label is it?

This might mean:

your model is providing the same output labels as the other model trained on the same dataset; this is not strange, mainly if the models are of the same type (e.g. tree-based models as your xgboost), mainly assuming the pretrained model outputs fit the real target values
the data is quite easy to "learn", so both models provide similar results over the same target labeling
to take into account: accuracy might not be an optimal metric, taking a look at how much unbalanced is the dataset (it is, what is the ratio between the different classes).

Nevertheless, I would make sure to find the real and precise corresponding targets to go on with confidence on your problem.


How to improve accuracy of Named entity recognition (NER) tagger on local data?

@Ravikm, excellent question. In Spacy, you can assign a word manually. For example, "Tesla" to ORG. Source: screenshot from Jose Portilla's NLP course on Udemy.


Use a Jupyter notebook. Write good descriptions in Markdown between blocks of code. Here are some examples showing machine learning in Python:

https://github.com/ogrisel/notebooks

https://github.com/ogrisel/notebooks/blob/master/Labeled%20Faces%20in%20the%20Wild%20recognition.ipynb


Can ReLU replace a Sigmoid Activation Function in Neural Network

To answer your question directly, yes you can replace the sigmoid activation function with the ReLU activation function.

I think the real question is "should you?"  This is a much harder question to answer.

The sigmoid function returns a number between 0 and 1, but the path of the function is never constant, decelerating as it approaches 0 and 1, and peaking at 0.5 when the input value is 0.

On the other hand, the ReLU function returns a 0 for input values less than 0, while input values above 0, the function returns a value between 0 and 1.

So, sigmoid will only "theoretically" return a 0 at the lowest most extreme value of the input $x$. For the mean input value of $x$, the sigmoid returns 0.5, and for the max value of $x$ will "theoretically" return a value of 1.  Based on a normally distributed variable, you would not expect to see a 0 or 1 very frequently.

With the ReLU function, there is nothing theoretical about the possibility of the function returning a 0.  It will return a 0 for approximately half the input values.

One other point, the activation function can absolutely change the behavior of a neural network, but a poorly chosen activation function may not necessarily prevent a NN from converging, but it will probably require a different model architecture, and probably make it harder to train.

At this point, the community appears to rely on heuristics and empirical assessment to choose one function over another.  There are certain patterns that seem to be accepted by the community as a general rule of thumb in certain domains. There are also, some patterns that are pretty well accepted as the only valid pattern. For example, when using the softmax activation function to determine the output, it is generally accepted that the previous hidden layer must employ the sigmoid function.

My guess is that most practitioners start with a sigmoid activation function unless there is domain knowledge that implies that it would be inappropriate, but I expect many different architectures are evaluated during model selection.


Does orange transfrom categorial variables into dummy variables when using hierarchical clustering?

The documentation states this is not the case. Categorical values are compared as either same or different.


Xgboost does the feature selection for you. If you want to report how valuable certain features are for prediction, print the feature importances. Those will, however, just tell you "feature $x_1$is very important for predicting the outcome, feature $x_2$ is nearly useless for predicting the outcome, etc.".

To get risk factors with p-values, you need a more interpretable model. You have to use linear classification, for example. Then you can make statements like "high $x_1$ correlates with a positive outcome".

If you really want to only use a subset of features, train an Xgboost model on a validation dataset and drop features that have low importance. Then run an Xgboost model with the remaining features on the remaining training set.

You need to think about what you want: Do you want to explain observations and extract explicit knowledge from data and try to find possible causal links? Use linear methods. Do you want to predict an outcome for new patient data? Use gradient boosting. You can obviously do both.


You might actually encounter problems if you model this as a regression problem without a suitable transformation. For instance, we know that most calls are probably answered during daytime and less during night time and early morning. A linear regression would have difficulty because the relationship is likely curvilinear, not linear. For the same reason, treating this as a classification task with logistic regression would also be problematic.

As suggested by other respondents, reclassifying your data into time periods will help, and I'd suggest you try something like a decision tree or random forest first.

That all said, this might be a case for simple descriptive statistics. If you plot the proportion answered calls by time of day (split by city or any other demographic), is there a clear best time?  If so, why complicate things with a model?


This is exactly your code just with digits data:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil
from sklearn.tree import DecisionTreeClassifier

np.random.seed(0)

# X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()

digits = load_digits()
X, y = digits.data, digits.target

indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10,
    param_range=param_range,n_jobs=psutil.cpu_count(),
    scoring="accuracy")

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.figure(figsize=(10,10))
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
#plt.fill_between(param_range, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.2, color="r")
plt.plot(param_range, train_scores_mean, label="Training score",
             color="r")
#plt.fill_between(param_range, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.2, color="g")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
             color="g")

plt.legend(loc="best")
plt.xticks(param_range)
plt.savefig('DSSE.jpg')
plt.show()

And this is the result:

So whatever the problem is, it's in your data that you didn't explain how you created it. Either double check it or post it here so your question can be answered. I assume somewhere in data preparation you made X and Y identical by mistake.


Remove noise by clustering on which step of pre-processing is better?

Looking at your different steps, the important thing to do is check which step would be affected by outliers.

Removing missing values is not affected because this step is not dependent on other data points present (or not) in the dataset.
However, normalizing your data is. Indeed, let's say your outliers contain extreme values, this will affect the normalized values of the non-outlier data points.

Therefore, intuitively, I would perform your noise removal at the very start or after step 1.

Ultimately, you should see what works better for your task. Perhaps removing outliers doesn't help as much as you'd expect. Same with your pre-processing. Feel free to experiment!


Clustering analysis for observations with lists as data

K-means would be a fine clustering method for you to start with, though you will have to provide the number of clusters you wish for it to return (not sure if you know that/can figure it out). Otherwise check out DBSCAN.

As for the mix of numerical vs categorical data types, all you will need to do is one-hot encoding on your categorical variables. What that does is that it will take all of the known possibilities for a category and it creates new features out of them. A 1 is assigned if the sample is part of that category, and a 0 if it is not. In this way you can use numerical and categorical at the same time, just make sure to normalize your numerical data!


Is there a real life meaning about KMeans error?

The K-means Error gives you, what is known as, total intra-cluster variance.

Intra-cluster variance is the measure of how much are the points in a given cluster spread.

The following cluster will have high intra-cluster variance

In the image below, even though the number of points are same as that of the image above, the points are densely distributed and hence will have lower intra-cluster variance.

K-means Error is interested in the total of such individual cluster variances.

Suppose for a given data, if clustering 'A' forms clusters like the first image and clustering 'B' forms clusters like the second image, you will in most cases choose the second one.

Although this does not mean that the K-means Error is a perfect objective to optimize on to form clusters. But it pretty much catches the essence behind clustering.

Code used for cluster plot generation -

import numpy as np
from matplotlib import pyplot as plt

sparse_samples = np.random.multivariate_normal([0, 0], [[50000, 0], [0, 50000]], size=(1000))
plt.plot(sparse_samples[:, 0], sparse_samples[:, 1], 'b+')
axes = plt.gca()
axes.set_xlim(-1000, 1000)
axes.set_ylim(-1000, 1000)
plt.show()

dense_samples = np.random.multivariate_normal([0, 0], [[5000, 0], [0, 5000]], size=(1000))
plt.plot(dense_samples[:, 0], dense_samples[:, 1], 'r+')
axes = plt.gca()
axes.set_xlim(-1000, 1000)
axes.set_ylim(-1000, 1000)
plt.show()

In both cases, a 1000 datapoints from a Bivariate Normal Distribution are  sampled and plotted . In the second case, the Covariance Matrix is changed to plot a denser cluster. np.random.multivariate_normal's documentation can be found here. Hope this helps!


What are the challenging benchmarks for self driving cars object detection beside Kitti

You may find Apolloscape dataset interesting. It was recently released by Baidu. From the article describing the dataset:

  In this paper, we present a large-scale open dataset,
  ApolloScape, that
  consists of RGB videos and corresponding dense 3D point
  clouds.

Apart from the article,
there have been some blog posts with summarized information on the dataset.


In R you can load the csv file easily by using the method read.csv and get the summary of the data using the method summary, you will get the most of the basic statistics of each columns like Min, 1st Qu, Median, Mean, 3rd Qu, Max and NA counts for numeric/integer rows and for string columns you will also get the some statics of count of different type of string if it has repeated multiple i.e factors.

Sample code:

Loading the csv file and check the few datasets:

> df <- read.csv("SampleData.csv")
> head(df)
  name age       Type
1    A  34     Active
2    B  56 Cancelled
3    C  12     Active
4    D  32 Cancelled
5   Z   34     Active

Basic statistics if numeric types: avg, stdev, mode, median

> summary(df)
 name        age               Type
 A :1   Min.   :12.0   Active    :3
 B :1   1st Qu.:32.0   Cancelled :2
 C :1   Median :34.0
 D :1   Mean   :33.6
 Z :1   3rd Qu.:34.0
        Max.   :56.0
>

Column type and Cardinality detection

> sapply(df, class)
   name       age      Type
  "factor" "integer"  "factor"
>
> nrow(df)
[1] 5
> ncol(df)
[1] 3
>

Find relationships between columns, if column A is X, then column B is always Y

For this you have to do some Basic calculation and you can go through this cookbook: http://www.cookbook-r.com/

I have attached Sample CSV Image here


Which cloud platform to maximize my impact as a data scientist?

It has been my experience that transitioning from local modeling to large-scale distributed programming is a lot more work than most data scientists realize, and leaves little room for anything BUT data engineering, similar to what @Emre said above.

If you're developing an infrastructure yourself (say, spark) using GCP or AWS VMs, installing and maintaining that is a LOT of work.  This is doubly true if you're running a multi-tenant system and/or supporting production jobs.  You will constantly be trying to solve the 'why didn't my job run?' or 'why does it take my job 14 days to run?' problems.

If you're using the built-in data science infrastructure to those systems (Redshift, Athena, ElasticSearch, etc), you can save some time but it is still EXTREMELY non-trivial to manage.  There is a reason why, year after year, Data Scientists' favorite tools always include Databricks or similar -- because managing these things is a pain, and requires a completely different skillset than actual modeling.

All that being said, I have two suggestions.  First, AWS is more mature and has more community answers that help overcome hurdles than GCP does.  You will run into issues with its IAM systems (which become a necessary evil of responsible big data engineering), and the quirkiness of its various products (lambda, for instance, will only run < 3m scripts), but overall everything you're trying to do has already been done and documented by someone else.  It is the better choice, IMO.

However, I'd urge you to take a closer look at a managed platform like Qubole or Databricks (I work for neither).  I set up and maintained a Qubole / AWS environment for over a year.  I learned a ton about data engineering, architecture, elastic spark infrastructure and all the nuances / pitfalls / limitations of distributed computing that they don't tell you about in the rote documentation, but was still able to maintain a functioning system.  My use case didn't include deep learning, but they could have, by me turning a few knobs.

Then, once you have taken your lessons from these environments that work, you can architect, deploy & support your own big data infrastructure to your heart's content (which will now be your full time job).  Hope that helps.


Autoencoder train and test accuracy shooting to 99% on few epochs

The issue turned out to be MinMaxScaler. The issue was sorted after using standardization instead of normalization. (I am not sure as I did not completely verify but best possible guess as compressed output was mostly mapping one feature) One of the feature X had much higher values after normalization then the other features (probably due to outliers in other features). So the the feature X pretty much mostly contributed to the Accuracy.


What is the standard terminology for the output matrix of a hidden layer in a CNN?

Not that I’m aware. The 2-d matrices indexed by the third dimension are often called channels. Other than that, “intermediate representations”, maybe? But since it isn’t really standard, you should explain whatever you decide to call it, to make sure it’s unambiguous.


How to handle different categorical embedding sizes in hold out data set

One possibility would be to represent the zip codes using some transformation that could be applied to new (unseen) zip codes as well. For example, could you re-represent zip codes as latitude + longitude?


using unsupervised learning algorithms on images

Have you included dropout in your model? It can help avoid overfitting issue.

For your problem, yes, you can use auto-encoders, GAN, etc. for feature learning.
However, I'm not sure if unsupervised learning can help, since it's more like a training issue. Your have label with your data so supervised learning is ideal, plus supervised learning generally shows better performance than unsupervised in image classification. You might want to check the false classification examples in your dataset, and try to alter the CNN structure based on that, which would be a more direct way.


How does " Sparsity of connections" in CNNs causes the network to have less parameters?

In fact these involve different aspects of parameters in a CNN.

Parameter sharing, means one parameter may be shared by more than one input/connection. So this reduces total amount of independent parameters. Parameters shared are non-zero.
Sparsity of connections means that some parameters are simply missing (ie are zero), nothing to do with sharing same non-zero parameter. Parameters in this case are zero, ignored. That means that not necessarily all (potential) inputs to a layer are actually connected to that layer, only some of them, rest are ignored. Thus sparsity of connections.


How many observations in a neural networks dataset?

A neural network is nothing but a set of equations. And the basic rule of any set of equations is that you must have as many data points as the number of parameters.

The parameters of any neural network are its weights and biases.

So that means that as the neural network gets deeper and wider, the number of parameters increase a lot, and so must the data points.

This being said, the more proper and detailed way to know whether the model is overfitting is to check if the validation error is close to the training error. If yes, then the model is working fine. If no, then the model is most likely overfitting and that means that you need to reduce the size of your model or introduce regularization techniques.


Updating the weights of the filters in a CNN

In a normal neural network, each neuron has its own weight.

This is not correct. Every connection between neurons has its own weight. In a fully connected network each neuron will be associated with many different weights. If there are n0 inputs (i.e. n0 neurons in the previous layer) to a layer with n1 neurons in a fully connected network, that layer will have n0*n1 weights, not counting any bias term.

You should be able to see this clearly in this diagram of a fully connected network from CS231n. Every edge you see represents a different trainable weight:

Convolutional layers are different in that they have a fixed number of weights governed by the choice of filter size and number of filters, but independent of the input size.

Each filter has a separate weight in each position of its shape. So if you use two 3x3x3 filters then you will have 54 weights, again not counting bias. This is illustrated in a second diagram from CS231n:

The filter weights absolutely must be updated in backpropagation, since this is how they learn to recognize features of the input.  If you read the section titled "Visualizing Neural Networks" here you will see how layers of a CNN learn more and more complex features of the input image as you got deeper in the network. These are all learned by adjusting the filter weights through backpropagation.


compare pandas dataframes with different lengths

You can accomplish your task by using the merge operation in pandas as follows:

In [16]: df1 = pd.DataFrame({'Object':['cup', 'brick', 'board', 'stone'], 'id':[2, 8, 9, 6]})

In [17]: df1
Out[17]:
  Object  id
0    cup   2
1  brick   8
2  board   9
3  stone   6

In [18]: df2 = pd.DataFrame({'Thing':['cup', 'board'], 'color':['blue', 'grey']})

In [19]: df2
Out[19]:
   Thing color
0    cup  blue
1  board  grey

In [20]: df = df2.merge(df1, left_on='Thing',right_on='Object', how='inner')

In [21]: df
Out[21]:
   Thing color Object  id
0    cup  blue    cup   2
1  board  grey  board   9

and then drop the column (df.drop('Object', inplace=True)) that you don't need. For more details look at the official documentation here. Also, check out this to see how you can use merge and join operations in pandas to do all kinds of dataframe manipulations!


There's some confusion about different kinds of output and their corresponding evaluation:

One can consider the top N results as predicted positive, any result lower than N as predicted negative. In this option one can use binary classification evaluation measure: precision, recall, f1-score would be the standard measures in this case.
One can consider the ratings/scores assigned to the full set of results. In this case there are two options:

if the numerical results are comparable, e.g. same kind of rating, then standard regression evaluation measures can be used, for instance RMSE.
if not, then it's still possible to compare the order of the results. Spearman rank correlation is a common evaluation measure in this case.

It seems that in your case you could use either the classification or the ranking evaluation measures. Of course, any of these evaluation methods requires gold standard results in order to compare the predictions against them.


Architecture help for multivariate input and output LSTM models

Basically every answer I was looking for was exampled and explained in this tutorial. Absolutely great resource for trying to understand how to model multi-output networks. This one goes through a lengthy walkthrough of a multi-output CNN architecture using Keras functional API. It only took me about three weeks to stumble upon, however.

https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses/


How to choose feedforward architecture for few number of features but very large instance?

Is your set of possible labels discrete or continuous?
If it's discrete and you have L possible different labels, then you should have L units in the output layer (then do a softmax if you want probabilities).
If it's continuous, then you should have 1 output unit.
Lastly, the number of units in the input layer is equal to the flattened dimension of one instance of the input. So if your data is 100 dimensions, then you must have 100 units in the input layer.


Clustering by Distributions of Groups of observations

You say "I don't want to calculate slopes or averages by groups and cluster because the distributions don't seem linear, or normal" but it looks to me like each group makes a nice compact cluster which would be represented well by the centroid of the group. Of course, you have 5400 groups and I am only looking at four of them in your sample. Maybe the other groups are not as nice. Nevertheless, let's take a look at the data that you provided. Here is your data in four of the seven dimensions. The group centroids are marked with an orange circle with a black edge.

It certainly seems to me like the centroids provide a good representation of the group position. Based on this,  I would recommend calculating the centroids by group and then clustering the centroids. (Of course,  you will need to normalize the variables first because they have such different scales.)

Code

Centroids = aggregate(df[,2:8], list(df$Grp), mean)

par(mfrow=c(1,2))
plot(df[,c(2,7)], pch=20, col=df$Grp)
legend("bottomright", legend=levels(df$Grp), pch=20, col=1:4)
points(Centroids[,c(2,7)], pch=21, bg="orange")

plot(df[,c(4,6)], pch=20, col=df$Grp)
legend("topleft", legend=levels(df$Grp), pch=20, col=1:4)
points(Centroids[,c(4,6)], pch=21, bg="orange")


Different number of features after using OneHotEncoder

One hot encoding is a way of converting output label for 3 categories like 2 into [0, 1, 0] or 3 into [0, 0, 1].

If you are using scikit learn to convert the value into one hot encoder then in training time you should use

enc = OneHotEncoder()
enc.fit(x_train)

If you are using scikit learn to convert the value into one hot encoder then in testing time you should use
enc.transform(x_test)

The reason we are using transform function in case of testing is we have to consider the label values on the basis of which we have converted data in training time. Because in testing time we not get all labels for that column


A few ideas to parse "events" from a text document

These documents can be very unstructured and machine learning rather takes perfectly structured data like a single clean table of numbers. The real hard part, how to translate all the interesting information (font size, bold, position, order, ...) into a clean table, isn't solved by machine learning. You see it's hard to create a sensible, purely numeric table from a document.

I think your best bet is to use a natural language processing toolkit and do named entity recognition. See http://www.nltk.org/book/ch07.html "5 Named entity recognition". Or see the recommendation here: https://stackoverflow.com/questions/11333903/nltk-named-entity-recognition-with-custom-data

With this auxiliary information, I'd rather spend time handcoding rules, which work on most documents. You can create a nested data structure and try to match patterns like "headline, enumitem1, enumitem2, ..."

The rules would throw errors on documents where they are unsure. For these documents you will have to manually check what's going on. But this way you don't lose information.

I'm no expert on the very advanced methods, but in theory, with a lot of labeled data, a very clever encoding schema and a fair bit of luck an approach like RNN might be successful. But even if all the mentioned requirements hold, the training of those RNNs require a lot of experience that is hard to learn from references.


Good approach to increase accuracy for a continuous value that is highly variable/sensitive to the inputs?

Taking the difference (ie speed1-speed2) as the target variable effectively dismisses any low-frequency variablitiy and targets only high-frequency variability, even noise.
One approach would be to bin the (highly-variable) target variable into fixed range bins and take the mid point (or any other fixed point) of each bin as the new target (stabilised) variable. This will decrease variability (and noise) a bit, or even a lot. Then any regression algorithm can be used on the new stabilised target variable with most probably better results (depending of course on the number of bins, more bins more variability, less bins less variability but there can be some balance point).


Why is Amazon P2 instance cost much more than G3?

p2.16xlarge has 16 Tesla K80 GPUs
g3.16xlarge has 4 Tesla M60 GPUs
p3.16xlarge has 8 Tesla V100 GPUs.

I believe the number is a reference to the vCPUs (they all have 64).  Comparing the 1 GPU instance of each type, p3 is the most expensive, then g3, then p2, as expected.


Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?

The first returns a probability density of the distributions. As you can see, they integrate to 1, i.e. they cover the same area (because they are probabilities, not the raw data).

The second returns actual frequencies, and that's why you have the actual scale of the data. Different histograms having different scales.


Saving images in a non-retraceable way, but still able to train R-CNN's on them

Wow. It is common for medical images to be heavily regulated (have to be kept on a separate server, cannot be copied, monitored access, etc...), but your situation is even tougher! Anyway, here's a few thoughts:

I don't think you can extract conv features, because a great deal of information (sometimes nearly everything) for reconstruction can be stored in those features (e.g., see Understanding Deep Image Representations by Inverting Them). Presumably this means you cannot simply train some kind of autoencoder and keep the latent representations either.
You could try to run the training in an "online" manner, meaning train on the new set of images each time they come. However, we must fight to avoid overfitting, so some heavy regularization (dropout, weight decay, early stopping, maybe even using a Bayesian neural network) and data augmentation probably would be needed. One methodology very relevant to this is work on catastrophic forgetting and continual learning. For instance, see Overcoming catastrophic forgetting in neural networks. You may be able to adapt these techniques to your case by treating each new set of data as a new task. One approach is to identify the important parameters each time you've trained on a new data subset, and prevent them from being changed to much when training on the next set.
A related approach is to use ensembles (similar to boosting). I imagine something like this (I'm sure you can think of something better): let $B_t$ be the $t$th data subset you receive.
Suppose you already have a set of models
$\mathcal{M}_{t-1} = M_1,\ldots,M_{t-1}$
with fixed weights from the previous time steps
(denote $\Theta_i$ as the parameters of $M_i$).
Let ${f}$ be a model that combines the outputs of $\mathcal{M}_{t-1}$ (i.e., if $M_i:\mathcal{I}\rightarrow{\mathcal{D}}$, then $f:2^\mathcal{D}\rightarrow\mathcal{S}$).
Then you train $f$ and $M_t$ on $B_t$, but your loss focuses on data examples where the outputs of $\mathcal{M}_{t-1}$ are poor.
The idea is to avoid overfitting by training $M_t$ to only care about learning things that the other older models don't already know.
This is a little tough in the detection case (I haven't thought about how to do it honestly), since the output is complicated (maybe an architecture like Deep Sets could help).
If it was simple regression ($\mathcal{S}=\mathbb{R}^n$), you could have $f(I)=\sigma_a(\sum_{\ell=1}^t c_\ell M_\ell(I))$, with a regularized loss function that enforces $c_u,\,u<t$ don't change too much, and $c_t$ isn't too large (note only all the $c_k$'s and $\Theta_t$ are trainable for $B_t$).
One other idea might be to train generative models (like GANs), and use them to generate novel data from the same distribution as these inputs. How to avoid forgetting for the critics is an issue of course.
Usually, you can generate a GAN sample $S$, find its nearest neighbour in the original set $N_S$, and (empirically) $S$ tends to be fairly far from $N_S$. But this may not be sufficient to pass the requirements in your case.
Finally, you may be able to use some work from the privacy and fairness machine learning literature. The idea is that you can destroy or ignore certain "undesirable" information in the model, data, and/or representations, in order to preserve privacy and/or enforce some notion of fairness in the model. In your case, you may be able to train some kind of "fair, minimal autoencoder" that can learn a representation of your data that does not contain problematic properties, yet is still useful for training. For instance, if the images are sensitive because they contain information about some particular property of a given person, you may be able to learn a representation that does not hold information about that property (and then save and keep those). It really depends on the specifics of your situation of course.
I am not an expert in this area, unfortunately, but a place to start might be Learning Adversarially Fair and Transferable Representations.


In normal image GANs, the generator receives random noise and generates an image. As you can see, there is no "visual relationship" between the input and output.
In cycle GANs, generators receive an image and generate another image. The idea is that the input image is of domain A and the output image is a "translation" of the input image into domain B. However, with the experience with normal GANs, why would the "translated image" have any visual relationship with the original one? If the cycle consistency loss were not there, a generator could simply generate images in the target domain that were totally unrelated to the input and the discriminator would be Ok with it.
The role of the cycle consistency loss is to ensure that the generated output image is actually a version of the input image where the domain is what changes, but the "contents" are kept.


Why Heaps' Law Equation looks so different in this NLP course?

The plot shows Heaps' Law but the formula is something different, it is Zipf's Law.

$f(w)$ is the relative frequency (or probability) of word $w$. That is, given a random word, it will be $w$ with probability $f(w)$. Therefore, if a document has $n$ words, it has on average $n\times f(w)$ occurrences of word $w$.

The formula can be re-written as follows:
$$f(w)=C(r(w)-b)^{-\alpha}$$
which is a power-law distribution that shows Zipf's Law, however with a slightly different parameterization by introducing cut-off $b$.

$r(w)$ denotes the rank of word $w$. For example, if we sort all the words in a  news corpus based on their frequency, $r(\text{'the'})$ would be 1, $r(\text{'be'})$ would be 2, and so on,
Cut-off $b$ ignores highly frequent words $r(w) \le b$, effectively shifting up the rank of remaining words,
$C$ is the normalizing constant, i.e. $C=\sum_{r=\left \lfloor  b \right \rfloor + 1}^{\infty}(r-b)^{-\alpha}$, which gives $\sum_{w,r(w)>b} f(w) = 1$, and
Exponent $\alpha$ denotes the rate of drop in probability when rank increases. Higher $\alpha$, faster drop.

Exponent $\alpha$ is determined by fitting the formula to some corpus, as shown in the table. Generally, lower $\alpha$ (in the case of twitter), thus slower drop,  means corpus has more word diversity.


Number of internal nodes in a Decision Tree

Building a decision tree is a process where the algorithm picks the first feature to split on $i_1$ from the set of features $n$ that it can split on $i_1={1,...,n}$.

After splitting, the algorithm has to decide whether to stop building the tree or split on another feature $i_2={1,...,n} \ $ AND $i_2 \ne i_1$, and the process repeats where the algorithm keeps deciding whether to stop or split on a new feature $i_3={1,...,n} \ $ AND $i_3 \ne i_2 \ne i_1$. This keeps going until the algorithm decided to stop building the tree or it has split on all $n$ features and there exists no feature $i_{n+1}={1,...,n}$ that can satisfy the condition $i_{n+1} \ne i_n \ne ... \ne i_2 \ne i_1$. Thus there are no more features to split on.

Thus the maximum number of nodes has to be less than or equal to the size ($n$) of the input feature.


What is the complexity of Mixture Hidden Markov Models (MHMM)?

Mixture hidden markov model is not one algorithm. It's made of forward algo, viterbi algo and forward-backward algo.

The time complexity of computing the forward variable used in hidden markov models is O(N2T) In which N is the number of states and T is the length of the observations sequence.


K-Means Clustering for data points with multiple attributes

In k-means clustering, the "k" defines the amount of clusters - thus classes, you are trying to define.

You should ask yourself: how many different groups (=clusters) of recipes am I looking for?

In your case, your data points (features) (=recipes), are of variable dimensions (attributes) (avg 8 dimensions).

What you must realize is that the dimension of each datapoint must be the of the same length. (So the amount of recipes for each ingredient should be the same).
This is because k-means calculates the distance of each datapoint to each centroid (k centroids).

What you can do to create datapoints of the same dimension, you can introduce "DTW".
More about this here:

https://stats.stackexchange.com/questions/26100/how-to-run-k-means-clustering-on-data-points-of-varying-dimensionality

An easier solution would be that you populate your dataset with a value for each ingredient present in the dataset, and set the value to "0" when the ingredient is absent.


Can we do multivariate time series analysis using holt-winter ( Exponential smoothing) method?

Exponential smoothing is usually defined on univariate data with no exogenous variables (see e.g. this book).

However, Rob Hyndman discusses different approaches of including exogenous variables in this blog post and gives some more references.


What model can I build with a limited dataset?

Based on your dataset you can also try to create a product recommendation model so that given a new customer that just purchased a particular item you can predict/suggest or recommend what other items they could purchase next during their shopping trip or after:
- Collaborative filtering can be used to achieve this
- Frequent pattern mining can also be used.


It is common for TFIDF to be a strong model. People constantly get high places in Kaggle competitions with TFIDF models. Here is a link to the winning solution that used TFIDF as one of its features (1st place Otto product classification). You will most likely get a stronger model if you combine the TFIDF and RNN into one ensemble. Other results from Kaggle:

2nd place: https://www.kaggle.com/c/stumbleupon/discussion/6184
4th place: https://www.kaggle.com/c/avito-demand-prediction/discussion/59881
3rd place: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52762

https://www.kaggle.com/c/avito-demand-prediction/discussion/56897:

  A good number of kernels are going the traditional route with
  CountVectorizer/TF-IDF, and some brave souls (I say brave because
  training is slower and the results don't seem as spectacular so far)
  have been experimenting with embeddings, as per the previous
  competitions.


How to use features when predicting aggregation?

You can pass the entire month's data as a single sample, which should predict the sales of that month. It would look like an index with multiple indexes. Something like MultiIndex in python's Pandas library.

Another idea would be to still try predicting the sales every day, then just sum up all predictions at the end to create your monthly aggregation.

Using something like a neural network may be overkill - it depends how much data you have - but will offer the flexibility to easily train a model with multiple inputs and a single output, as is your case.


I tried to estimate a few of your data values from the scatter plot you provided.

Then I performed a power model regression and came up with

$y = (5.777 \cdot 10^{-16}) \cdot 1.404^{x}.$

The estimated values I used are below.

$(70, 0.01), (75, 0.012), (80, 0.015), (90, 0.025), (95, 0.075), (98, 0.15), (99, 0.20), (99.5, 0.25), (99.9, 0.32)$

Of course, your actual model will differ because you have the actual data set. I just eyeballed a few points so I could test the power fit.


The definition of an outlier per si is already quite dubious, but we can define them as those values that surpass the limit of +- 1.5 * IQR. In this case, either the standard deviation method or the Tukey method are valid options. We just need to try and see which gives better results.

# Tukey Method

n = 2 #In this case, we considered outliers as rows that have at least two outlied numerical values. The optimal value for this parameter can be later determined through the cross-validation
indexes = []

for col in df.columns[0:14]:
    Q1 = np.percentile(df[col], 25)
    Q3 = np.percentile(df[col],75)
    IQR = Q3 - Q1

    limit = 1.5 * IQR

    list_outliers = df[(df[col] < Q1 - limit) | (df[col] > Q3 + limit )].index # Determine a list of indices of outliers for feature col

    indexes.extend(list_outliers) # append the found outlier indices for col to the list of outlier indices

indexes = Counter(indexes)
multiple_outliers = list( k for k, v in indexes.items() if v > n )

Once you detect the outliers, you can either remove them or replace them with max/min limit values. In the first case, you just need to do this:

df.drop(multiple_outliers, axis = 0)

df = df.drop(multiple_outliers, axis = 0).reset_index(drop=True)

But for the second case, you should do this:

#Setting the min/max to outliers using standard deviation
for col in df.columns[0:14]:
    factor = 3 #The optimal value for this parameter can be later determined though the cross-validation
    upper_lim = df[col].mean () + df[col].std () * factor
    lower_lim = df[col].mean () - df[col].std () * factor

    df = df[(df[col] < upper_lim) & (df[col] > lower_lim)]

Finally, you can also use Isolation Forest or LocalOutlierFactor (more appropriate for Anomaly/Fraud Detection Problems).


Hyperparameter tuning does not improve accuracy?

Welcome! You haven't given us enough information to be able to diagnose this issue completely, but you should check your grid search code to see how each cross-validated model is being trained and note which parameters are different from those used with the 92% model.

If it has something to do with the unbalanced data, it's because you're not stratifying your sampling of the dataset with respect to the labels (making sure training and validation sets have the same ratio of classes (in this case, 38% survive, 62% don't survive.))

I would guess that something is going on with your cross validation process.  If I had to guess specifics (and again, we can't say for sure given what you've posted here) , I would say that something you're doing in the CV probably results in those models not using as much training data as the 92% model.


Evaluate imbalanced classification model on balanced testing sample

The test set should represent what your model will encounter in practice when you use it to make predictions. It serves as a resource to demonstrate that what you did will actually work.

In case your test set does not represent what the true problem is or will be in the future, you can‘t say much on how well your model works.

If you test on a balanced set while you have trained on a unbalanced set (which is harder to tackle), your test predictions will likely look better or worse than what you could reasonably expect with the „real“ unbalanced problem. So you cannot say anything with this test about the quality of your model.


Find the number of homes with 3000 sq ft , 3 number of bedrooms and 40 years of age?

If it's "find the house" then that implies that you need to find the house with those specific attributes in the dataset. I'm not sure what programming language you're using but it should be pretty simple to do such a thing anyways.
If it's "find the price of a house with..." that implies there probably isn't such a house in the dataset and you need to train a predictive model to predict the price of such a house. You said to use linear regression, which is a fine choice, if your data is linear... But I would instead suggest an SVR (Support Vector Machine Regressor), and fine-tune some things like the kernel and regularization.


Triplet loss function for face recognition?

$\alpha$ is known as the margin.

Not only that we want to minimize $D(A,P)$ and maximize $D(A,N)$, that is we want  $D(A,P)-D(A,N)$ to be small. Not only that we want it to be non-positive, we want it to be sufficiently negative.

That is not only that we want  $$D(A,P)-D(A,N) \le 0$$

We want $$D(A,P)-D(A,N) \le -\alpha.$$


adding summary to tensorflow model error

Check out dimension

 X = data[0]
 Y = data[1]

Its likely that have dimensions like (1, ) what means that's 1 dim vector.

That should be work

import numpy as np
X = np.array(X).reshape(-1,1)
Y = np.array(Y).reshape(-1,1)


What can help decrease outliers' influence on non-tree models?

Here are couple of other options:

Set a threshold and remove all values larger than the threshold

RobustScaler which removes the median and scales the data according to the quantile range.

QuantileTransformer which transforms the feature to follow a uniform or a normal distribution.


Does torch.cat work with backpropagation?

Yes, torch.cat works with backward operation.


Logistic Regression is a large margin loss not classifier. (in literal sense, meaning there is a margin with the loss function not the classification boundary

There are two terms in the softmax loss, true values minus the exp or log of the observations (roughly and depending on the variant of loss) this just implies that distance of observed example from its true decision boundary needs to beat the log sum of the distances from all of the decision boundaries.

Because the softmax function is a probability distribution, the largest the log softmax can be is 0, so the log softmax returns a negative value (i.e. a penalty) that approaches 0 as the probability of the true class under the softmax function approaches 1.


When is Non-Stochastic Global Optimization Preferable or Necessary?

Non-stochastic global optimization could be practical for the directed search of the solution space. If there is a priori knowledge, that knowledge can direct the optimization to specific regions. Stochastic global optimization might explore inefficiently explore regions of the solution space.
A specific example of non-stochastic global optimization is Bayesian optimization. In Bayesian optimization, samples are not picked at random. Samples are picked based the calculated posterior distribution, often to maximize the expected improvement.
Bayesian optimization has been used in many applications, including machine learning hyperparameter tuning.


Pandas throwing "Error tokenizing data. C error" while loading data sets from URL

The path that you are accessing from is a Github repository page which is a webpage, it does not return CSV. You have to click on 'raw' option in Github and then pass the URL which in your case is:

test = pd.read_csv('https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/test.csv')


Multi task learning architecture for Multi-label classification

You should design a multi-task model (MTM).
MTM has the ability to share learned representations from input between several tasks. More precisely, we try to simultaneously optimize a model with m types of loss function, one for each task. Consequently, MTM will learn more generic features, which should be used for several tasks, at its earlier layers. Then, subsequent layers, which become progressively more specific to the details of the desired task, can be divided into multiple branches, each for a specific task.

You need an architecture like the following:

# Your input and hidden layers
inp = Input(...)
x = Layer1(...)(inp)
x = Layer2(...)(x)
...
x = Layer_N-1(...)(x)

# Your 'm' output layers
out_1 = Layer_N(x)
out_2 = Layer_N(x)
...
out_m = Layer_N(x)

# your model
MTM = Model(inputs=inp, outputs=[out_1,out_2,...,out_m])
MTM.compile(loss=[loss_func_1, loss_func_2,..., loss_func_m],
      optimizer='xxx',
      metrics=xxx)

MTM.fit(train_data, [train_labels_1,train_labels_2,...,train_labels_m],
          batch_size = xxx,
          epochs = xxx)

If you want a real example in Keras, I have implemented a 2-task ConvNet for my project here.


Ideally, the threshold should be selected on your training set. Your holdout set is just there to double confirm that whatever has worked on your training set will generalize to images outside of the training set.
This is the reason why hyperparameters tuning like GridSearch and RandomizedSearch in python has a cv parameter to cross-validate between different folds of your training set instead of allowing to choose the best parameters based on metric measured using the holdout set.


hypeparameters tuning neural network according to loss vs according to scoring function

If you look for example at the L2 loss function:

$$
\sum (y - \hat{y})²
$$

It takes into account how much the predicated value differs from the actual value. Note that because of the squaring of errors, L2 is sensitive to outliers. So, as you can see it takes into account how far our predictions are.

And that is exactly the problem: Most of the time we are not interested in how far our predictions are from the ground truth.

Imagine a real world application where the model predicts an image label. The application won't give you n examples together with their probabilities. It will just output the image with the highest probability.

That means, in the case of a classification problem where we use a softmax together with cross-entropy-loss it is not important if the predicted probability for a class is 0.49 or 0.01. We only care about if our model got the class correct or wrong.

Also, accuracy is much more tangible and better to interpret. That's why accuracy is used much more often.


Classification - get some label value to check how close to another class (Python)

The problem is that Naive Bayes is a classification algorithm that outputs the probability of an input belonging to a class. So you cannot use Naive Bayes to achieve what you want. To get 'in-between' values you need to use a regression algorithm instead. I'm not sure what library you're using, but try to use regression versions Knn and SVM models instead.


Instead of binarising discrete variables for NN algorithms, could I attempt to assign them a value by splitting into buckets and sorting?

Just taking the mean seems better, that way the distances between the groups is better represented than just indexing them by order. With the target you should be a bit careful not to include the current row for the mean, since for the test set and the actual predictions you cannot use that target either (for obvious reasons). To get even more information about the category you try to get rid of is to also take the variance or standard deviation of the target within this group and take the mean and variance of important numerical features for these categories. If you have multiple categories you can also take the expected value for features and targets conditioned on multiple categories at once. This is a nice compromise between keeping enough information on high cardinality while keeping the dimensionality low.

Here's an example where the c is the categorical variable, n are numeric and y is your target. We want to get rid of c (because c could have 1000s of values).

xc_1 | xn_1 | xn_2 | y
A    | 2    | 4    | 1
A    | 1    | 2    | 1
A    | 4    | 3    | 0
B    | 3    | 5    | 0
B    | 5    | 4    | 1
B    | 6    | 6    | 0

To keep things a little more simple we will disregard the values of the current row, which is better anyway. For the first row we will look at all other A rows, take the averages (and maybe other statistics) of the numeric features and of y, we will do the same for B, then we get:

xc_1 | xn_1 | xn_2 | xnc_1 | xnc_2 | yc_1 | y
A    | 2    | 4    | 2.5   | 2.5   | 0.5  | 1
A    | 1    | 2    | 3     | 3.5   | 0.5  | 1
A    | 4    | 3    | 1.5   | 3     | 1    | 0
B    | 3    | 5    | 5.5   | 5     | 0.5  | 0
B    | 5    | 4    | 4.5   | 5.5   | 0    | 1
B    | 6    | 6    | 4     | 4.5   | 0.5  | 0

We have captured a lot of information about category 1, which we can now discard.


HDBSCAN cluster: still unclear to me how to chose 'min_cluster_size`

Optimal in which sense?

The crucial thing with clustering is that there is no optimal solution. Different solutions tell you a different part of the story. And to be able to get different views, you will need parameters. It is a exploratory technique.

Various attempts at defining "optimal" solutions have failed for practical use, just think of k-means.


Search for keywords within csv/word files which are listed in an excel sheet

Try something like the following in python:

import pandas as pd

df = pd.read_csv(path_to_file, header=0) # header=0 means the first row contains column names

# locate the records where none of the following columns are null ('~' means not)
mask = ~df[['Project Name', 'County', 'State', 'Country']].isnull()
df[ mask ]

I'm not entirely sure what you want to do with the 'Order' columns, but if it's to get just the "Order" column from the above statement, then just slightly modify it to the following:

df['Order'][mask]

If you want to set the value of "Order" for those columns that aren't null, then just do this:

# we need to adjust our mask because the current version won't work with the .loc function that we'll use below

mask = df[(~df['PROJECTNAME'].isnull())&(~df['County'].isnull())&(~df['State'].isnull())&(~df['Country'].isnull())]

df.loc[ mask, "Order"] = whatever you want

If you want to save all the orders where the specified columns aren't null to a separate csv, do this:

df[ mask ].to_csv(name_your_file)


Platt's Scaling for multi-label classification?

There are a few multiclass variants of Platt scaling. The easiest approach is as you have described; simply perform one Platt scaling on each class.

However, there are more sophisticated options--a very simple one to implement is training a standard logistic regression on the logits (the values before the softmax activation is applied). This has called matrix scaling and can overfit pretty easily, so only use this if you have a large calibration set. Alternatively, a fewer-parameter version called vector scaling is relatively simple to implement, where the weights matrix inside the logistic regression is restricted to be a diagonal matrix. Finally, a very simple option that has been shown to work well for neural networks is temperature scaling, where all logits are simply scaled by a single scalar parameter.

You can read more about these and their application to neural networks in Section 4.2 of "On Calibration of Modern Neural Networks" (2017) - available here


Creating a neural network, composed of n times a different network. Is it possible?

I think it's possible. I don't know how well it will work out for you, but I think it's technically possible (which is the best kind of possible).

For starters, I would suggest using the functional API for Keras for this kind of thing. It makes it super easy to do composition of networks like this, and residual networks like this implementation on GitHub are composed in a similar fashion. Something like the following might get you going (note this isn't meant to be a full solution, sorry):

from keras.layers import Input, Dense
from keras.models import Model

def single_net(input_shape, filters):
    def f(input):
        h0 = Dense(filters, activation='relu')(input)
        h1 = Dense(filters, activation='relu')(h0)
        prediction = Dense(2, activation='softmax')(h1)
        return prediction
    return f

input_a = Input(shape=input_shape)  # whatever your input shape is
input_b = Input(shape=input_shape)
# ... repeat for however many people you have ...
shared_net = single_net(input_shape, filters)  # define once
network0 = shared_net(input_a)
network1 = shared_net(input_b)
# ... again repeat for each person
combined_nets = keras.layers.concatenate([network0, network1, ...])
dense = Dense(output_shape, activation='relu')(combined_nets)  # or however you want to combine them
model = Model(inputs=[input_a, input_b, ...], outputs=dense)
model.compile(optimizer='adam', loss='binary_crossentropy')  # whatever optimizer, loss you like
model.fit([data_a, data_b, ...], labels, ...)

By defining your repeated network only once (shared_net above) and then using that same object for each input, you are sharing those layers across all of those inputs. Basically when you back-propagate, all inputs/outputs will train a single instance of the network, even though you're basically using duplicates of it for each input.

The big question is how do you combine all of your individual models, and how do you deal with changing numbers of models. I don't know how to deal with an uncertain number of models, but I suggest looking at the shared layers section of the Keras functional API documentation for some help on combining, and dig in to some examples on GitHub. What's shown here ought to get you started, but there are probably some errors hiding throughout. Like I said, not a complete answer, sorry.


Cash-flow prediction with machine learning

There are number of different time series forecasting algorithems other than LSTM's and other types of RNN's. In general the other methods that use statistical approaches take less time to run and are easier to understand.

From my experience some of the most important things is the data preprocessing rather than the algorithms (eg finding the seasonal period, finding how many seasonal periods there are identifying the trend to make a stationary series, removing outliers).
Once you have done this then you can evaluate different models with a test train split to see how they perform and choose the best model rather than starting off by choosing an algorithm.

You will probably find some interesting things will trying to account for the monthly seasonality due to the lengths of the months changing over time. One way to adjust for that is to use a dummy variable for the start of the month.

Depending on the problem there are a number of options.
Forecasting using statistical methods:

Naive (uses the mean as the prediction)
Seasonal Naive
Arima (a Class of algorithms)
Tbats

If there is a feature that is known in the forecasting window:

Regressing

If there are multiple Features:

Vector auto regression

I found this site very useful for learning what can be done with time series forecasting. (Its the html version of Forecasting: Principles and Practice Rob J Hyndman and George Athanasopoulos)


Why there is Different accuracy for two same trained model?

Somewhere in the code there should be some parameter that is initialised randomly, this is usually called the random seed. It could be the different initialisation of your neural network weights that is affecting the results, or maybe the k-fold being different if done at random.
The extent of the difference between the performance suggests that you might not have enough data to reliably learn a good model, or your model is underspecified and gets stuck at different local minima. Consider, changing the learning rate and other hyperparameters and see if the effect stays.