How to handle sparsely coded features in a dataframe

The first question you have to answer is whether these are actually missing or simply sparsely coded!
E.g. if the variables are "supposed" to only show doctors recommended diet, insulin or lifestyle change then we could naturally conclude that any NaN is actually a "No" in which case you do not even have to impute the data but rather straight-up replace with "yes = 1 and NaN = 0.
How do you differentiate this case (sparsely coded) from the other case: actual missing values? Besides applying your domain knowledge you should also test whether the following rules apply:

If diabetes == No then all treatment variables are missing
If diabtes == Yes then at least one treatment variable is "Yes"
Treatment variables are always only "Yes" or NaN

If this is the case your dataset is likely sparsely coded, otherwise you have actual missing values.
Take note however that should the values be actual missing values (e.g. because you identify some cases where non-diabetics have a treatment or some cases are labeled "No" in the treatment variables) you can assume they are MAR but not MNAR. In this case I would recommend to remove these variables because imputation as "No" or o does not make sense if you do not have any information what the NaN actually means.


It looks to me like what you propose makes sense, but there has been some research done around these questions of time representation already. I'd suggest you check the state of the art in this domain, if only not to reinvent the wheel or miss important cases.
I'm not very knowledgeable about it but I can at least point you to TimeML and the related publications. There are certainly other recent works building on TimeML, for example this one (disclaimer: I know the author).


Configure BRAT so that if no `.ann` file is found, then an empty `.ann` file is created

I run into the same problem with BRAT from time to time. The way I solved it for a folder with .txt files is to run this Bash one-liner directly in the shell when inside the directory with these files:

for f in *.txt; do touch $(basename $f .txt)".ann"; done

What is does is the following: With a for-loop, it loops over every .txt file, and creates an empty file with touch having the same basename, but a .ann extension instead of a .txt extension.


Create a new column based on two columns from two different dataframes

You were almost there!

Sample DFs:

In [387]: df1
Out[387]:
   id
0   1
1   2
2   3
3   4
4   5

In [388]: df2
Out[388]:
   first_id
0         7
1         6
2         5
3         1
4         3

Solution:

In [389]: df2['new'] = df2.first_id.isin(df1.id).astype(np.int8)

Result:

In [390]: df2
Out[390]:
   first_id  new
0         7    0
1         6    0
2         5    1
3         1    1
4         3    1


Simple 3 layers Neural Network cannot be trained

The results you are getting are the following:

[[1.85572928e-04]  =  0.000185572928         ~ 0
 [9.99755942e-01]  =  0.999755942            ~ 1
 [5.21248255e-09]  =  0.000000000521248255   ~ 0
 [9.99767481e-01]  =  0.999767481            ~ 1
 [9.99963580e-01]  =  0.999963580            ~ 1
 [2.07334909e-04]] =  0.000207334909         ~ 0

These are indeed very close to your expected results. You are computing and predicting floating point numbers, and not binary zeros and ones. You could for example add a simply rule that will accepts values below a threshold to be zero and above the threshold to be one.


Which value should I take into account for saying that my model has accuracy of ...?

None. Accuracy is practically meaningless in such class imbalanced settings; the metrics of interest here are precision, recall, and f1 score.
Now, it's true that the values of these metrics also fluctuate between runs, much similarly with the reported accuracy. But this is to be expected due to the small-sample effects - your sample is so small that even a difference in the classification of a couple of samples in your (even smaller) validation set is enough to give the observed discrepancies.

Does it make sense to run a model where there are so a few values = 1 for the dependent variable?

Indeed it does; there are plenty of applications where the positive values are a very small percentage of the whole dataset (thing of sick people versus the general population of healthy ones, or engine faults versus long running times where no fault is present). That's why this is a (large enough) sub-topic of machine learning called class imbalance or imbalanced classification, with its own specific approaches. I suggest you start googling ruthlessly...


Opensource tools for help in mining stream of leader board scores

This isn't a full solution, but you may want to look into OrientDB as part of your stack. Orient is a Graph-Document database server written entirely in Java.

In graph databases, relationships are considered first class citizens and therefore traversing those relationships can be done pretty quickly. Orient is also a document database which would allow you the kind of schema-free architecture it sounds like you would need. The real reason I suggest Orient, however, is because of its extensiblity. It supports streaming via sockets, and the entire database can be embedded into another application. Finally, it can be scaled efficiently and/or can work entirely through memory. So, with some Java expertise, you can actually run your preset queries against the database in memory.

We are doing something similar. In creating an app/site for social science research collaboration, we found ourselves with immensely complex data models. We ended up writing several of the queries using the Gremlin Traversal Language (a subset of Groovy, which is, of course, Java at its heart), and then exposing those queries through the binary connection server of the OrientDB. So, the client opens a TCP socket, sends a short binary message, and the query is executing in Java directly against the in-memory database.

OrientDB also supports writing function queries in Javascript, and you can use Node.js to interact directly with an Orient instance.

For something of this size, I would want to use Orient in conjunction with Hadoop or something like that. You can also use Orient in conjunction with esper.

Consider:
An introduction to orient: http://www.sitepoint.com/a-look-at-orientdb-the-graph-document-nosql/

Complex, real-time queries: http://www.gft-blog.com/business-trends/leveraging-real-time-scoring-through-bigdata-to-detect-insurance-fraud/

A discussion about streaming options with java and orient: https://github.com/orientechnologies/orientdb/issues/1227


Yes, its compressed, read more for advanteges:

Parquet, is an open source file format for Hadoop, it stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in row-oriented approach, parquet is more efficient in terms of storage and performance. Parquet stores binary data in a column-oriented way, where the values of each column are organized so that they are all adjacent, enabling better compression. It is especially good for queries which read particular columns from a “wide” (with many columns) table since only needed columns are read and IO is minimized.

When we are processing Big data, cost required to store such data is more (Hadoop stores data redundantly I.e 3 copies of each file to achieve fault tolerance) along with the storage cost processing the data comes with CPU,Network IO, etc costs. As the data increases cost for processing and storage increases. Parquet is the choice of Big data as it serves both needs, efficient and performance in both storage and processing.

To conclude main advantages of parquet:

Organizing by column allows for better compression, as data is more homogeneous. The space savings are very noticeable at the scale of a Hadoop cluster.
I/O will be reduced as we can efficiently scan only a subset of the columns while reading the data. Better compression also reduces the bandwidth required to read the input.
As we store data of the same type in each column, we can use encoding better suited to the modern processors’ pipeline by making instruction branching more predictable.


Ridge regression model creation using grid-search and cross validation

Is 0.9113458623386644 my ridge regression accuracy(R squred) ? if it is, then what is meaning of 0.909695864130532 value.

These are both R^2 values.
The first score is the cross-validation score on the training set, and the second is your test set score.   The first is perhaps a little biased, since those models are built using hyperparameters selected while using that dataset (but a different cv-split I think).  The second score should be unbiased, and should probably be your reported test score.  (The second score is a bit better than what should be a optimistically-biased score; perhaps just randomly, perhaps because of the train/test split, and perhaps because that model has seen more data.)

I want check whether my regression model building steps correct or not? can some one explain it?

Things look OK to me.
You could save yourself some code and training time; by default GridSearchCV refits a model on the entire training set using the identified hyperparameters, so you don't need to fit in the last code block.  It also has the cv_results_ and best_score_ attributes to provide you with cross-validation scores, but since you've used the them in selecting the optimal hyperparameters, the best score is no longer an unbiased estimator for future performance.
Note too that there's a builtin for tuning the regularization parameter, but it uses a different CV approach and has RMSE as default scorer.


The best emulation (although I didn't check full equivalency):

import mxnet as mx

SEQUENCE_LENGTH = 100

stack = mx.gluon.rnn.SequentialRNNCell()
stack.add(mx.gluon.rnn.LSTMCell(16, prefix='first')
stack.add(mx.gluon.rnn.LSTMCell(1, prefix='first')

x, _ = stack.unroll(length=SEQUENCE_LENGTH,
    inputs=x,
    layout='NTC',
    merge_outputs=True,
)

Here layout='NTC' is for the case when input shape is (samples, timesteps, channels).


If you look at the wikipedia page of Wilcoxon signed-rank test, under the section of the original test, it's mentioned that

  The original Wilcoxon's proposal used a different statistic. Denoted by Siegel as the $T$ statistic, it is the smaller of the two sums of ranks of given sign; in the example given below, therefore, $T$ would equal $3+4+5+6=18$


R random forest on Amazon ec2 Error: cannot allocate vector of size 5.4 Gb

Here's some advice (use at your own risk!):

make sure that your R environment on EC2 is identical to the one on your laptop
make sure you're using 64-bit images for your virtual machine instance
try to create/enable swap space: https://stackoverflow.com/a/22247782/2872891
see this discussion: https://stackoverflow.com/q/5171593/2872891
see this discussion: https://stackoverflow.com/q/12322959/2872891
see this discussion: https://stackoverflow.com/q/1358003/2872891

If the above-mentioned simpler measures don't help OR you want to achieve more scalability and/or performance, including an ability to parallelize the process on a single machine or across multiple machines, consider using bigrf R package: http://cran.r-project.org/web/packages/bigrf. Also see this discussion: https://stackoverflow.com/q/1358003/2872891.


How to improve an existing (trained) classifier?

In order to improve your classifier, you have few options.

Ensembling - make a group of classifier and let them predict together. Stacking, blending, bagging, boosting. Choice is yours.
Hyper parameter tunning - you did not mention your tool, but i suppose that in every solid one is option to search in parameter space to find the best combination
Sampling - you can try undersample or create new samples (SMOTE) to give your classifier more data of class which you'd like to predict
Feature engineering - get rig of noisy features and work only with those which have an effect on your prediction, consider PCA too
Scaling - normalize your data can improve the performance, some classifiers require this
Data quality - missing values, coding categorical variables or suspicious values can influence performance


How to create dictionary with multiple keys from dataframe in python?

In order to be able to create a dictionary from your dataframe, such that the keys are tuples of combinations (according to your example output), my idea would be to use a Pandas MultiIndex. This will then generate a dictionary of the form you want.

First I just recreate your example dataframe (would be nice if you provide this code in the future!):

import pandas as pd

# Create the example dataframe
df = pd.DataFrame(["4-Grain Flakes", "4-Grain Flakes, Gluten Free", "4-Grain Flakes, Riihikosken Vehnämylly"])
df["id"] = [11005, 35146, 32570]
df["energy"] = [1404, 1569, 1443]
df["fibre"] = [11.5, 6.1, 11.2]
df.columns = ["name"] + list(df.columns[1:])

print(df)
                                     name     id  energy  fibre
0                          4-Grain Flakes  11005    1404   11.5
1             4-Grain Flakes, Gluten Free  35146    1569    6.1
2  4-Grain Flakes, Riihikosken Vehnämylly  32570    1443   11.2

Now we can create the combinations of each value in "name" with each of the other column names. I will use lists, within a list comprehension, where I bundle up the values together into tuples. We end with a list of tuples:

names = df.name.tolist()
others = list(df.columns)
others.remove("name")         # We don't want "name" to be included

index_tuples = [(name, other) for name in names for other in others]

We can create the MultiIndex from this list of tuples as follows:

multi_ix = pd.MultiIndex.from_tuples(index_tuples)

Now we can create a new dataframe using out multi_ix. To populate this dataframe, notice that we simple need to row-wise values from columns ["id", "energy", "fibre"]. We can do this easily by extracting as an n * 3 NumPy array (using the values attribute of the dataframe) and then flattening the matrix, using NumPy's ravel method:

df1 = pd.DataFrame(df[others].values.ravel(), index=multi_ix, columns=["data"])

print(df1)

                                                  data
4-Grain Flakes                         id      11005.0
                                       energy   1404.0
                                       fibre      11.5
4-Grain Flakes, Gluten Free            id      35146.0
                                       energy   1569.0
                                       fibre       6.1
4-Grain Flakes, Riihikosken Vehnämylly id      32570.0
                                       energy   1443.0
                                       fibre      11.2

Now we can simply use to to_dict() method of the datframe to create the dictionary you are looking for:

nutritionValues = df1.to_dict()["data"]

print(nutritionValues)

{('4-Grain Flakes', 'energy'): 1404.0,
 ('4-Grain Flakes', 'fibre'): 11.5,
 ('4-Grain Flakes', 'id'): 11005.0,
 ('4-Grain Flakes, Gluten Free', 'energy'): 1569.0,
 ('4-Grain Flakes, Gluten Free', 'fibre'): 6.1,
 ('4-Grain Flakes, Gluten Free', 'id'): 35146.0,
 ('4-Grain Flakes, Riihikosken Vehnämylly', 'energy'): 1443.0,
 ('4-Grain Flakes, Riihikosken Vehnämylly', 'fibre'): 11.2,
 ('4-Grain Flakes, Riihikosken Vehnämylly', 'id'): 32570.0}

It is also possible to get your final example of a multidict, directly from the multi-indexed dataframe. You need to just use multi-index slicing:

fibre_df = final_df.loc[(slice(None), ["fibre"]), :]
print(fibre_df)

                                                 0
4-Grain Flakes                         fibre  11.5
4-Grain Flakes, Gluten Free            fibre   6.1
4-Grain Flakes, Riihikosken Vehnämylly fibre  11.2

You can then generate a dictionary as before:

d = final_df.loc[(slice(None), ["fibre"]), :].to_dict()[0]
print(d)

{('4-Grain Flakes', 'fibre'): 11.5,
 ('4-Grain Flakes, Gluten Free', 'fibre'): 6.1,
 ('4-Grain Flakes, Riihikosken Vehnämylly', 'fibre'): 11.2}

And you can drop the "fibre" value from the tuple-keys with a simple dictionary comprehension:

final_dict = {k[0]: v for k, v in d.items()}
print(final_dict)

{'4-Grain Flakes': 11.5,
 '4-Grain Flakes, Gluten Free': 6.1,
 '4-Grain Flakes, Riihikosken Vehnämylly': 11.2}


How to score arbitrary sentences based on how likely they are to occur in the real world?

Typically you would use a perplexity value. For example, if your LSTM model is word-based and you have a sentence $[x_1, x_2 . . . x_N]$, and your model predicts the words that appear in that sentence with probability $p(x_i|x_0..x_{i-1})$ (where $x_0$ is a "start token" or whatever you use to start your RNN prediction sequence). Then you might quote a per-word perplexity for that sentence under your model as

$- \frac{1}{N}\sum_{i=1}^N log_2(p(x_i|x_0..x_{i-1}))$

Using an LSTM to predict consecutive words, it is practical to construct an array of probabilities $[p_1, p_2 . . . p_N]$ by running the network on the sentence and noting the probabilities for the correct matching word - i.e. $p_i = $ the predicted probability of the correct class $x_i$ at each step, which simplifies the expression:

$- \frac{1}{N}\sum_{i=1}^N log_2(p_i)$


Why Davies-Bould chose a number ob cluster higher than Silhouette or Calinsky Harabasz?

Each clustering evaluation metric follow different ideologies:
Silhouette analysis can be used to study the separation distance between the resulting clusters.Silhouette coefficients near +1 indicate that the sample is far away from the neighboring clusters. A value of 0 indicates that the sample is on or very close to the decision boundary between two neighboring clusters and negative values indicate that those samples might have been assigned to the wrong cluster.
Calinski–Harabasz rewards clusterings in which the cluster centroids are far apart and the cluster members are close to their respective centroids.
Davies-Bouldin is defined as the average similarity measure of each cluster with its most similar cluster, where similarity is the ratio of within-cluster distances to between-cluster distances. Thus, clusters which are farther apart and less dispersed will result in a better score. It prefers clusters equally-distanced from each other.
We can't really say what clustering quality measure is good or not.It depends on what you want to evaluate.You have to look it seems relevant for the kind of clustering you are doing.
For more info refer this discussion.


Multiply Tensorflow sequential layer by fixed integers

This turned out to be really simple.  I just used a Lambda layer to multiply the previous output layer by the multiplication vector.

model.add(tf.keras.layers.Dense(3)) # previous output layer
model.add(tf.keras.layers.Lambda(lambda x: x * [1,1,0])) # new output layer

It appears to be working correctly.


Predicting High-School test scores after a disciplinary action

If your data is correct then i see a factor of gender also. I hope that is taken into account and also take Disp_action as categorical variable (one hot encoding).
Also, try to club pre and post action scores in one fields as output variable. Another field can simply tell if the output score is pre action or post action.
This feature engineering, I hope will give better results.


How does exactly class_weight in Keras work?

A weighted classification layer computes the weighted cross entropy loss for classification problems. Weighted cross entropy is an error measure between two continuous random variables. For prediction scores Y and training targets T, the weighted cross entropy loss between Y and T is given by

$$ F = -\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{M} w_i \cdot T_{ni} \cdot log(p_{ij}))$$

where $N$ is the number of instances, K is the number of classes, and w is a vector of weights for each class.

The cost function per label is measuring how good our classifier is doing per label, but it is affected by the number of instances $N$ that don't contain this label.


Machine Learning Dataset: Easy enough for fully connected, but not easy enough for logistic regression

Here's a one-dimensional problem that should be impossible for logistic regression:

Generate x uniformly on [0, 10]
Let y = sin(2 * pi * x)

If you want a classification problem define y_disc as:

0 if y > 1/sqrt(2)
1 if 1/sqrt(2) > y > -1/sqrt(2)
2 if y < -1/sqrt(2).

This is non-linear, so logistic regression should do poorly. Further, if you decide to put higher powers of x into your logistic regression classifier, I think you'll need a lot of powers to accurately represent the Taylor series near 10.

If fact, you could try adding a small noise term to y before discretizing - I suspect that will result in your logistic regressor overfitting long before it's able to accurately approximate the behavior near x=10.


Dimension change of convolutional layer after applying the feature map

In the video the speaker says, "center the filter on each pixel and perform a dot product which gives one value per position."
So I'm guessing it is a dot product between (L1, L2, L3) dot (F1, F2, F3) where L's are pixel values in the input and F's are whatever values are in the filter.


$\nabla_D$ means the gradient w.r.t. the discriminator parameters. The authors explained below as "the log gradient value of updating D".

BTW it's not standard to write something like $\nabla-a-b$ as this paper, which confuses readers as "$\nabla$ minus $a$ minus $b$". It's much better to write $-\nabla(a+b)$.


strings as features in decision tree/random forest

In most of the well-established machine learning systems, categorical variables are handled naturally. For example in R you would use factors, in WEKA you would use nominal variables. This is not the case in scikit-learn. The decision trees implemented in scikit-learn uses only numerical features and these features are interpreted always as continuous numeric variables.

Thus, simply replacing the strings with a hash code should be avoided, because being considered as a continuous numerical feature any coding you will use will induce an order which simply does not exist in your data.

One example is to code ['red','green','blue'] with [1,2,3], would produce weird things like 'red' is lower than 'blue', and if you average a 'red' and a 'blue' you will get a 'green'. Another more subtle example might happen when you code ['low', 'medium', 'high'] with [1,2,3]. In the latter case it might happen to have an ordering which makes sense, however, some subtle inconsistencies might happen when 'medium' in not in the middle of 'low' and 'high'.

Finally, the answer to your question lies in coding the categorical feature into multiple binary features. For example, you might code ['red','green','blue'] with 3 columns, one for each category, having 1 when the category match and 0 otherwise. This is called one-hot-encoding, binary encoding, one-of-k-encoding or whatever. You can check documentation here for encoding categorical features and feature extraction - hashing and dicts. Obviously one-hot-encoding will expand your space requirements and sometimes it hurts the performance as well.


What are machine learning/deep learning models for generating contextually related words and synonyms?

For synonyms I would directly use WordNet.

[added] For contextually similar words the traditional approach is to extract a context vector for every target word:

for every occurrence of a target word extract the words within a -/+ N window (e.g. N=5).
for every target word aggregate all its context words in a single context vector over the whole vocabulary.

Finally once a context vector has been calculated for every target word a similarity measure can be used, for example cosine. That means for every target word, compare its vector against any other candidate.

The same approach can be used with word embeddings instead of context vectors.


Aggregating target-encoded array-like categorical features?

There are several options:

Domain knowledge - Given what you know about the domain, combine the categories that make the most sense.

Empirical - Treat combing categories as a hyperparameter. Search through the space of options and pick the best combinations based on cross-validation score.


Attention model with seq2seq over sequence

The output of an Attention layer  - the Context - is typically the SUM of the weighted inputs. Each of the input is diminished or magnified by the attention weights based on how relevant it is at that time-step.
So the context will have the same shape as the input. This is typically (batch_size, Encoder_Embedding_dimension). You need to generate this context at every timestep of the decoder. So you will automatically get max_length context's. So the attention layer logic is correct above, you now need to call this in the decoder in a loop.
See "def train_step(inp, targ, enc_hidden):" in the link you have shared where this is done.


370 rows is quite a few, RF does bootstrap but it is still few info.
Having too many columns will lead to a more complex model (since the algorithm will work 1 000 dimensions).

Consider doing a pipeline with all the steps and search for hyperparameters and feature selection there. https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html

You can check a simple tutorial here

In summary, try to build a pipeline with all the steps but maybe the problem is in your data 370 rows is a small sample.


Premise:
The confusion matrix that you mention above is only correct if

there is as much Positive and Negative cases in the data
we use a random classifier that assigns Positive and Negative class both with probability 0.5

So you derive the confusion matrix for a very particular random classifier for a very particular dataset. This very particular random classifier is indeed, as you point out, only a point in the ROC plot. To obtain the whole ROC curve, we have to vary the probability with which we assign the positive class, from 0 to 1.
So in effects, the ROC curve is a graphical evaluation of the performance of infinitely many classifiers!
Each one of these random classifiers with a different probability will have a different expected confusion matrix.

Derivation: Here I derive the AUC of a random classifier on a dataset with an arbitrary class imbalance.
Assuming that there is a fraction $x$ of positive cases and a fraction $1-x$ of negative cases, and that our classifier consists of randomly assigning the positive class with probability $\rho$ and the negative class with probability $1-\rho$. The confusion matrix of a random classifier will have the following expected proportions
$$TP = \rho x \\ FP = \rho (1-x) \\ FN = (1-\rho)x \\ TN = (1-\rho)(1-x)$$
Then we calculate the True Positive Rate (sensitivity) and False Positive Rate (1-specificity) of our random classifier
$$TPR=\frac{TP}{TP+FN}=\frac{\rho x}{\rho x + (1-\rho)x}=\rho\\ FPR=\frac{FP}{TN+FP}=\frac{\rho(1-x)}{\rho(1-x)+(1-\rho)(1-x)}=\rho$$
So as you can see TPR and TFR do not depend on the class proportion $x$, which means that the ROC-AUC will also be independent of $x$. Moreover TPR=FPR, meaning that the ROC-curve will look like a straight line from (0,0) to (1,1). Now we calculate the AUC as the integral between 0 and 1 of the area under TPR as a function of FPR as we vary the threshold $\rho$.
$$AUC=\int_0^1 \rho d\rho' = \frac{\rho'^2}{2}|_0^1 = 1/2$$
So the area under the ROC curve for a random classifier is 0.5 regardless of the class proportion.


file name pattern matching and identifying

This does not sound like a machine learning problem. It seems to me that you could come up with a list of rules that determine whether two files belong together.

As S van Balen said, you should take a look at the glob and re packages in Python. (If you are on Linux, a Bash script would probably be much more elegant than a solution in Python).

The following links might be helpful:

Walk through a directory and its subdirectories to get a list of all files: https://stackoverflow.com/a/954522/4743630.
Group strings in a list based on partial matches (for instance: turn ['file_name1_xyz', 'file_name2_xyz', 'file_name1_abc', 'file_name2_def'] into [ ['file_name1_xyz', 'file_name1_abc'], ['file_name2_xyz', 'file_name2_def'] ]): https://stackoverflow.com/questions/40167651/python-iterate-through-a-list-of-strings-and-group-partial-matching-strings.

You mention that files may have different file types. Be careful when converting between types: Make sure for instance that separators and quote characters are treated correctly.


Does high error rate in regression imply the data set is unpredictable?

It's a little hasty to make too many conclusions about your data based on what you presented here. At the end of the day, all the information you have right now is that "GBT did not work well for this prediction problem and this metric", summed up by a single RMSE comparison. This isn't very much information - it could be that this is a bad dataset for GBT and some
other model would work, it could be that the label can't be predicted from these features with any model, or there could be some error in model setup/validation.

I'd recommend checking the following hypotheses:

1) Maybe, with your dataset size and the features you have, GBT isn't a very high-performance model. Try something completely different - maybe just a simple linear regression! Or a random forest. Or GBDT with very different parameter settings. Or something else. This will help you diagnose whether it's an issue with choice of models or with something else; if a few very different approaches give you roughly similar results, you'll know that it's not the model choice that is causing these results, and if one of those models behaves differently, then that gives you additional information to help diagnose the issue.

2) Maybe there's some issue with model setup and validation? I would recommend doing some exploration to get some intuition as to whether the RMSE you're getting is reasonable or whether you should expect better. Your post contained very little detail about what the data actually represents, what you know about the features and labels, etc. Perhaps you know those things but didn't include them here, but if not, you should go back and try to get additional understanding of the data before continuing. Look at some random data points, plot the columns against the target, look at the histograms of your features and labels, that sort of thing. There's no substitute for looking at the data.

3) Maybe there just aren't enough data points to justify complex models. When you have low numbers of data points (< 100), a simpler parametric model built with domain expertise and knowledge of what the features are may very well outperform a nonparametric model.


In Keras how to get the `class_indices` or prediction labels for an existing model

Don't know if you solved already.
I faced this problem yesterday, and the only solution that came to my mind is saving a dictionary to a numpy file after a train_generator is set; something like:

np.save('FILENAME', self.train_generator.class_indices)

when you finish your training.
Then, when you do your single prediction:

single_pred = np.squeeze(self.model.predict(image))

if isfile('FILENAME.npy'):
    class_indices = np.load(self.indices_file + '.npy').item()

results_dict = dict(zip(class_indices.keys(), single_pred))


Change the last layer's Neuron count to 2.
model.add(keras.layers.Dense( 2, activation="softmax"))
OR
Change your last layer's activation to Sigmoid and keep y_train with single column
model.add(keras.layers.Dense( 1, activation="sigmoid"))


Loading Data into DW: Direct Insert from PreProcessing vs PreProcess and then loading from CSV files

Loading the data directly from the preprocessing script to the DW means not storing the results of your processing on storage.

Not storing those results on storage may imply:

Needing to do the processing again if for some reason the data is needed again, for a new DW or for a new research.
Not saving the data as a data sink and so not contributing to a data lake architecture, where data sinks are just saved for future reusing on "unpredicted" situations.

From a technical perspective, some tradeoffs are mentioned on the Google BigQuery docs on Streaming data, from which the following could be highlighted:

Consistency management, as for errors or duplicates.
Waiting time until data is available for copy and export operations

Still, it would be great to have more comments about the technical perspective on when to use direct transfer and when to use a file for staging preprocessing results.

Thanks


What is difference between Bayesian Networks and Belief Networks?

Both are literally the same. A Belief network is the one, where we establish a belief that certain event A will occur, given B. The network assumes the structure of a directed graph. The term Bayesian was coined after the name of Thomas Bayes.


Should I build a different model for each subset

Build one model including the class variable as a categorical feature.

Since it is a high cardinality feature, there are different techniques that you can use:

One hot encoding (would create around 300 variables based on different classes)
Label encoding (would be a single variable but will assign weights to different classes - not ideal)
Hashing trick - use a hashing function to reduce the number of features
Embeddings
Cluster the different classes together


Deep Learning approach for Signature extraction from emails

Detecting a signature in an email is more like a detection problem in an image rather than a natural language processing (NLP) problem. It does not particularly matter what the signature says, you simply want to be able to detect it.

In such a case, detection in images is currently dominated by convolutional neural networks (CNN). This is a very powerful technique which has proved to be the best for a multitude of image-based tasks. However, it requires a very large dataset. So you will need to collect a lot of varied examples and annotate them yourself.

The output of the CNN can provide either a binary output 0/1 determining if a signature exists. Or it can provide an output which is the coordinates of the center of the signature (X, Y coordinates) such that you can know where it is placed in the document.


eta and learning_rate different in xgboost

Seems like eta is just a placeholder and not yet implemented, while the default value is still learning_rate, based on the source code. Good catch.
We can see from source code in sklearn.py that there seems to exist a class called 'XGBModel' that inherits properties of BaseModel from sklearn's API.
Tracing this to compat.py, we see there's an import statement:
from sklearn.base import RegressorMixin, ClassifierMixin

These are sklearn's Mixin classes for classifiers/regressors. Basically, they're the shells from which any specific classifier/regressor is generated, to guarantee that the descendant class has a score() method.
Looking carefully at the init function, we see:
__init__($self, /, *args, **kwargs)

From Yasoob's PythonTips (See link for examples of *args/**kwargs):

*args and **kwargs allow you to pass a variable number of arguments to a function. What variable means here is that you do not know beforehand how many arguments can be passed to your function by the user so in this case you use these two keywords...
*args is used to send a non-keyworded variable length argument list to the function.
**kwargs allows you to pass keyworded variable length of arguments to a function. You should use **kwargs if you want to handle named arguments in a function.

Now for a fun experiment to prove this. Try the following in your own notebook/console:
>>> XGBClassifier(eta=5, potato=1234)
XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
   colsample_bytree=1, eta=5, gamma=0, learning_rate=0.1,
   max_delta_step=0, max_depth=3, min_child_weight=1, missing=None,
   n_estimators=100, n_jobs=1, nthread=None,
   objective='binary:logistic', potato=1234, random_state=0,
   reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
   silent=True, subsample=1)

See that our argument was stored in the object:
`...objective='binary:logistic', potato=1234, random_state=0,`

We're pretty sure potato is not a hyperparameter right? :)


Using GANs to generate synthetic tabular data to improve supervised learning

GANs have many known problems. The main ones are:

Lack of convergence.
Vanishing gradients when discriminator is "too good", leading to stagnation of the generator.
Mode collapse: the diversity of the generated samples tends to be very low, generating always the same values.

GANs for image generation have been studied extensively. Other domains, like speech filtering, have also been studied, but not so extensively. In other domains, like text generation, GANs are not very successful. For tabular data generation via GANs, the amount of released work is scarce: medGAN, VeeGAN, ehrGAN, TableGAN, CTGAN.
I think that one of the main problems preventing us from devising better GANs in non-image domains is the evaluation. With images, you can eyeball the results and quickly determine if they are of good quality and diverse. However, with other domains, it is not easy to evaluate both the quality and diversity of the generated data.
I think most people nowadays stick to classical oversampling methods to generate tabular data.


Xgboost interpretation: shouldn't cover, frequency, and gain be similar?

Gain = Total gains of splits which use the feature. (In my opinion, features with high gain are usually the most important features)

Frequency = Numbers of times the feature is used in a model.

In my experience, these values are not usually correlated all of the time. I have had situations where a feature has the most gain but it was barely checked so there wasn't alot of 'frequency'.

Also, binary coded variables don't usually have high frequency because there is only 2 possible values. When it comes continuous variables, the model usually is checking for certain ranges so it needs to look at this feature multiple times usually resulting in high frequency.

For future reference, I usually just check the top 20 features by gain, and top 20 by frequency. If a feature appears in both then it is important in my opinion. Also, I wouldn't really worry about 'cover'. I don't think there is much to learn from that.


Is it possible to patent a new general method of optimizing some industrial processes?

Disclaimer: you need to ask a legal expert: software patents is a very complex and specialized domain, so don't rely on strangers on the internet ;)

As far as I know some algorithmic methods are patentable, but there are conditions to satisfy, in particular about the novelty of the method. You would probably have to prove the novelty by doing a thorough literature review, it's probably not enough to show that the software leads to efficiency improvements.


Predicted features combined with original ones

This procedure exists and is called stacked generalization or simply stacking. See stacking section from wikipedia page. Strating from there you can red more by following references from the page. The first paper on subject was published by Wolpert in 1992.

[Later edit]

Do not combine the results with original features, but keep only the predictions combined only with the target.


I assumed "where" meant "where condition is true, replace with X", but it's actually "Where this condition is true, KEEP the value".  This is odd to me but I found a different documentation page that more clearly explained the .where() method.

As it turns out, the .mask() method would return what I expected from .where().  Specifically:

df.mask(lambda x: x > 4, lambda x: x + 10)


Predicting household energy consumption?

At first sight, the total acumulated energy consumption seems to have a linear relation with time, so I suggest to try a linear regression at first. There are several libraries you can use to code it. I recommend you do it with pandas and sklearn, here is an answer related to this: answer.

If the relation is not linear, so I could try with a more complex model (but I suggest to keep simplicity at first). Since you are trying to predict a temporal serie, I would try with an LSTM model. Here is a tutorial to implement an LSTM neural network with keras.


Convert Pandas Dataframe with mixed datatypes to LibSVM format

As you've mentioned you can use the function by sklearn, I don't see the problem using it (perhaps I'm missing something)
import pandas as pd
from sklearn.datasets import dump_svmlight_file

def df_to_libsvm(df: pd.DataFrame):
    x = df.drop('label', axis=1)
    y = df['label']
    dump_svmlight_file(X=x, y=y, f='libsvm.dat', zero_based=True)

Regarding the categorical features with 10^6 unique categories, you can use a simple embedding for it into a binary vector.
One way to do it will be to map each username to a unique integer number.
Then you can convert the number to binary representation; that way you'll have a simple embedding of approximately size 20 (2^20=1,048,576) i.e. this feature is represented by 20 binary features.
Of course if the usernames are all unique, they probably shouldn't be a feature (identical to id).


What does it mean for a method to be invariant to diagonal rescaling of the gradients?

(Copied from my answer on stats.SE. (This meta.SE answer seems to approve of this copy-paste pattern.))

The original Adam paper briefly explains what it means by "invariant to diagonal rescaling of the gradients" at the end of section 2.1.

I would try to explain it in some more detail.

Like stochastic gradient descent (SGD), Adam is an iterative method that uses gradients in order to find a minimum of a function.
(By "gradients" I mean "the values of the gradient in different locations in parameter space". I later use "partial derivatives" in a similar fashion.)

But in contrast to SGD, Adam doesn't really use gradients. Instead, Adam uses the partial derivatives of each parameter independently.
(By "partial derivative of a parameter $x$" I mean "partial derivative of the cost function $C$ with respect to $x$", i.e. $\frac{\partial C}{\partial x}$.)

Let $\Delta^{(t)}$ be the step that Adam takes in parameter space in the $t^{\text{th}}$ iteration. Then the step it takes in the dimension of the $j^{\text{th}}$ parameter (in the $t^{\text{th}}$ iteration) is $\Delta^{(t)}_j$, which is given by:
$$\Delta^{(t)}_j=-\frac{\alpha}{\sqrt{{\hat v}^{(t)}_j}+\epsilon}\cdot {\hat m}^{(t)}_j$$
while:

$\alpha$ is the learning rate hyperparameter.
$\epsilon$ is a small hyperparameter to prevent division by zero.
${\hat m}^{(t)}_j$ is an exponential moving average of the partial derivatives of the $j^{\text{th}}$ parameter that were calculated in iterations $1$ to $t$.
${\hat v}^{(t)}_j$ is an exponential moving average of the squares of the partial derivatives of the $j^{\text{th}}$ parameter that were calculated in iterations $1$ to $t$.

Now, what happens when we scale the partial derivative of the $j^{\text{th}}$ parameter by a positive factor $c$?
(I.e. the partial derivative of the $j^{\text{th}}$ parameter is just a function whose domain is the parameter space, so we can simply multiply its value by $c$.)

${\hat m}^{(t)}_j$ becomes $c\cdot{\hat m}^{(t)}_j$
${\hat v}^{(t)}_j$ becomes $c^2\cdot{\hat v}^{(t)}_j$
Thus (using the fact that $c>0$), we get that $\Delta^{(t)}_j$ becomes:
$$-\frac{\alpha}{\sqrt{c^2\cdot{\hat v}^{(t)}_j}+\epsilon}\cdot c\cdot{\hat m}^{(t)}_j=-\frac{\alpha}{c\cdot\sqrt{{\hat v}^{(t)}_j}+\epsilon}\cdot c\cdot{\hat m}^{(t)}_j$$
And assuming $\epsilon$ is very small, we get:
$$\begin{gathered}
-\frac{\alpha}{c\cdot\sqrt{{\hat v}^{(t)}_j}+\epsilon}\cdot c\cdot{\hat m}^{(t)}_j\approx -\frac{\alpha}{c\cdot\sqrt{{\hat v}^{(t)}_j}}\cdot c\cdot{\hat m}^{(t)}_j=\\
-\frac{\alpha}{\sqrt{{\hat v}^{(t)}_j}}\cdot{\hat m}^{(t)}_j\approx-\frac{\alpha}{\sqrt{{\hat v}^{(t)}_j}+\epsilon}\cdot{\hat m}^{(t)}_j
\end{gathered}$$

I.e. scaling the partial derivative of the $j^{\text{th}}$ parameter by a positive factor $c$ actually doesn't affect $\Delta^{(t)}_j$.

Finally, let $g=\left(\begin{gathered}g_{1}\\
g_{2}\\
\vdots
\end{gathered}
\right)$ be the gradient. Then $g_j$ is the partial derivative of the $j^{\text{th}}$ parameter.

What happens when we multiply the gradient by a diagonal matrix with only positive elements?
$$\left(\begin{matrix}c_{1}\\
 & c_{2}\\
 &  & \ddots
\end{matrix}\right)g=\left(\begin{matrix}c_{1}\\
 & c_{2}\\
 &  & \ddots
\end{matrix}\right)\left(\begin{gathered}g_{1}\\
g_{2}\\
\vdots
\end{gathered}
\right)=\left(\begin{gathered}c_{1}\cdot g_{1}\\
c_{2}\cdot g_{2}\\
\vdots
\end{gathered}
\right)$$
So it would only scale each partial derivative by a positive factor, but as we have seen above, this won't affect the steps that Adam takes.

In other words, Adam is invariant to multiplying the gradient by a diagonal matrix with only positive factors, which is what the paper means by "invariant to diagonal rescaling of the gradients".


If you need it for a presentation, open the Notebook with a text editor, find the brackets where cell numbers fall and change them manually. If it's something you have to do once.


Difference between explicit and implicit density with and without the relation to neural network

This paragraph in this paper may be of help, methinks.


Are validation sets necessary for Random Forest Classifier?

You can use Out-of-bag error as you validation error, if you are short of data.

As you may know, Random Forest fits multiple decision trees, and for each tree it only fits on a subset of data. So data that hasn't been used for fitting a given tree is called Out of Bag data, and it could be used as your validation set 1

Sklearn in Python has a hyperparameter of Out-of-bag error


How to work with different Encoding for Foreign Languages

Nevermind, the solution was trivial. Since I had the .bin file I could just open it in binary form. If somebody doesn't really have the .bin file, they could consider converting the .txt file to .bin and solve further.


How do I decide if I need to go for Normalization and not Standardization or vice-versa?

Before we start keep in mind that in most cases it doesn't play much of a difference which of the two you'll choose.

Now to answer your question, generally speaking the choice should be made based on what model you want to employ:

If you use a distance-based estimator (e.g. k-NN, k-means) it's better to normalize your features so that they occupy the same exact range of values (i.e. $[0,1]$). This forces your estimator to treat each feature with equal importance.
If you're using Neural Networks, it's better to standardize your features, because gradient descent has some useful properties when your data is centered around $0$ with unit variance.
Tree-based algorithms don't require any form of scaling, so its irrelevant if you scale or normalize your features.

As a rule of the thumb, I usually standardize the data (unless I'm going to strictly work with distance-based algorithms).


Here is my understanding of your problem:

# Import
from torch import nn

# Define custom class
class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(10, 20) for _ in range(10)])

    def forward(self, x, indices):
        x = self.linears[indices](x)
        return x

# Intialize custom class
net = MyModule()

# Access networks layers based on additional input
additional_input = 1
if additional_input == 1:
    idx = 0
    print(net.linears[idx].in_features)


Support Vector Machines with soft margin: solving the dual form

if you want to find an analytical solution for the problem I think that it doesn't exist.
At this point you should apply decomposition (don't try to work on all your variables,
but iteratively considers only a subset of them called working set), if (and only if) you use a working set W, |W|=2, then your subproblem has an analytical solution.
Due to the large number of alpha variables, decomposition is the basis of svm implementations, in particular the svm light algorithm uses a working set W of cardinality 2.


BERT - How Question answering is different than classification

For Question Answering, you need 2 logits : one for the start position, one for the end position. Based on these 2 logits, you have an answer span (denoted by the start/end position).
In the source code, you have : pooled_output = self.pooler(sequence_output)
If you take a look at the pooler, there is a comment :

# We "pool" the model by simply taking the hidden state corresponding
# to the first token.

So the sequence output is all the token representations, while the pooled_output is just a linear layer applied to the first token of the sequence.

In classification case, you just need a global representation of your input, and predict the class from this representation. No need to access every token of the sequence, you want a single representation of the sequence.
But in the Question Answering case, in order to predict the answer span, you need to decide which token will be the start and which token will be the end of the span. For this, you need to consider every token independently.


How to support a dynamic shape input for tf.where()

Just do tf.zeros_like(s) instead of using the shape explicitly.


Aggregate categorical feature by the target

Interesting problem...

If I understand correctly you'd like to obtain clusters of states which have similar patterns/proportions of symptom+diagnosis, right?

If yes, I would suggest you reorganize the data so that one instance represents a state, with its features being the frequency of each pair (symptom, diagnosis). Based on this representation you could cluster states which have similar prevalence for a pair.

The disadvantage of this idea is that it considers pairs (symptom, diagnosis) as distinct even if only one of the two is different. Of course the same process could be done by considering only symptoms or only diagnosis. There are probably better approaches but this simple one might already provide some insight.


What is the appropriate name for this dataset?

Well, I would not call this "Test" set you defined as "Test"! I would say: Train, Validation, Test. And this "Test" is the dataset that your model has not seen yet. This is pretty standard even explained in details in Andrew Ng courses like this very useful and important lesson on splitting data and the significance of their distributions.


X_train, y_train from ImageDataGenerator (Keras)

in python 2:

X_train, y_train = train_generator.next()
 X_test, y_test = validation_generator.next()

in python 3:

X_train, y_train = next(train_generator)
X_test, y_test = next(validation_generator)


Is shuffling training data beneficial for machine learning?

Shuffling the training data is generally good practice during initial preprocessing steps.

When you do a normal train_test_split,  where you'll have a 75% / 25% split, your split may overlook class ordering in the original data set. For example, class labels that might resemble a data set similar to the iris data set would include target variables that resemble the following:

For example: [0, 0, 0, 1, 2, 2, 2, 3, 3, 3, 3, 3]

You could see from this example above, that splitting your data without shuffling might lead to very poor performance in your test set evaluation. Said another way, you may only capture the classes 0, 1, and 2 in your training data and only 3 will be represented in your test data. Specifically for classification tasks, but also for other ML tasks it may be useful to shuffle your data. However, each situation is different so the best idea would be to try it both ways to see whether you see a significant improvement or not.

Hope this answers your question. Drop a comment if you would like any further clarification.


Getting the positive impacting features using SHAP

Depending on your model there may be some better model-specific approaches than SHAP. It is also important to note that SHAP is an approximation of Shapley value, with the main assumption of not having too much correlation between your features.

That being said, taking the mean instead of the mean absolute values seems to be the most efficient approach in the continuity of what you are doing. Just keep in mind that :

SHAP don't have a "physical" interpretation, in terms of direct impact on the output. It may lack some meaning for real life users.
Taking the mean can "hide" skewed SHAP profile : a variable with high impact on a small subgroup of instances and no impact on the rest may get the same average than a variable with small impact on every instance.


An Artificial Neural Network (ANN) with an arbitrary number of inputs and outputs

The answer may depend on the significance of the length of the input vector or how it originates.
However, the simplest solution is usually to know the largest size input and use that as number of vectors. If the given input has lesser length, you can do the padding with zeros or appropriate symbols. So instead of having a vector $[1, 2, 3]$ and $[1, 2, 2, 3]$ you can have vectors $[1, 2, 3, 0]$ and $[1, 2, 2, 3]$.
The same can apply for output. If output expected is $[1, 2, 1]$ and $[1, 3, 4, 1]$ you can treat the first output as $[1, 2, 1, 0]$
Isn't this just a hack?
Typically, Neural networks does function approximation. Ideally, it represents vectors (matrices) as input and vectors (matrices) as output. This is why it is always desirable that size of your input vector be fixed.


Can I tune a model after training it? (Convolutional Neural Network & Classification)

Before answering your questions, let's understand a few points -
1. Model training is nothing but finding the right weights going through the guiding path based on the loss function o/p using y_true and y_pred.
Training starts with random initialization of weights(definitely following some rules)
2. Neural Network training is an incremental training approach i.e. it doesn't see all the data in one go like the DecisionTree

Can I use the current model with weights and bias to retrain on augmented data (from the same initial dataset)? I am guessing there are 2 possible outcomes, it tunes the model or it "resets" the weights and bias

Yes, learning will continue without any big disruption as augmented images will produce a similar gradient like the real images.
It will not reset, as I said the training is an incremental process.
Any new image which has completely different features can definitely cause a large gradient to flow and might disrupt the weights but with augmented images, I don't think any such thing should happen.

Can I perform hyperparameter tuning with RandomSearchCV or GridSearchCV using on my existing model? I believe this will tune the model.

You can do this but you don't have too many hyperparameters left as Model is fixed. You may tune Learning rate.


what is difference between set() and word_tokenize()?

There are two differences between word_tokenize and set.

Word_tokenize

Returns a list (try print(type(word_tok)))
Returns all the tokens, regardless if there are duplicates

Set

Returns a set (try print(type(set_all)))
Returns all the unique tokens

Try this instead

sentence = 'jainmiah jainmiah jainmiah I love you but you are not bothering about my request, please yaar consider me for the sake'

word_tok = word_tokenize(sentence)
print(word_tok)

set_all = set(word_tokenize(sentence))
print(set_all)


Regularization - Combine drop out with early stopping

Avoid early stopping and stick with dropout. Andrew Ng does not recommend early stopping in one of his courses on orgothonalization [1] and the reason is as follows. For a typical machine learning project, we have the following chain of assumptions for our model:

Fit the training set well on the cost function

↓

Fit the dev set well on the cost function

↓

Fit the test set well on the cost function

↓

Performs well in the real world

And what we want are tools that can target one of these four objectives and not the others in order to keep improving our models more efficiently, and this concept is called "orthogonalization." An analogy would be the different options in photo editing apps such as brightness, contrast, and saturation adjustments, which are independent of, or "orthogonal" to one another. Examples of orthogonal optimization strategies for the four goals are listed below:

Fit the training set well on the cost function (e.g. bigger neural network; Adam optimization)

↓

Fit the dev set well on the cost function (e.g. regularization; bigger training set)

↓

Fit the test set well on the cost function (e.g. bigger dev set)

↓

Performs well in the real world (e.g. change the test set; change the cost function)

Because early stopping both fits the training set less well and improves the dev set performance at the same time, it is not orthogonal and Ng advises us not to use it.

Reference:
[1] Week 1 of Course 3: Structuring Machine Learning Projects of Coursera Deep Learning Specialization


Why does Bagging or Boosting algorithm give better accuracy than basic Algorithms in small datasets?

SVMs finds a boundary to divide the data between the two classes, and the rule of thumb is to use SVMs for binary classification problems. Typically, when you have linear dependencies SVMs performs better than random forest since there is a distance to measure. But random forest performs better with categorical values / non-linear dependencies. Random forest does benefit from more data, but nature of data even in small data-sets matters.

Logistic regression follows similar logic, if your underlying relation is linear in nature then it probably perform better than Random Forest.


Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak.

Think of it like this, the test set is supposed to be a way of estimating performance on totally unseen data. If it affects the training, then it will be partially seen data.

I don't think the order of scaling/imputing is as strict. I would impute first if the method might throw of the scaling/centering.

Your steps should be:

Splitting
Imputing
Scaling

Here are some related questions to support this:

Imputation before or after splitting into train and test?

Imputation of missing data before or after centering and scaling?


Daily new data for my neural network, and I want transfer(?) learning

So, I believe the question you are asking is how to re-train the model on a single day's worth of data, rather than training on all 180 day's worth of data.

It definitely seems reasonable that you have seen overfitting, this can be due to the complexity of the model (related to depth of network).

So changing the model's architecture to suit this classification task is definitely a good idea. For this, without knowing your model architecture, I would look at the overall depth (i.e. number of layers in neural network).

To clarify, LSTMs (like RNNs [Recurrent Neural Networks]) are typically used for sequential data as data  $x_i \in \textbf{x}$, where $0 < i < N$ ($N$ is number of data examples in sequence) is fed sequentially in the model to either output at each time step $i$ or later in the input sequence.

One suggestion from the top of my mind is, since a RNN is simply an normal NN with a recurring hidden layer, we can use transfer learning (which you spoke about). Transfer learning essentially when we pre-train a model on usually a large quantity of (generic) data and then we take this pre-trained model and alter the output section of the network's architecture to suit the classification task (in your case, add a softmax layer to output a probability distribution over the 2 classes). We then train this altered model with different data (in your case your single day's worth of data).
To prevent the model from losing generalisation performance, we typically freeze parameter updates within the pre-trained model's hidden layers and only allow the altered section of the model to update their parameters. This also has the added benefit of reducing time taken to train the model.
So, that would be my suggestion, firstly pre-train your model with the 180 day's worth of data, then alter your pre-trained model to suit this task to classify for a day's worth of day, train the model but only update parameters in the altered section of the model.


What loss function does the 'multinomial' distribution with the gbm package in R use?

Here is the relevant source code. I'm pretty sure it's using cross-entropy for multiclass:

    //  GBM by Greg Ridgeway  Copyright (C) 2003

#include "multinomial.h"

CMultinomial::CMultinomial(int cNumClasses, int cRows)
{
   mcNumClasses = cNumClasses;
   mcRows = cRows;

   madProb = new double[cNumClasses * cRows];
}

CMultinomial::~CMultinomial()
{
   if(madProb != NULL)
   {
      delete [] madProb;
   }
}

GBMRESULT CMultinomial::UpdateParams
(
   double *adF,
   double *adOffset,
   double *adWeight,
   unsigned long cLength
)
{
   // Local variables
   unsigned long ii=0;
   unsigned long kk=0;

   // Set the probabilities for each observation in each class
   for (ii = 0; ii < mcRows; ii++)
   {
      double dClassSum = 0.0;
      for (kk = 0; kk < mcNumClasses; kk++)
      {
         int iIdx = ii + kk * mcRows;
         double dF = (adOffset == NULL) ? adF[iIdx] : adF[iIdx] + adOffset[iIdx];
         madProb[iIdx] = adWeight[iIdx] * exp(dF);
         dClassSum += adWeight[iIdx] * exp(dF);
      }

      dClassSum = (dClassSum > 0) ? dClassSum : 1e-8;

      for (kk = 0; kk < mcNumClasses; kk++)
      {
         madProb[ii + kk * mcRows] /= dClassSum;
      }
   }

   return GBM_OK;
}

GBMRESULT CMultinomial::ComputeWorkingResponse
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adF,
    double *adZ,
    double *adWeight,
    bool *afInBag,
    unsigned long nTrain,
    int cIdxOff
)
{
    unsigned long i = 0;

    for(i=cIdxOff; i<nTrain+cIdxOff; i++)
    {
       adZ[i] = adY[i] - madProb[i];
    }

    return GBM_OK;
}

GBMRESULT CMultinomial::InitF
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adWeight,
    double &dInitF,
    unsigned long cLength
)
{
    dInitF = 0.0;
    return GBM_OK;
}

double CMultinomial::Deviance
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adWeight,
    double *adF,
    unsigned long cLength,
    int cIdxOff
)
{
    unsigned long ii=0;
    double dL = 0.0;
    double dW = 0.0;

    for(ii=cIdxOff; ii<cLength+cIdxOff; ii++)
    {
        dL += -adWeight[ii] * adY[ii] * log(madProb[ii]);
        dW += adWeight[ii];
    }

    return dL/dW;
}

GBMRESULT CMultinomial::FitBestConstant
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adW,
    double *adF,
    double *adZ,
    unsigned long *aiNodeAssign,
    unsigned long nTrain,
    VEC_P_NODETERMINAL vecpTermNodes,
    unsigned long cTermNodes,
    unsigned long cMinObsInNode,
    bool *afInBag,
    double *adFadj,
   int cIdxOff
)
{
      // Local variables
    GBMRESULT hr = GBM_OK;
    unsigned long iNode = 0;
    unsigned long iObs = 0;

   // Call LocM for the array of values on each node
    for(iNode=0; iNode<cTermNodes; iNode++)
    {
        if(vecpTermNodes[iNode]->cN >= cMinObsInNode)
        {
         // Get the number of nodes here
         double dNum = 0.0;
         double dDenom = 0.0;
         for (iObs = 0; iObs < nTrain; iObs++)
         {
            if(afInBag[iObs] && (aiNodeAssign[iObs] == iNode))
                {
               int iIdx = iObs + cIdxOff;
                    dNum += adW[iIdx] * adZ[iIdx];
               dDenom += adW[iIdx] * fabs(adZ[iIdx]) * (1 - fabs(adZ[iIdx]));
                }
         }

         dDenom = (dDenom > 0) ? dDenom : 1e-8;

         vecpTermNodes[iNode]->dPrediction = dNum / dDenom;
        }
    }

    return hr;
}

double CMultinomial::BagImprovement
(
    double *adY,
    double *adMisc,
    double *adOffset,
    double *adWeight,
    double *adF,
    double *adFadj,
    bool *afInBag,
    double dStepSize,
    unsigned long nTrain
)
{
    double dReturnValue = 0.0;
    double dW = 0.0;

   unsigned long ii;
   unsigned long kk;

   // Calculate the probabilities after the step
   double *adStepProb = new double[mcNumClasses * mcRows];

   // Assume that this is last class - calculate new prob as in updateParams but
   // using (F_ik + ss*Fadj_ik) instead of F_ik. Then calculate OOB improve
   for (ii = 0; ii < mcRows; ii++)
   {
      double dClassSum = 0.0;
      for (kk = 0; kk < mcNumClasses; kk++)
      {
         int iIdx = ii + kk * mcRows;
         double dF = (adOffset == NULL) ? adF[iIdx] : adF[iIdx] + adOffset[iIdx];
         dF += dStepSize * adFadj[iIdx];
         adStepProb[iIdx] = adWeight[iIdx] * exp(dF);
         dClassSum += adWeight[iIdx] * exp(dF);
      }

      dClassSum = (dClassSum > 0) ? dClassSum : 1e-8;

      for (kk = 0; kk < mcNumClasses; kk++)
      {
         adStepProb[ii + kk * mcRows] /= dClassSum;
      }
   }

   // Calculate the improvement
    for(ii=0; ii<nTrain; ii++)
    {
        if(!afInBag[ii])
      {
         for (kk = 0; kk < mcNumClasses; kk++)
         {
            int iIdx = ii + kk * mcRows;
                dReturnValue += adWeight[iIdx] * adY[iIdx] *
                               (log(adStepProb[iIdx]) - log(madProb[iIdx]));

            dW += adWeight[iIdx] * adY[iIdx];
         }
      }
    }

    return dReturnValue/dW;
}


Why is the result of CountVectorizer * TfidfVectorizer.idf_ different from TfidfVectorizer.fit_transform()?

TfidfVectorizer will by default normalize each row. From the documentation we can see that:

  norm : ‘l1’, ‘l2’ or None, optional (default=’l2’)
  Each output row will have unit norm, either: * ‘l2’: Sum of squares of vector elements is 1. The cosine similarity between two vectors is their dot product when l2 norm has been applied. * ‘l1’: Sum of absolute values of vector elements is 1. See preprocessing.normalize

Setting norm to None will give the result you expect:

tfidf_docs = TfidfVectorizer(vocabulary=vocab_docs, norm=None)


Pivot DataFrame while calculating new values

The magic word is "pivoting":

records = [(1871,"SS",0.0,3.0,1.0), (1871,"2B",30.0,1.0,0.0)]
pandas.DataFrame.from_records(records, columns=("yearID", "POS", "PO", "A", "E")
     ).assign(result = df.apply(lambda x: x['E']/(x['PO']+x['A']+x['E']), 1)
     ).pivot(index='yearID', columns='POS', values='result')

Result:

| POS    | 2B | SS   |
|--------|----|------|
| yearID |    |      |
| 1871   | 0  | 0.25 |

I went to the trouble of looking at your file now that you linked to it, and the problem is that you have removed to much information, so your index is not unique. Indices always have to be unique, so either you add that information back, or you preprocess the data before pivoting such that the duplication is resolved. I chose to retain the extra columns; I don't know if this is what you want, but it should help you understand what it takes to make pivoting work:

from pandas import read_csv

field_cols = ("playerID", "yearID", "teamID", "POS", "PO", "A", "E")
df = read_csv('Fielding.csv', usecols=field_cols).dropna(
             ).query('not PO == A == E == 0')

df.assign(ID = df[['playerID', 'yearID', 'teamID']].apply(tuple, 1),
          FP = df.apply(lambda x: 1-x['E']/(x['PO']+x['A']+x['E']), 1)
          ).drop(set(df.columns) - {'POS'}, 1).drop_duplicates('ID'
          ).pivot(index='ID', values='FP', columns='POS')

Result:

| POS                    | 1B  | 2B  | 3B  | C   | CF  | LF  | OF  | P        | RF  | SS  |
|------------------------|-----|-----|-----|-----|-----|-----|-----|----------|-----|-----|
| ID                     |     |     |     |     |     |     |     |          |     |     |
| (aardsda01, 2006, CHN) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |
| (aardsda01, 2007, CHA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 0.857143 | NaN | NaN |
| (aardsda01, 2008, BOS) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |
| (aardsda01, 2009, SEA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 1.000000 | NaN | NaN |
| (aardsda01, 2010, SEA) | NaN | NaN | NaN | NaN | NaN | NaN | NaN | 0.833333 | NaN | NaN |
  ...


Problem with escape character in paste-function in R

This works:
> paste("C:\\",filename,sep="")
[1] "C:\\testfile.txt"

Although you see TWO backslashes, there's really only one there, because usually when printed a backslash means "the next character is encoding a special character". In this case the special character is literally a backslash.
You can see this by counting the number of characters in two backslashes:
> nchar("\\")
[1] 1

And anyway you should probably be using file.path to construct paths and not fiddling around with backslashes:
> file.path("C:",filename)
[1] "C:/testfile.txt"

(Note forward slashes here also work in Windows and don't suffer from having to be backslash-encoded).


Different Initial Q-Values in Q-Learning

In the long-run, tabular Q-learning converges toward the optimal regardless of initialization.
However, the speed of convergence may be affected, similarly to an n-armed bandit setting : http://incompleteideas.net/book/first/ebook/node21.html
For more on initialization in Q learning, I recommend "Potential-based shaping and Q-value initialization are equivalent" by Eric Wiewiora.


How to use machine learning to create combine of opposite images side by side

Maybe something like GANs would work. It can use some base picture and change the context, add some opposite attributes. There is an example of Generative Adversarial Style Transfer Networks to clarify what I mean http://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w41/Palsson_Generative_Adversarial_Style_CVPR_2018_paper.pdf In this example age is change by the model and it looks very natural, therefor the state of appearance could be change too (from poor to rich or vice versa)
https://neurohive.io/en/news/gated-gan-style-transfer-with-multiple-styles/
The first image with a girl conveys the idea of gan using, but it should be thought over more.


ValueError: Error when checking target: expected dense_1 to have shape (7,) but got array with shape (1,)

This normally appears in the training.py/training_utils.py file from the keras package.

I was having a problem when I was using the LIME package on a keras model. (This is a work-in-progress, so the workaround mentioned here may not be the correct approach)

It appears that the shape of the data file differs from the one that the module is expecting. I did a workaround by collapsing the shape of the data object to the one required. Seems to have worked.


Can we remove features that have zero-correlation with the target/label?

Can I drop these features to improve the accuracy of my classification problem?

If you are using a simple linear classifier, such as logistic regression then yes. That is because your plots are giving you a direct visualisation of how the model could make use of the data.

As soon as you start to use a non-linear classifier, that can combine features inside the learning model, then it is not so straightforward. Your plots cannot exclude a complex relationship that such a model might be able to exploit. Generally the only way to proceed is to train and test the model (using some form of cross-validation) with and without the feature.

A plot might visually show a strong non-linear relationship with zero linear correlation - e.g. a complete bell curve of feature versus target would have close to zero linear correlation, but suggest that something interesting is going on that would be useful in a predictive model. If you see plots like this, you can either try to turn them into linear relationships with some feature engineering, or you can treat it as evidence that you should use a non-linear model.

In general, this advice applies whether or not the features are derived features. For a linear model, a derived feature which is completely uncorrelated with the target is still not useful. A derived feature may or may not be easier for a non-linear model to learn from, you cannot easily tell from a plot designed to help you find linear relationships.


Statistical test to establish that event B is caused by event A

This would be done with a controlled experiment, similar to clinical tests:

Randomly assign customers into to groups 1 and 2. Group 1 is given the attractive offer, group 2 is not (control group).
It's important to avoid any external bias, so for example ideally nobody in the company should know whether a customer is in group 1 or group 2. The goal is to make sure that the only difference between the two groups is the offer.

After a pre-determined duration (e.g. 3 months), collect the results (how many customers in group X have left/stayed). Then a simple Chi-square test can be performed on the 2x2 contingency table in order to assess whether there's a significantly different rate of leaving between the two groups. Since the offer is the only possible explanation, this proves that the offer causes the difference.


You get the same effect from including a bias term, i.e. $\frac{1}{1+\exp(-\beta x + bias)}$, and the defaults for most software is to include such bias.

To see why, you can do the math, expanding $\beta(x-a)$ and seeing that you can express the sum of those differences as a single variable.


Are Denoising Autoencoders for anomaly detection on structured data?

When you run denoise auto-encoders or any process on image data, what you did first was to convert run image data on a structured dataset with RGB arrays arranged in a column-like order.

Which means you turned your image data into structured data and then you applied the denoise-autoencoders, this means that you used the autoencoders on structured data.

Actually, almost no process is done in unstructured data. They run on the structured version of the structured data.

So the answer is yes, the denoise autoencoders does anomaly detection on structured data.


Math behind 2D convolution for RGB images

If your image is 3D then your kernel should be 3D too. Of course, you can also apply the 2D in which the same filter will be applied to all channels.
Image Source (Content is also well).

However, normally you apply a 3D filter to a 3D image. So if you apply 16 filters of size 3x3x3 to an image of size 6x6x6, then you will get 16 outputs of size 4x4. If you would apply 16 filters of size 3x3 filters, you would get 16 outputs of size 4x4x3. It would be treating each channel separately. But when you use a 3D filter, your output of convolution operation depends on all three dimensions. In other words, you multiply your 27 points from your 3x3x3 filter with the corresponding 27 points (3x3 pixels and their 3 channels) from the image, and then add them to get the result. Thus, 1 more dimension would be there for you to handle (16x4x4x3 instead of 16x4x4).
The answer to your question 1 is Yes, you would apply the filter 1 channel at a time.
Check the link for a very good explanation by Andrew NG.


Pytorch doing a cross entropy loss when the predictions already have probabilities

You can implement categorical cross entropy pretty easily yourself. It is calculated as

$$
\text{cross-entropy} = -\frac{1}{n} \sum_{i=0}^{n} \sum_{j=0}^m \mathbf{y}_{ij} \log \hat{\mathbf{y}}_{ij}
$$

where $n$ is the number of samples in your batch, $m$ is the number of classes, $\mathbf{y}_i$ is the one-hot target for example $i$, $\mathbf{\hat{y}}_i$ is the predicted probability distribution, and $\mathbf{y}_{ij}$ refers to the $j$-th element of this array.

In PyTorch:

def categorical_cross_entropy(y_pred, y_true):
    y_pred = torch.clamp(y_pred, 1e-9, 1 - 1e-9)
    return -(y_true * torch.log(y_pred)).sum(dim=1).mean()

You can then use categorical_cross_entropy just as you would NLLLoss in the training of a model. The reason that we have the torch.clamp line is to ensure that we have no zero elements, which will cause torch.log to produce nan or inf.

One difference you'll have to make in your code is that this version expects a one-hot target rather than an integer target. You can easily convert your current target list like so:

one_hot_targets = torch.eye(NUM_CLASSES)[targets]

where targets is a torch.tensor with integer values and NUM_CLASSES is the number of output classes that you have.


Good explanation for why regularisation works

The "same angle hyperplane" does not have the same cost. It is the same decision boundary as you describe it, but perpendicular distances to it are larger wrt the norm of the weights. In effect with higher weights in the same ratio (i.e. without any regularisation effect), the classifier will be more confident in all of its decisions. That means the classifier will be more sensitive to getting as many observations as possible in the training set on the "right" side of the boundary. In turn this makes it sensitive to noise in the observations.

Your estimated probability for being in the positive class is:

$$p(y=1|X) = \frac{1}{1+e^{-W^TX}}$$

This includes $w_0$ and fixed value 1 for $x_0$. If you take the midpoint, decision line where $W^TX$ is zero (and the output is at the threshold value 0.5), that defines your decision hyperplane in $X$ space.

When $W$ has the same factors but a larger norm with $w_0$ compensating to make the same hyperplane, then $X$ values that were on the decision hyperplane still give threshold value of 0.5. However, $X$ values away from the hyperplane will deviate more strongly. If instead of 0 you had $W^TX=1.0$ and doubled the weights keeping the same hyperplane, you would get $W^TX=2.0$ for that example. Which changes your confidence from 0.73 to 0.88.

The usual cost function without regularisation for logistic regression with example vectors $X_j$ and targets $y_j$ is:

$$J = - \sum_{\forall j} y_jlog(\frac{1}{1+e^{-W^TX_j}}) + (1 -y_j)(1  - log(\frac{1}{1+e^{-W^TX_j}}))$$

The cost is more sensitive to distances from the hyperplane for larger weight values. Looking your example for the imaginary item (with 0.73 or 0.88 confidence), when the categorisation is correct (i.e. y=1) the score would improve by 0.19 for that example if the weights doubled. When the categorisation is wrong (y=0) then the score would worsen by 0.81. In other words for higher weights, with same weight ratio, the same miscategorisations are punished more than correct categorisations are rewarded.

When training, weights will converge to specific balanced weight vector for the minimum cost, not to a specific ratio that forms a "best decision hyperplane". That's because the hyperplane does not correspond to a single value of the cost function.

You can demonstrate this effect. Train a logistic regression classifier - without any regularisation to show it has nothing to do with that. Take the weight vector and multiply by some factor e.g. 0.5.Then re-train starting with those weights. You will end up with the same weights as before. The cost function minimum clearly defines specific weight values, not a ratio.

When you add regularisation, that changes the cost and how the weights will converge. Higher regularisation in effect makes the classifier prefer a boundary with lower confidence in all its predictions, it penalises "near misses" less badly because the weights are forced down where possible. When viewed as a hyperplane, the boundary will likely be different.


Understanding regularisation and a preference for small weights

Ridge or L2 regularization is used to prevent over-fitting when having multi-col-linearity in your features. Here is an example : suppose you want to train a linear regression model on following dataset:

  x1      x2      y
 1.0    -1.0      5
-1.0     1.0      5

Training a linear regression model, we are searching for an hypothesis function with following format :

As you can observe, there are multiple solutions to this problem. For example :

theta0 = 5, theta1 = 1 and theta2 = 1

But, there are also multiple other solutions to this problem. Including :

theta0 = 5, theta1 = 1000000 and theta2 = 1000000

Now imagine you have a new observation such as:

      x1      x2      y
 1.00001    -1.0      5

A solution with small theta coefficients is most likely to yield better results than a solution with large theta coefficients. To understand L2 regularization, we have to understand that multicollinearity can yield to less robust solutions because is can appear ridges in cost function surface and so result in a less good solution. By adding a penalization term, we will smooth the cost function surface to make learning focus on small theta coefficients :

So, going back to your temperature example, ridge regression will reduce temperature theta coefficient as much as possible in order to avoid a potential multicollinearity with other features. By adding L2 regularization, it is now possible to address overfitting issues in model if there is multicollinearity in your features. However, if your regularization strength is too large, you risk to force your learning to focus on small values of theta only and so it could yield a bias emergence in your model and so an underfitting situation.


Intuition: why are ReLu activations boundary lines linear?

As far as i understand, relu gives linear boundary because it is linear for X>0. If you expand it to write it in the form of an equation like Z=f1(x1)+f2(x2)....fn(xn), we simply get a linear equation even if we consider the dead neurons. The part where x<0 makes sure that no -ve signals are carried forward but dosent add any non linearity to the final equation. The effect of dead neurons probaly the the final coefficients of the equaion will have coefficients with slightly less value than had the activation been just a normal fuction since there is no negative component which is added to a neuron.


how to decide categorical variables for prediction

Short answer : yes.

Gradient boosting relies on decision trees. The leaves of your decision trees are built in a fashion to discriminate optimally your features. For numeric features, this means finding the best separation value to decline your dataset into two subsets. One contains observations with a value above or equal to this separation value, while the other presents values that are below this separation value.

It would not make any sense to have a leaf splitting your data on a criterion such as $Store >= 5$. However, it would make sense to have a separator such as $Store_5 = 1$ (vs $Store_5 = 0$). This is precisely why dummy variables are created for categorical values in ensemble methods, such as gradient boosting.


How to start building a statistical regression analysis model with multiple categorical/discrete input variables of high dimension in Python

First, you have to pre-process your data. it includes encoding your categorical variables. You can either do it using pandas.get_dummies, or sklearn.preprocessing.OneHotEncoder in your pipeline. Based on the algorithm you want to use, you usually have to standardize your numerical variables. this can be done using any of sklearn.preprocessing methods such as StandardScaler.

If you want to do logistic regression, you have to categorize your output variable - time - to classes like ['less than 1000 min', 'more than 1000 min']. Otherwise, if you want to keep time as it is, you are doing a multiple regression.


How to perform Anomaly Detection on a force profile?

I strongly recommend to start with a simple statistical data analysis. In this approach you take the Moving Average/Median of signals and if one signal shows a magnitude -/+ 3 times standard deviation, you mark it as anomaly. Please have a look at this answer for Python code.


Hi Soumyadeep and welcome to Data Science/Stack Exchange
What you are describing is called regression imputation, and it is a valid method to use on missing data. However, if the data is sparse (lots of missing values), this issue will be more difficult to handle.
In general, missing data can be handled in several ways (row deletion, imputation, substitution, etc). Regression imputation can be used if you have little or no knowledge about the data, but usually it is better to use another method. If you have some domain knowledge about the missing values, like you have an idea what the value should be, usually you can use that knowledge to fill in the missing values. Try some different methods and see which one works best.

A person pointed out that I should check for multicollinearity if the
features are both independent. Does it basically mean that one feature
is falling in the span of the other feature?

Definition of multicollinearity:
There exist one or more exact linear relationships among some of the variables

References:
https://en.wikipedia.org/wiki/Multicollinearity
https://stats.stackexchange.com/questions/234870/is-multicollinearity-the-issue-here


400 positive and 13000 negative: how to split dataset up (train, test, validation)

For the problem of an imbalanced dataset, you can look into stratified sampling, or stratified-cross-validation (as mentioned here). One idea might be to create stratified batches from the data.

I would probably make all attempts to get train/val/test splits, because you will otherwise face issues when claiming a final test accuracy, as the model might well have seen your entire dataset.

One could imagine splitting to have e.g. 300/9750 (pos/neg) in the training dataset, and during training, you create stratified batches from those 1050 images, so each batch e.g. of 50 images, might contain, 10 positives and 40 negatives. This is still somewhat imbalanced, but you are pushing the balance into a more favourable direction in that the model should be able to learn more efficiently.

In medical research it is often the case that there are too few samples (in addition to class imabalnces), and so there is usually a huge effort that goes into data augmentation, which you might also be able to make use of it. Here is some related literature  (extremely fresh - edited one week ago!).

Here is another approach, whereby the authors (William Fithian & Trevor Hastie) devise a subsampling method, which uses the features of the samples to accept/reject them. They design it for the simplist case (logistic regression), but perhaps it might give you ideas:

  ... using a pilot estimate to preferentially select examples whose responses are conditionally rare given their features.

Something to be especially aware of when using the ideas I mentioned above is overfitting. Cross-validation is probably what can best help you out in this respect.


If I had to calculate such a function, I would:

Calculate the probability (not Z-score) for $x$ in a two-tailed test (https://en.wikipedia.org/wiki/One-_and_two-tailed_tests) for the probability that $x$ belongs to the distribution for both distributions. Then you have $p_s(x)$ and $p_c(x)$ for self-match/cross-match. Don't use probability $0.1\%$ to reject, but to compare.

Then calculate $$\frac{p_s(x)}{p_s(x) + p_c(x)}$$

Which tends to $1$ if $p_s(x)$ is the larger term, is $0.5$ if they are the same, and tends to $0$ if $p_c(x)$ is the larger term.

At the far tails, $p_s(x)$ and $p_c(x)$ both tend to zero, but the order of magnitude will select the good probability.


Probability or score on how likely an article is motivated by content marketing

If you have a broad set of testing data, I think this is feasible. I've had luck in getting basic models to identify abstract concepts like someone's politics or real news vs fake news, so I think this could work if people write content marketing different from normal news.

I just used this link in another post, but here is a tutorial in Python of taking unstructured articles, passing them through an NLP pipeline, and classifying them into multiple groups.


You can generate a config file in your home directory by running:

jupyter notebook --generate-config

which should leave you with a file ~/.jupyter, called jupyter_notebook_config.py.

Have a look at all the possible configurations in the Jupyter documentation.

In addition to many server options, there is specifically an options to override Tornado configuration.

  NotebookApp.tornado_settings : Dict Default: {}

  Supply overrides for the tornado.web.Application that the Jupyter
  notebook uses.


What does discriminator of a GAN should do?

The discriminator must classify individual elements as being fake (i.e. created by the generator) or real (i.e. taken from the training dataset). The discriminator generates labels (real/fake) for each element in the batch. The loss functions are computed based on those labels.

Elements are fed to the discriminator in batches of the same type (i.e. all elements in the batch are real or all elements in the batch are fake). This is because the fake data batch is directly generated by the generator as part of the same computational graph (i.e. the output of the generator is
 directly connected to the input of the discriminator). This is so to be able to propagate gradients of the generator parameters through the discriminator.


You just need to make sure that your x and y values have the same row dimension. This can be done using numpy.transpose or numpy.swapaxes.
import numpy as np

t = np.random.rand(150528, 1235)
np.transpose(t).shape # (1235, 150528)
np.swapaxes(t, 0, 1).shape # (1235, 150528)

This should give the shapes of (1235, 150528) for x and (1235, 154457) for y.


I would say hierarchical clustering is usually preferable, as it is both more flexible and has fewer hidden assumptions about the distribution of the underlying data.

With k-Means clustering, you need to have a sense ahead-of-time what your desired number of clusters is (this is the 'k' value).  Also, k-means will often give unintuitive results if (a) your data is not well-separated into sphere-like clusters, (b) you pick a 'k' not well-suited to the shape of your data, i.e. you pick a value too high or too low, or (c) you have weird initial values for your cluster centroids (one strategy is to run a bunch of k-means algorithms with random starting centroids and take some common clustering result as the final result).

In contrast, hierarchical clustering has fewer assumptions about the distribution of your data - the only requirement (which k-means also shares) is that a distance can be calculated each pair of data points.  Hierarchical clustering typically 'joins' nearby points into a cluster, and then successively adds nearby points to the nearest group.  You end up with a 'dendrogram', or a sort of connectivity plot. You can use that plot to decide after the fact of how many clusters your data has, by cutting the dendrogram at different heights.  Of course, if you need to pre-decide how many clusters you want (based on some sort of business need) you can do that too.  Hierarchical clustering can be more computationally expensive but usually produces more intuitive results.


I am staying quite generic since you asked for enlightenment, just mentioning some possible directions that you can explore.

You have basically two possibilities:

Classification of the text (Supervised learning).
Supervised means
that you need first to externally apply labels (for example manually by humans) to examples of texts
(labels could be "politics" or "show") and then use one of the
classification algorithms.  You have extracted words from the text,
so you could use a "bag of words" approach for classifying.  There exist adaptations
of classification algorithms (multi-label classification) in order to
provide multiple labels (such as one text is labelled both with
"music" and "movie").
You can find text corpora already pre-labelled, to train the algorithm and partly avoid the manual effort.
Clustering the text, topic modelling (unsupervised learning).
In this case you do not need to provide examples with labels but the algorithm will cluster the texts according to parameters like the similarity of two texts or on keywords/topics extracted from the texts.
Although this method do not require labeled examples, the clusters you get as output will require fine tuning, e.g. number of clusters to produce, names of the clusters, etc.

Since you mentioned SKlearn, you can find some directions on their web site: Text Feature Extraction.

Here is a comprehensive (bit older) summary: Machine Learning in Automated Text Categorization by Fabrizio Sebastiani


Neural Networks : Can I use both sigmoid and tanh as activation functions?

Yes you can. There are no hard rules against having different activation functions in any layer, and combining these two types should give no numerical difficulties.

In fact it can be a good choice to have tanh in hidden layers and sigmoid on the last layer, if your goal is to predict membership of a single class or non-exclusive multiple class probabilities. The sigmoid output lends itself well to predicting an independent probability (using e.g. a logloss (aka cross-entropy) objective function).

Whether or not it is better than using sigmoid on all layers will depend on other features of your network, the data and the problem you are trying solve. Usually the best way to find out which is better - at least in terms of accuracy - is to try out some variations and see which scores best on a cross-validation data set. In my experience, there often is a small difference between using tanh or sigmoid in the hidden layers.


How to deal with TypeError: ufunc 'isnan' not supported for the input types

There may be column of type object in your dataFrame. Remove it or convert it into float,int etc. if possible

From Documentation of Seaborn:

data : rectangular dataset

2D dataset that can be coerced into an ndarray. If a Pandas DataFrame is provided, the index/column information will be used to label the columns and rows.

so you can try

sns.heatmap(features.drop(['columnName01_OfTypeObject','columnName02_OfTypeObject'],axis=1), annot=True, annot_kws={"size": 7})

This will drop your columns temporarily.


Is an interesting question.

You are solving a optimization problem

$max$
$AUC(\alpha_1*X_1 + \alpha_2*X_2,Y)$

$s.t.$
$\sum^2_{i=1}\alpha = 1$

You are maximizing the AUC using the models 1 and 2 (which are $X_1$ and $X_2$) subject to the sum of weights is 1.

You may any optimization algorithm, even Excel's solver to do this.


In Keras, the first argument in LSTM gives the dimensionality of the cell state.


What does high variance mean in a binary classification machine learning model?

You have correctly intuited that variance isn't as useful a concept in this case. Statisticians typically look at the binomial deviance instead (see here for a thorough technical development).

If you do want to think about variance, we can recognize that many binary classifiers output an estimate of the label $\hat{Y}$. It helps here to think of a randomized classifier which estimates $P(Y=1|X=x) = E[Y|X=x]$ (call this estimate $\hat{p}(x)$) and generates $\hat{Y} = 1$ with probability $\hat{p}(x)$, $\hat{Y} = 0$ otherwise. We have $$var(\hat{Y}|X=x) = E[(\hat{Y} - E[\hat{Y}|X=x])^2|X=x].$$ This will be smaller when the classifier is more certain about the classification once it sees the covariates. I.e., if $\hat{p}$ is always either 1 or 0, then the variance is 0 (but of course the bias will be huge). If there are equal numbers of each class in the training data and the classifier is able to learn nothing from the covariates, then $\hat{p} = \frac{1}{2}$ for any $x$ and so the variance is $\frac{1}{4}$--a coin flip, as we'd expect.


Use deep reinforcement learning with recursive actions?

Your description doesn't seem so much like a "recurrent action" as a "failed to progress" situation.

The simplest ways to deal with possibility of a blocked/failed action are:

Leave state and time step unchanged and pick next best action, without any new policy evaluations. This is effectively the approach used by many turn-based game playing RL, such as AlphaGo, although the rejection of non-valid actions is done using the game engine and not technically tried repeatedly in the environment, that does not make a big difference in all cases.

Or

Update the time step, receive a reward (maybe zero) and try again on next time step. The state information might usefully include blocked actions (or they can be applied programmatically as in the first option). You would typically use this approach if there was a real cost to attempting the action (i.e. a negative reward for losing time) and/or it is possible from the state information to predict which actions could be blocked, and you want to include these factors in the agent's behaviour.

The second option is the approach you might use in a maze solver, if it could only sense walls by bumping in to them.

If the blocked actions are completely unpredictable from the agent's observations, then it doesn't matter too much which approach you use. However, if there is a cost for failing to act, then the second approach would give you more accurate action values in case you are using a value-based approach or need to assess the performance of your agent numerically.


Predict datetime variable with orange canvas

It appears those numbers are Unix timestamps. The numbers you quote correspond to the following human-readable dates in ISO format:

Epoch date  Human readable date (GMT)
1450396800  2015-12-18 00:00:00
1438905600  2015-08-07 00:00:00
1438732800  2015-08-05 00:00:00

You can use an online epoch converter to confirm.

Apparently, Predictions widget treats datetime variables as continuous variables (which is ok) but fails to present them nicely. You should probably submit a bug report about it.


Condense matrix of values into one column?

You can use quite easily using the dplyr and tidyr packages as follows:
df %>%
    # convert the columns to rows
    gather(key="response", value="value", R1, R2, R3, R4, R5, R6, R7, R8, R9, R10) %>%
    # select only rows where the reponse corresponds to the question
    filter(qnum == substr(response, 2, numchar(response)))


Changing Timestamp format for Date-Time in Excel/Pandas/Python?

Using python's pandas library
strftime() helps obtain specific format
to_datetime() of pandas helps convert string to datetime object

import pandas as pd
dt = "2019-06-10T14:05:00+05:30"
dt = pd.to_datetime(dt, errors='ignore')
transformed_date = dt.strftime("%Y-%m-%d %H:%M")
print(transformed_date)
print(type(transformed_date))
dateType = pd.to_datetime(transformed_date)
print(type(dateType))