Is there a known convolutional net architecture to calculate object masks for images?

I'm surprised nobody mention fully convolutional neural networks (FCNs) for semantic segmentation.

They are inspired by the original AlexNet style convnets that ended with one or two densely connected layers and softmax classifier. But FCNs dispense with the dense layers and stay fully convolutional all the way to the end.

Here's the basic principle. Take AlexNet, or VGG or something like that. But instead of using the parameters in the classifier to compute a scalar for each category, use them to compute a whole array (i.e. image) using a  1x1xNUM_CATEGORIES convolution. The output will be NUM_CATEGORIES feature maps, each representing a coarse-grained "heat map" for that category. A map of dogness, a map of catness. It can be sharpened by including information from earlier layers with "skip connections".

EDIT: Just one further bit of good news: the authors of that paper provide implementations of their nets in Caffe's Model Zoo. Tweak away!


Is there any ongoing research on known patterns ML algorithms can not mine?

The closest I can think of is research into "adversarial" transformations of data (usually images) that cause classifiers (usually CNNs) to fail. Here's a blog post which discusses a bit about how these transformations are constructed, some approaches that have been tried to address them, and what some of the consequences of their existence might be.

You might also be interested to read about the No Free Lunch Theorem.


PCA is a matrix transformation from your original dataset to a set of orthogonal features. The transformation matrix which is applied to the training set is maintained and used in the future with your testing data such that the original testing set features will be mapped to same space as the training set transformed by PCA.

If the training set has $n$ instances and their are $m$ features, the training matrix is of size $n \times m$. The PCA transformation matrix is of dimensions $m \times k$, where $k$ is the number of retained PCA features, the top eigenvalues. Thus we can transform a single instance $1 \times m$, by the $m \times k$ transformation matrix. This results in a $1 \times k$ vector.

I have some text files that I vectorized using bag-of-words. The training set is shown below on the left side and the testing set is on the right side. Each row is a text file and the columns is the word count.

If we plot the first 2 features of this dataset we get

Now we will fit our PCA transformation matrix and we will apply this transformation to both the training and testing set.

from sklearn.decomposition import PCA
pca = PCA(n_components=2, copy=True)
pca.fit(X_train)
train_PCA = pca.transform(X_train)
test_PCA = pca.transform(X_test)

This gives the following plot. The purple and yellow points are the 2 different classes from the training set. Then the light blue points are from the testing set. You can see that the points in the testing set after being transformed by PCA will line up alongside the training set.


How to write out the definition of the value function for continous action and state space

this can be written as the integral, is this correct?

Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(\boldsymbol{s},\boldsymbol{a})$. An stochastic reward model would be $p(\boldsymbol{s}', r|\boldsymbol{s},\boldsymbol{a})$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)

  Are my integral versions correct?

Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial:
$$f(n) = nf(n-1);f(0)=1$$
Which is unfolded as:
$$f(n) = n [(n-1) [(n-2)[...]]]$$
However, the difference is that the index in Bellman equation  is going forward since current value depends on future values not the previous ones.


Proof that Gini Impurity in a Decision Tree is Monotone Decreasing?

I think I've got a proof.
Any node can be identified with the vector (p_1,...,p_k) that gives the proportion of points with each classification. The Gini impurity function can then be viewed as a function from R^k to R. The weighted average of the proportions of points in S_left and S_right belonging to a certain class is equal to the proportion of points in S belonging to that class. Thus the inequality is just stating that the Gini impurity function is concave. But that is true; its matrix of second derivatives is just a diagonal matrix whose diagonal entries are all -1.


What are key dataset requirements for topic models and word embeddings?

Similarity of documents can be done with varied approaches.

As your documents are based on domain based words, you could employ a tfidf representation for each document and compute similarity based on this.

Previously, I have used word2vec representations of words and constructed document vectors by taking an average of all the vectors ( as one of the approaches ) and did a cosine similarity between these vectors. Basically an n*n similarity computation.

When you do topic modelling, even though the idea is to find documents which have similar topic distribution, there is a prior to the approach i.e identifying number of topics. Just using randomly 100 topics without looking at the distribution of the words might lead you on a wrong path.

A very interesting approach I have recently come across is combining topic modelling and word vectors, here is the link to a blog by stitchfix : http://multithreaded.stitchfix.com/blog/2016/05/27/lda2vec/


Why use sum and not average for sentiment analysis?

Consider the following two sentences:

My awesome girlfriend bought me a delicious popsicle at the store.
0--1-------0----------0------0--0-1---------0--------0--0---0-----:2:11

My awesome girlfriend, Joyce, drove to the grocery store to buy me a delicious Dole popsicle.
0---1------0-----------0------0-----0--0---0-------0-----0--0---0--0-1---------0----0-------:2:16

See how you can sum the values and get 2 for both, but average the sentiments and get 2/11 vs. 2/16.  There are cases where people have seen better results measuring the total number of good sentiment words rather than letting zero value words cloud the picture.  In this case, I would argue that the sentences have the same sentiment and averaging artificially pulls down the sentiment of the second sentence.  In certain methods like TFIDF, the word vectors are normalized against are relevant corpus, so this may change things.  In other methods, the stop words may have already been removed, so this has less effect.

In twitter examples, users are limited to 140 characters, so one might consider this to be an inherent denominator which makes averaging really averages of averages.  Its very dependent on the specific application and methodology.

Without a more specific reference, it is difficult to be more specific in my answer.

Hope this helps!


How to treat missing data for survival analysis

There are several different approaches here.

One (which you've already described) could be to impute missing values with their mean. You could then add an extra column which keeps track of whether that value was originally missing or not. So, in the example you provide, we would end up with

id  age sex dropout s1_q1 s1_q2 s1_q3 s1_q4 s1_q5.... s5_q10 s5_q10_missing
217 50  m   0       2     3     3     3     2         3      0
202 58  f   0       4     9     10    10    10        6.25   1
222 72  m   1       3     8     9     10    9         6.25   1
207 50  m   0       2     7     6     7     7         6      0
277 55  f   0       2     4     5     5     5         6      0
281 62  m   0       4     10    10    10    10        10     0

Another could be to get rid of rows that have missing values, if you have enough data (which in your case you probably don't).

A slightly more involved approach would be to predict the missing values using the non-missing data. So you could build a predictive model to predict s5_q10, train it on the non-missing data, and then use that model to predict s5_q10 for the rows where it's missing.


Dataset with features as SNOWMED codes and outputs as names of medications prescribed.What are some ML algorithms that suit this case?

In that case, Multi Class Classification will be the best approach. You can use Neural Network approach or K-Nearest Neighbours for this problem. Your DataFrame is going to be really sparse if you take a numeric approach.

From my point of view, the best approach is to build a vector model.

After you arrange your data this way, it's easier to build a topic model using NLP techniques. It can also tell you which illnesses are most similar and which codes are similar.

This is what I could think of. These sort of questions are very open ended and can often lead to creative approaches. I am sure there'll be something better if you think hard enough.


No expert here, but predicting missing value using existing variables seems suspicious to me. Does it really work?

Say $v_5$ is missing and you want to impute its value based the other variables available, using some arbitrary function $g$:
$$v_5 = g(v_1, v_2, v_3, v_4)$$
and the target variable T is:
$$T = f(v_1, v_2, v_3, v_4, v_5)$$
which is essentially
$$T = f(v_1, v_2, v_3, v_4, g(v_1, v_2, v_3, v_4))$$
or
$$T = h(v_1, v_2, v_3, v_4)$$
So essentially you are estimating $T$ using available information regardless of the missing value. Why bother to impute $v_5$ using arbitrary function?

I tend to think that imputation only makes sense only if you have a good reason for the value imputed -- often they come from domain knowledge, which means that you need to impose an external restriction on the functional and parametric form of $g$ based on your prior.


What features are generally used from Parse trees in classification process in NLP?

By using a parse tree, you divide your sentence into parts. Suppose, in the example of sentiment analysis, you can use those parts to assign a positive/negative sentiment to each part and then take the cumulative effect of those parts.

This image will help you understand more. The first half has a negative sentiment(mainly because of the word "dry") but because of the word "but" and the usage of the word "enjoyed", the negative sentiment is turned into a positive sentiment.

As for using them, you can simply generate a word vector representation of the individual words in the sentence and use neurons in place of the parent nodes. Each neuron should be connected to another neuron through weights. All the leaf nodes will be the word vector representations of words of the sentence. The top parent neuron(in this case the top blue + symbol) should generate a positive/negative sentiment according to the sentence. This tree structure can be trained in a supervised manner.

Read this paper for a more through understanding.

Image credits: cs224.stanford.edu


Understanding Loss function and Learning Algorithm

Learning Algorithm like SGD or Adam don't have any Loss/cost function associated with them. First You have to build a Model. Then pass the output of the model and target (Your labelled output) to the cost function.Then the cost function will compare the differences in output of target and model and also and calculate the gradient of loss function with respect to parameters.Then it is the optimizer like SGD that will use the gradient from previous step along with a learning rate to update the parameters of model. Cost function and Optimizer are two distince seperate things. You can basically use any cost function (even of your own) in combination with any optimizer.
It is easy to visualize like this:
MODEL ----> LOSS FUNCTION ----> OPTIMIZER


Obtaining a confidence interval for the prediction of a linear regression

You can estimate the standard deviation of your prediction:

stdev = np.sqrt(sum((linreg.predict(X_train) - y_train)**2) / (len(y_train) - 2))

Then, for any significance level you want, you should check correspondent Gaussian critical value (for example, for significance level 95% it is 1.96).

Finally, confidence intervals are (prediction - 1.96*stdev, prediction + 1.96*stdev) (or similarly for any other confidence level).

Another approach is to use statsmodels package.


Does importance of SVM parameters vary for subsample of data?

In theory, if you have a large enough random sample of your data set, it should be representative of the characteristics in the larger population of data that will affect the relationship between parameter values and performance. It seems to me that a $5\%$ sample of your data might be too small for what you're trying to do. When I'm performing cross-validation on a test/dev data set, I usually set aside somewhere in the neighborhood of $10\%$ for the hold-out test/evaluation data, and use $90\%$ for my development work, but, as you pointed out, this may be too big given the computational constraints you're up against. Depending on the type of application you're working on, you could look at feature distribution in your sampled data and compare it to that in the larger population. If the two sets are comparable, then you might be ok to do your dev. work on the smaller data set. I say it depends on the type of application you're developing, because many statisticians would consider this cheating, in that it's generally not good model evaluation practice to look at hold-out data at all. Here's the procedure I'd recommend:

Separate out $10\%$ of your data for hold-out evaluation.
Divide the remaining $90\%$ into $5\%-10\%$ parameter optimization, and the remainder for final parameter eval.
Compare the feature distributions of the parameter optimization and final parameter eval data sets, if they're not comparable, redraw the samples (within the context of the $90\%$ development sample)
When you have a good subsample, run your parameter optimization experiments on the small parameter optimization data set.
With your final parameter settings, evaluate with cross-validation on the the $90\%$ combined parameter optimization and final parameter eval data sets.
Perform final model analysis by training on the $90\%$ data set and classifying on the $10\%$ hold-out evaluation set.

It might also be worth looking into optimizing your analytics pipeline. For example, is file I/O, feature generation and feature extraction part of the the workflow you're using, or have you done all of that off-line and are just concerned with the SVM evaluation part?


What the good general regression technqiue for a problem with 50 independent varaibles

In no way is this going to be an exhaustive answer, but it will definitely give you a starting point in Python -

Data Exploration

Start with Pandas Profiling. It will give you HTML reports of your variables. If the quality of the data is good, it will provide some insights into the fill rate, depending upon the variable type some statistics for each variable

Correlational matrix

The pandas profiling report includes the coorelation matrix. But if you are looking to compute by hand, use pd.corr(). You can vary parameters to get different correlation metrics like ‘pearson’, ‘kendall’, ‘spearman’

Dimension Reduction -> PCA (Dimension reduction)

There are many ways to do this. Keep in mind if you are looking for accuracies only and don't care about how X is influencing y, (1) is an optional step (applies to (2) as well).

Analyse the correlation matrix and use VIF to dump variables with high correlation
Factor Analysis / PCA for dimensionality reduction
Use LASSO to fit a model, check the coefficients and the ones that are 0 or going to 0 can be thought of as weak indicators and can be eliminated.
Keep all 50, and use Ridge Regression and vary the alpha parameter to fine-tune accuracy (or whatever metric you are trying to optimize)
If the model still doesn't seem to be stable, try to cook non-linear features with sklearn's Polynomial Features, regularize and repeat.
Probably the most important in the real world, ask the domain expert on what he/she thinks might be the important variables

Basic Linear Regression technique

Playing with hyperparamters to get good cross-validation/test score is the key here for a basic Linear Regression model.
Try as many techniques as you can from here and here


How to get summary statistics in Orange?

I found 3 ways to do it :

you have to use the Data Info widget who gives the number of rows of your dataset and the Box Plot widget who prints the average and the standard deviation of each feature in your dataset. Pro: easy to use. Con: you are unable to use this data.
you can create a Python script who will compute the average and the standard deviation using the statistics package. Pro: you can reuse your data. Con: you have to write a script and format the output data correctly.
it exists an add-on call "Orange3-Timeseries" with a Moving Transform widget who can compute the average and the standard deviation of your series. Pro: easy to use. Cons: the package is still a beta-version and have some bugs, the widget only works with timeseries.


Value of features is zero in Decision tree Classifier

I'm pretty sure that your feature importances are 0 because your classifier isn't doing any classifying. From the code, it looks like you're training only on positive examples, and giving the fit function a label vector that consists entirely of 1s. The classifier has no information; the decision rule is just "when given an example, predict 1".

There's no way to measure which features are most strongly associated with the label because they're all equally associated - there's only one label, so there's no way to associate the features with anything else.

Is there a reason you're not using the negative examples?  It seems like you have the dataframe available.

When you ran naive bayes and logistic regression, did you also give those models only the positive examples?


NCF Recommender- The target encoded within the model input, why doesn't it overfit easily?

The input of the system is a given user and a given item (each respectively one-hot encoded). The output of the system is binary (picked or not).

The authors filtered the data to require each user to have picked at least 20 different items. Thus, enabling the system to learn the general preferences of a user.

The system could overfit. The system could memorize that users would only pick items they have already picked.

The system would fail on the chosen evaluation protocol - held-out last interaction. The model performs better than other recommendation systems on this evaluation protocol which is evidence that this system is not overfitting.


How should I configure my Neural Network to accept a column of numpy.ndarrays as input?

You should set the input_shape parameter in the first dense layer. Like this:
model = Sequential()
model.add(Dense(12, activation='linear', input_shape=(33222,)))
model.add(Dense(1, activation='linear'))


How to continue incremental learning when a categorical variable has been assigned additional category labels?

Looks like a special case for incremental learning / life-long learning. There are many papers in this line of work:

https://arxiv.org/abs/1611.07725

https://openreview.net/forum?id=BkloRs0qK7

Most try to preserve the model parameters in a way that reduces the loss of previous knowledge ("catastrophic forgetting"), but some also try to find (or keep) some samples and preserve the original model's response on those samples.


Remove part of the string in R - dropping some words from it

I wouldn't use them as time at first place. What would I do:

String <- "Sat Jan 01 15:30:27 CET 2011"

As your dates have same number of elements in them we could just count what part of the String we would like to get.

date <- substr(String, start = 5, stop = 10)

year <- substr(String, start =25 , stop = 28)

Attach them togheter

final <- paste(date,year, sep = " ")

as.Date(final, format = "%b %d %Y")

Little bit more information about as.Date
https://www.statmethods.net/input/dates.html
I hope it helps you :)


Best Programming Language for Data Science

This is no doubt a duplicate, but here's how I'd weigh in on the major languages:

R:

Fantastic support for packages and specialised stats analysis
community, you can find a package to do just about anything you need
and it will be relatively easy to use.
Is good for throwing together prototypes and performing exploratory analysis.
Is Free and Open Source.
Slower than Python. Basically don't loop over anything. It's an odd language for a programmer to use (coming from a software dev
background). Clearly designed by mathematicians.
Relatively little choice of good IDEs

Python:

Fast.
Also very good as a general purpose language so has 'broader' package support. Free and Open Source.
Easy to use for Big Data applications.
Not as streamlined for analysis as R.
Syntax can be difficult to read (no surrounding braces to make it obvious where functions/ if statements end).
Can be particularly tedious working with Dataframes compared to R.

MATLAB:

Generally slower.
Has very impressive packages for signal processing/image recognition and all the cool stuff.
Is very readable and easy to comprehend generally.
Is NOT free. Student licenses are available. Was quite complicated for me to get my hands on one though...
Has very good support for mathematical analysis similar to R, but much better matrix functions.

Personal recommendation: Python. Kill two birds with one stone, learn good general to advanced programming concepts and data science at the same time.

Good article: https://www.linkedin.com/pulse/r-vs-python-matlab-octave-julia-who-winner-siva-prasad-katru


How does SelectKBest() perform feature selection?

Your question is really more about f_classif than SelectKBest. It's to drop duplicate labels; note the np.unique(y):

X, y = check_X_y(X, y, ['csr', 'csc', 'coo'])
args = [X[safe_mask(X, y == k)] for k in np.unique(y)]
return f_oneway(*args)

f_oneway still only gets passed the feature matrix, but a subset of it.


How to find the most important attribute for each class

If you must split the dataset for each class then I suggest you try PCA. Principle Component Analysis is basically used for dimensionality reduction in the sense that it gives you the subset of attributes which represent the distribution of data best. You could use that for all classes and in that way get the attributes which affect the class data distribution the best.


Random Forrest Sklearn gives different accuracy for different target label encoding with same input features

Yes. With y being a 1d array of integers (as after LabelEncoder), sklearn treats it as a multiclass classification problem. With y being a 2d binary array (as after LabelBinarizer), sklearn treats it as a multilabel problem.

Presumably, the multilabel model is predicting no labels for some of the rows.  (With your actual data not being multilabel, the sum of probabilities across all classes from the model will probably still be 1, so the model will never predict more than one class.  And if always exactly one class gets predicted, the accuracy score for the multiclass and multilabel models should be the same.)


What is the difference in xgboost binary:logistic and reg:logistic

There is no difference in the algorithm, they are the same. When you use  reg:logistic the default evaluation metric is rmse, but you can change this parameter.

The error rate and the rmse may differ depending on the distribution of your output, as the error rate uses a limit of $0.5$ if you have the output values concentrated in $0$ or $1$ it will be much smaller than rmse, even though its correlated metric the model can be very different, the application will depend on your problem.

To evaluate performance of logistic regression models is usual to calculate auc, logloss, accuracy, pseudo-$R^2$ and some others.


How to visualize change in a distribution with a few outliers that account for a very large percent of the total?

To visualize change in the size of multiple entities that are contributing to a total through time, e.g. total (t) = book_1 (t) + book_2 (t) + ..., we can use Stacked Area Plot. This plot can be used for normalized and un-normalized (absolute) values.

Preprocessing

For large number of entities, to avoid cognitive load, we can keep only
those entities that become significant (whale) at some point in the plot, and group all the others under an "ordinary" entity. This way, cognitive load is minimized and only those entities that matter at some point are distinguished. For example, distinguishing books that have more than 10% of traffic at some point in the plotted time span.
If total fluctuation is very high, logarithm of values can be plugged into the plot.


Comparing English word pronunciation complexity

I think first you need to define "pronunciation complexity." Is it a measure of the amount of time it takes the average L2 learner to correctly pronounce the word? Or, a measure of the diversity of phonemes in the word. Operationalizing your variable will help immensely in clearing up your goal.

Suppose you want to measure complexity by the diversity of phonemes in the word. You could then find a dataset of phonemes for each English (or other language) word. Then you could do something like, calculate the entropy of each phoneme set per word. Or maybe the Shannon diversity (although this might not perform well for sets with very low specie occurrences). You could also vectorize each word and perform some measure of similarity between words.

Of course, this is all hypothetical. Once you update your question with a clear goal in mind it will be easier to help you more concretely!


How can we show that one model might have higher accuracy than another model but at the same time lower AUC?

Normally this can happen when there's imbalance in your classes. Imagine that you want to predict these 5 values:

y_test = [1,0,0,0,0]

and your models M1 and M2 predict the following:

M1_pred = [0,0,0,0,0]
M2_pred = [1,0,0,1,1]

Clearly we can see that M1 is biased towards class 0, getting 4/5 results correct (80% accuracy!). While the second one, only gets 3/5 values right (60% accuracy).

However, because AUC ROC is defined by both true positive rate against the false positive, combining those the M2 classifier gets a 75% auc, while M1 only a 50% because it has failed the only one positive, thus penalising the final score.

Try running this toy example:

from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_curve, auc
import matplotlib.pyplot as plt

y_test = [1,0,0,0,0] #ground truth

M1_pred = [0,0,0,0,0] #predictions M1
M2_pred = [1,0,0,1,1] #predictions M2

#Calculate both accuracy scores
M1_acc = accuracy_score(y_test, M1_pred)
M2_acc = accuracy_score(y_test, M2_pred)

#Calculate both ROC AUC
fpr1, tpr1, _ = roc_curve(y_test, M1_pred)
roc_auc1 = auc(fpr1, tpr1)
fpr2, tpr2, _ = roc_curve(y_test, M2_pred)
roc_auc2 = auc(fpr2, tpr2)

#Plot both graphs
plt.figure(figsize = (12,6))

#Graph M1
plt.subplot(121)
lw = 2
plt.plot(fpr1, tpr1, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)\n ACC Score = %0.2f' % (roc_auc1, M1_acc))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic ROC')
plt.legend(loc="lower right")

#Graph M2
plt.subplot(122)
plt.plot(fpr2, tpr2, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)\n ACC Score = %0.2f' % (roc_auc2, M2_acc))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic ROC')
plt.legend(loc="lower right")

plt.show()

and you will get something like this:


statsmodels ols does not include all categorical values

When a logistic model is built using a categorical variable with N levels, it only considers N-1 levels, as the remaining level is used as a reference by the model.

What this means to your model as a whole is that, each level (when remaining variables remain same) is compared to the reference level.

In your example, "community" level is used as the reference. So, ending_addiction contributes 0.0749 units more than "community" to the dependent variable.

Hit this up if it's not clear -->
https://community.alteryx.com/t5/Alteryx-Designer-Discussions/In-the-Logistic-Regression-Report-Factor-Missing/m-p/10100/highlight/true#M5169


Using Random Forest Regression correctly

It seems good. (Understanding that you are not getting any error)

You are fitting your model in some data, then evaluating on other (basic validation).

Now you have to find data that have the same format and similar distribution and predict there.

You can have a look at the random forest documentation to see what else you can do

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html


Do I classify all types if they are mutually exclusive

You won't be able to train your model correctly if you set only two classes. Because for all the samples the model will only predict Up or Down. This will cause even the Middle classes to be predicted as Up or Down, which will reduce your model's accuracy. Hence you need to keep all three classes as labels.


sklearn : missing pruning for decision trees

This is something which is planned to be done. Setting the minimum number of samples required at a leaf node or a split as well as setting the maximum depth of the tree are how you want to work around this.


How to identify a field as holding personal identifiable information from the name of the field itself using ML model in python?

In such cases, where the data is not availible, it will be better to make the dataset on our own. Just create a google form and sent to a few friends and family and hence you will end up with a dataset. Apart from this it you can check this article on anonymizing information on kaggle


Having trouble installing and loading tidyverse- No DIB package

It sounds like you do not have a dependency installed which tidyverse needs to run correctly.

Usually dependencies install with the package, but at times you need to coerce them to do so, so I usually install with the coercion for all packages.

install.packages("tidyverse", dependencies=TRUE)

In this case, if that was the only error, or you installed in this manner and did not get DBI, then I would just install it directly:

install.packages("DBI", dependencies=TRUE)

Once you have done that, if you are still having problems, explicitly load DBI:

require(DBI) or library(DBI)

and you should be good to go.

If you have a problem still, comment and I will try to help you chase it down.


Given a 12x12 binary image (only black and white pixels) what is its dimensionality? And how can I define dimensionality of a data space?

There is some ambiguity in dimensionality you ask for. The vector space that your input lies in is $12\times12 = 144$ dimensional. So, if you're going to apply some kind of dimensionality reduction algorithm, e.g. PCA, this is the dimension. But, we sometimes tend to refer to dimension as the shape of the tensor, which is $2$ in this case. Some libraries might read the image into $12\times12\times3$ tensors, in which the third entry indicates RGB decomposition, even if the image is black and white. In this case, the tensor dimension is $3$.


How to create clusters of position data?

I doubt any of the clustering algorithms will work well.

Instead, you should look into:

segmentation (yes, this is something different), specifically time series segmentation
change detection (as you said, there is a rather constant distribution first, then a change, then a rather constant distribution again
segment-wise regression may also work: try to find the best fit that is constant, linearly changing, and constant again. It's essentially four parameters to optimize in this restricted model: average before and after + beginning and end of transition.


what is the loss function in char recognition using Tensorflow?

The loss function is correct, you just need to convert categorical variables into numerical representations using one-hot vector encoding. Please take a look at this.


What is the use of function build in custom layers in tensorflow keras?

build is called by the __call__ function which is implemented in the parent Layer class.
From the TF docs:
https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer
"build(self, input_shape): This method can be used to create weights that depend on the shape(s) of the input(s), using add_weight(). __call__() will automatically build the layer (if it has not been built yet) by calling build()."


How to justify the usage of 200 dimensions in word vectors instead of the 300 dimensions?

Dimensionality around word vectors is already answered here : https://stackoverflow.com/questions/45394949/what-is-dimensionality-in-word-embeddings

The reasons 200-300 dimensions are chosen normally, is that they have seen it produce the results that are very close to or equivalent to when they have chosen higher dimensions.

When it comes to training time, the time taken to generate word embeddings for 200-300 dimensions are not significantly different for most of the experiments I have personally come across.


Error after merging two Deep Learning models VGG16 and ResNet50

The 2D feature map is extended to the output layer and hence throwing errors at the Softmax Layer.
 Add Flatten Or GlobalAveragePool after Concatenate.
mergedOutput = keras.layers.Concatenate()([vgg.output, resnet.output])

mergedOutput = keras.layers.GlobalAveragePooling2D()(mergedOutput)  # This line

x = keras.layers.Dense(256, activation="relu")(mergedOutput)


Have you seen this paper?
Optimizing Search Engines using Clickthrough Data

I stumbled upon this the other day, and I'm still reading through it, but the author attempts to deal with the problem you describe. You may also find Improving Web Search Ranking by Incorporating User Behavior Information useful.


Why can't ARIMA model large lags and/or long range dependence?

An ARIMA model with order of differencing d equal to 1 can model a time series with a unit root, which is a form of long-range dependence. If d=0 one has an ARMA model, which cannot model long-range dependence unless the number or AR or MA parameters is very high. To model long range dependence in a parsimonious manner, look at fractionally integrated ARMA models, known as ARFIMA models.


What do you do with one hot encoding items that are a non-match for all classes in a confusion matrix?

It depends on the design of your task, there are two options:

The task is regular multiclass classification, i.e. every instance must belong to exactly one class. In this case it would be a mistake to one-hot-encode the class, it can simply be encoded as an int (for example with LabelEncoder). The model will always predict exactly one class for one instance so the case of zero class is impossible.
The task is multi-label clasification, i.e. every instance can belong to zero, one or multiple classes. In this case an instance can be predicted as belonging to no class at all, this is normal. In this setting the confusion matrix should not be done with a $n \times n$ matrix across classes, because the classes are independent (btw it's not only about the case of zero class, the case of multiple classes would also be impossible to represent this way). Instead there should be one binary confusion matrix for every independent class.


How to collect info about unseen bugs given user's comments/feedbacks?

It sounds like you are looking for an unsupervised learning approach (meaning you don't need to manually label your data).
Something like k-means clustering could work well. This would allow you to group you comments into k distinct clusters. You could then view counts of comments in those clusters and explore the clusters to determine their meaning.
In order to perform the clustering, you need to transform your data from text to a numerical vector space. A common approach would be tf-idf, but you may find that something else works better.
Since you mentioned Python, both k-means and tf-idf can be accomplished using sklearn:

sklearn.cluster.KMeans
sklearn.feature_extraction.text.TfidfVectorizer

There's a pretty nice example of k-means clustering using tf-idf on Kaggle.


How do we make data Obfuscate or "De-identificate" to make it anonymous and share it publicly?

I have had a couple of talks about this once and I think you should know about three different concepts.

Different attributes

Attributes can be divided into three different categories. Hard identifiers refer to a specific person, e.g. full name or passport number. A combination of soft identifiers can also be used to identify a single person, because the combination is unique. Sensitive attributes are the ones that infringe privacy and that should be protected.

Measuring privacy

There are several measures for how anonymous your dataset is. These measures help you to focus on the attributes that need to be altered. In order of increasingly stronger requirements you have:

A dataset is $k$-anonymous if every combination of soft identifiers occurs at least $k$ times.
A dataset is $i$-diverse if for every combination of soft identifiers there are at least $i$ different values of the sensitive attributes.
A dataset is $t$-closed if for every combination of soft identifiers, the distribution of sensitive attributes is not more than $t$ (i.e. KL-divergence) different from the global distribution of sensitive attributes.

Privacy protecting modifications

Hard identifiers can be treated with masking or pseudomization. Masking refers to deleting the hard identifier. A less drastic approach is  pseudomization where each records hard identifiers gets a pseudonym (e.g. hash) that is only known by the dataset creator.

Soft identifiers can be treated with suppression, generalisation or randomisation. Suppressing values of soft-identifiers refers to replacing it by a default value (e.g. '*' or the mode). Another strategy, called generalisation, is to create groups of variables that are alike (e.g. 'age 10-20' instead of age 19). A third way is to add randomness to quantitative variables by adding zero-mean noise.

Sensitive attributes can be treated with permutation or randomisation. Permutation refers to permuting the sensitive attributes within every combination of soft identifiers (within a dataset that is at least $2$-anonymous). Like soft identifiers, quantitative sensitive attributes can also be randomised by adding noise.

Ps. as an R-package you could consider sdcMicro, and this guideline.


You can try RAKE(Rapid Automatic Keyword Extraction) and there is a python implementation here. RAKE is an document-oriented keyword extraction algorithm and also language-independent(theoretically, since RAKE use a generated stop word list to partition candidate keywords, and considering different languages, we need to find a better way to generated stop word list.). However, about English documents, RAKE can extract keywords(or tags) in a acceptable precision and recall. RAKE is also efficient, because to use it we don't have to training a whole corpus, RAKE can generate a keyword list by calculating the word's degree and frequency then comes up a score for every candidate keyword then pick the top N words.
Hope this answer helps you or lead you a direction for your next step investigation.


Clasification - ROC Curve with very high number of false negatives

Notice how the precision is very high and all of the other metrics are very low.  Now look at the class balance of your problem:

$$TP+FN=Actual Positive=31,245$$
$$TN+FP=Actual Negative=508$$

So your data is heavily skewed toward positives.  To have gotten a model that is producing this poorly, I think you may have provided the model with the precision as the cross validation metric.  The precision is a very bad cross validation metric in this case since it will result in poor accuracy and poor recall.  accuracy is also not a good metric as your model could classify everything as positive and get an accuracy of:

$$AC=\frac{31,245}{31,763}=.984$$

For cases like this where the classes are grossly weighted toward one value, I suggest using the F1-score as your cross validation metric.  The F1-score is the harmonic mean of precision and recall and hence balances these two factors nicely.  Wikipedia actually has a very nice explanation of classification metrics here and this paper is top notch if you even need to understand multi-class metrics and confusion matrices.

Hope this helps!


Getting uniform distribution over topics from gensim's LDA?

I solved this issue. There is a parameter for minimum probability in gensim's LDA which is set to 0.01 by default. So topics with prob. < 0.01 are pruned from output.

Once I set min. prob to a very low value the results had all topics and their corresponding probability.


How does L1 regularization make low-value features more zero than L2?

The penalty on coefficients of L1 is more aggressive on values close to zero than it is for L2.

With L1, when a weight value comes closer to zero, it tends to get even closer, because of the $\epsilon\lambda$ penalty which stays constant.

With L2, the $\epsilon\lambda w^{(t)}$ term gets smaller, so the regularization gets smaller as a weight comes closer to zero, with the update depending only on $\epsilon\Delta E(w)$.

So the main argument for this to my knowledge is that
$$
\lim_{w\rightarrow0}\lambda = \lambda\\
\lim_{w\rightarrow0}\lambda w = 0
$$


Neural Network for predict a set in a certain time stamps

Since you have a transaction data set on your hands, we are talking clear-cut supervised learning here. And while you could certainly try to use neural networks, I think you should start simpler and switch to more complex and costly algorithms once you see the simple ones failing (this is generally a good idea).

Now form what you describe, you have some sort of time-series data, with a categorical quantity that you want to predict, for which you will want some sort of modified ARMA model. You could set up a logistic regression for each good you are interested in (i.e. beer, chocolate, potato, etc.) that will predict whether or not he will buy the product. The variables you feed into the model are a set of binary observations describing if he bought the product a day ago, a week ago, four weeks ago, etc.. Play around with those lags. The nice thing about logistic regression here is that you'll see right away what is the period of the purchases. If you see a large coefficient on the one-week-ago-variable, you know that he eats avocados weekly and so forth. Note that if something is purchased weekly, it's also purchased every four weeks, so you will have some correlation issues.

If you don't get decent predictions, you could try different models, e.g. trees or ensemble methods such as random forests or boosted trees. If those don't work either, you probably have a time series without clear periodicity. In that case you could try setting up your variables differently, e.g. using the number of days since he bought the product.

One last word of advice: You might want to check correlation between purchases. Maybe he buys avocados, tomato, and onion together to make delicious guacamole. If he then has some onions leftover, he might be inclined to buy more avocado and so on.


Keras: How to normalize dataframe with continuous and categorical data?

Yes it does, you're doing well!

In most cases, categorical features(columns) should be one-hot encoded. However, continuous features might be a little complicated.

There are two common ways to preprocess continuous feature:

scaling features to range [0, 1] (as you have done)
removing the mean and scaling to unit variance(make the feature has zero mean and 1 standard variance)

In my practice, I take these two ways depending on my dataset.


Here is my current understanding: Q-value is just a value for a particular action taken, and the Value function of a state sums over the Q-value of every action possible in that state.
Would this description be accurate?

Not quite.
First the Q value is a type of value function, it is often called the action value. Both $Q(s,a)$ and the state value function $V(s)$ calculate the expected future return given their parameters, a known environment and a known policy $\pi$ that describes how an agent will select actions in that environment.
In the case of action value $Q(s,a)$, the expected future return is based on the agent taking the action $a$ in state $s$ first, and afterwards following $\pi$. Whilst the state value $V(s)$, the expected future return will depend on what action $\pi$ chooses in state $s$.
There a few ways to write the relationship between $V(s)$ and $Q(s,a)$. If the policy function is deterministic, action in state $s$ given by $\pi(s)$ then:
$$V(s) = Q(s, \pi(s))$$
If the policy function is stochastic, with probability of selecting action $a$ in state $s$ given by $\pi(a|s)$:
$$V(s) = \sum_a \pi(a|s)Q(s, a)$$
This is a sum, as you suggest, but weighted by the probability of taking each action.


TL;DR

If you have unlimited time and use a 64-bit version of Excel, you can get as far with Excel as any other data analysis tool.

Time

I mention time as my first factor, because Excel only has basic funcitonality built in, such as summing, random number generation, lookups etc. These correspond to a kind of standard library, which Python and R also have. Using these basic functions, with enough time, you can build up pretty much any analysis tool out there. Don't expect good runtime performance. In Python and R, however, there are many many packages that people have already created, which perform well and have been tested by lots of people and so are trusted.

Memory

My second point about 64-bit Excel is because that allows a lot more memory to be used by a single instance of Excel. It allows many more cells to be filled. Using 32-bit excel will limit you to projects of around 2Gb. That is a fair amount of data, but it is a hard limit.

Then steps in 64-bit Excel, which basically means no more memory limits - only those that come from your hardware, and that means Python and R will also be stopped in their tracks.

To provide some numbers, we can simply compute the number of bits able to be stored in each version. Here in Python's interactive prompt:

In [1]: (2**32) / 10**9          # 10^9 means the result is 4.3 Gb
Out[1]: 4.294967296

In [9]: (2**64) / 10**18         # 10^15 means the result is 18.4 Eb
Out[9]: 18.446744073709553

Eb means exa-bytes. This means 18.4 million million million gigabytes.

I notice the computation shows 4Gb for 32-bit, while I said 2 Gb above. I read there is a kind of hard limit on the 32-bit version. I don't know or care why that is... I use Python and R  ;-)

In any case, I hope that is enough to convince you that memory is not an issue, if you are a brave person willing to invest all your time building tools from the ground up!

Summary

If you have complicated business logic, where the actual analysis is mathematically simple, stick to Excel. Business people will love you for it.

If you want to do more than linear regression, use Python or R.

Caveats

As far as I know you cannot run remote or distributed tasks using Excel, whereas that is relatively easy using Python and (a little less so in my opinion) R. So at that point, I would give up on Excel. You'd likely have to implement your own tools in C# or C++ using the .Net framework.


How to read contents of a CSV file inside zip file using spark (python)

You need to ensure the package spark-csv is loaded; e.g., by invoking the spark-shell with the flag --packages com.databricks:spark-csv_2.11:1.4.0. After that you can use sc.textFile as you did, or sqlContext.read.format("csv").load. You might need to use csv.gz instead of just zip; I don't know, I haven't tried.


P-values are used to provide insight into the solidity or likeliness of your results within the framework of a pre-determined hypothesis. Therefore, in order to determine the p-value, first you'll need to identify what you are curious about within the scope of the problem and develop a hypothesis test that mirrors that question.

For example, let's re-frame your data such that we have a clear question that we are trying to answer:

We have a sample of individuals and are given their ages and whether or not they are considered overweight based on their weight being above or below some threshold. Given this data, I would like to know if people who are overweight tend to be of a higher age than people who are not overweight. Given this question, I can then construct the following hypothesis in which $\mu(x)$ is the population mean age of overweight individuals and $\mu(y)$ is the population mean age of non-overweight individuals:

$$H_{0}: \mu_{x} = \mu_{y}$$

$$H_{A}: \mu_{x} > \mu_{y}$$

From there, based on the distribution of your data, you will be able to find the p-value, which is the "probability of observing a result equal or more extreme in your sample, assuming that the null hypothesis is true. If we observe a very small p-value (usually smaller than 0.05), then it is unlikely that our null hypothesis was true. Conversely, with a large p-value, we would "fail to reject" the null hypothesis that the mean ages are equal.


Why my training and testing set are about 99% but my single prediction does wrong prediction?

It looks like the new data has a different distribution from the training data. It looks like the training data is just a single fruit, with white background, and the new image you've passed is a picture of bananas with blue background. The model has probably learned something like: if blue image, then blueberries, and for this reason it classifies the blue bananas picture as blueberries.
Whenever the distribution of new data is different from the data you've trained on, don't expect the model to work very well, as ML models just interpolate.


How to classify neutral sentiments using BERT

Welcome to DS.SE, @prashanth

Yes! In general, it is possible to classify the documents to more than two classes of positive and negative sentiments as long as you have such labels in your training set.

Please take a look at this for some more general information about the sentiment analysis, tools, and applications.


hyperparameter tuning with validation set

It seems to me that you're manually iterating through the hyper-parameters.

scikit-learn has a number of helper functions that make it easy to iterate through all of the parameters using various strategies:
https://scikit-learn.org/stable/modules/grid_search.html#grid-search

Secondly, if I was 'manually' tuning hyper-parameters I'd split my data into 3: train, test and validation (the names aren't important)

I'd change my hyper-parameters, train the model using the training data, test it using the test data.  I'd repeat this process until I had the 'best' parameters and then finally run it with the validation data as a sanity check (should have similar scores).

With scikit-learn's helper functions, I just split the data into two parts.  Use GridSearchCV with one part and then at the end using the best parameters (stored in the attribute best_estimator_) run a sanity check with the second part.

# define parameter sweep
param_grid = { 'max_iter' : [100, 1000, 10000] }

# define grid search
clf = GridSearchCV(mlp, param_grid, cv=5)

# perform search
clf.fit(X, y)

# best estimator
clf._best_estimator

For info, every estimator and their respective scores are also available as attributes (see https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV) which means you can check the result of every CV if you wanted to make sure you got the best parameters.

As you can see you can avoid thinking too hard about whether CV was the right approach by using a final validation set as a sanity check.


How does YOLO algorithm detect objects if the grid size is way smaller than the object in the test image?

Each grid predictor in YOLO should only have a high score that an object is within it, if it detects the centre of the bounding rectangle is inside itself. So a grid point that contains only the wing mirror of a car should decide it has a low probability of containing the centre of the car.

The predicted bounding rectangles are not constrained in the same way - YOLO can (and often does) predict bounding box dimensions from the centre that are larger than the grid cell dimensions.

Each grid point is independently able to predict whether or not it contains an object's centre, what the bounding box dimensions are for the object, and what the object class is.

If you trace the layer connectivity, you will see that the grid cells are effectively interconnected in lower output layers, so the network as a whole "sees" more of each object and can influence individual object predictions, suppressing some and encouraging others, when objects span multiple grid locations. The grid cells are not isolated into sections, or restricted to only using data from the area that they cover for the prediction. The concept of what part of each image a feature "pixel" can access from the base image in a CNN is called the "receptive field" of the network, and can be calculated based on the architecture as explained in this blog on Medium.


What you're doing right now is a traditional classification using supervised learning. This is a great method for predicting outcomes, but I suspect there are much better ways to complete this sentiment analysis project you're working on.

Without knowing what the goal of your analysis is, I would suggest you look at the NLTK package. A lot of work has been done to idenify how positive or negative a collection of words is, and you could piggyback on the work of some really smart people.

If you're having trouble starting, you can look at NLTK's how-to tutorial, some Kaggle kernels, or find some articles online.

Good luck!


Classification of skills based on job ads

I don't know about RapidMiner, but for a beginner, Orange seems an awesome open-source tool for the job.

Particularly if you want to do unsupervised learning, i.e. clustering. You'd probably want to preprocess your textual data in some bag of words model and save it as CSV, I guess.

I also hear Orange has a text mining add-on, but I haven't tried it yet.


Applying a robust scaler to column 0 to 6 of my data frame?

Check out the documentation for sklearn's columnTransformer. This allows you to apply transformations to specific column indices in your dataframe.
Note the 'passthrough' option for the transformer parameter - this will be needed for the columns that you do not wish to scale/modify.
Example taken from the documentation:
>>> import numpy as np
>>> from sklearn.compose import ColumnTransformer
>>> from sklearn.preprocessing import Normalizer
>>> ct = ColumnTransformer(
...     [("norm1", Normalizer(norm='l1'), [0, 1]),
...      ("norm2", Normalizer(norm='l1'), slice(2, 4))])
>>> X = np.array([[0., 1., 2., 2.],
...               [1., 1., 0., 1.]])
>>> # Normalizer scales each row of X to unit norm. A separate scaling
>>> # is applied for the two first and two last elements of each
>>> # row independently.
>>> ct.fit_transform(X)
array([[0. , 1. , 0.5, 0.5],
       [0.5, 0.5, 0. , 1. ]])


The kernel trick is based on some concepts: you have a dataset, e.g. two classes of 2D data, represented on a cartesian plane. It is not linearly separable, so for example a SVM could not find a line that separates the two classes. Now, what you can do it project this data into an higher dimension space, for example 3D, where it could be divided linearly by a plane.

Now, a basic concept in ML is the dot product. You often do dot products of the features of a data sample with some weights w, the parameters of your model. Instead of doing explicitly this projection of the data in 3D and then evaluating the dot product, you can find a kernel function that simplifies this job by simply doing the dot product in the projected space for you, without the need to actually compute projections and then the dot product.
This allows you to find a complex non linear boundary that is able to separate the classes in the dataset. This is a very intuitive explaination.


As @David Masip mentioned, Principal Component Analysis would be a good method to use here. Essentially PCA is a method by which a mapping is found between a high dimensional space to a lower dimenional space while keeping as much variation in the data as possible - perfect for dimensionality reduction of high dimensional data.

However, you mention that you want to use this reduced data to train a neural network model. It may be best to first train the neural net model and see how well it performs, as neural nets are usually very good at identifying interactions between features as well as other hidden structures in the data. If it doesn't perform well, then one approach to improve performance may be to use PCA - although this is highly dependent on your use case, content / type / amount of data, neural network architecture etc.

p.s PCA is also good to visualize high dimensional data (reduce the dimensionality to 2 or 3 dimensions, then plot it. This is better than plotting only 2 features at a time as you have done above).


How can I fix regression model interpretation of feature?

Unless there's a mistake in your code, or the coefficient on $X_1$ is not significant, I'd be inclined to trust the model output.

It's not unusual for data to behave this way. Just because $X_1$ and $Y$ are inversely related with respect to the marginal distribution of $(X_1, Y)$, as can be concluded from a scatterplot of the two variables, does not mean this relationship holds conditional on other variables.

Here is an example where $(X_1, Y)$ are inversely related, but are positively related conditional on another value, $X_2$. (The example is generated using R -- you've tagged python, but this concept is language-agnostic):

library(tidyverse)
library(broom)
set.seed(1)
N <- 100
dat <- tibble(
    x2 = sample(1:4, size = N, replace = TRUE),
    x1 = x2 + rnorm(N) / 3,
    y = x1 - 2 * x2 + rnorm(N) / 5
)
ggplot(dat, aes(x1, y)) +
    geom_point(aes(colour = factor(x2))) +
    theme_bw() +
    scale_colour_discrete("x2")

Here are the outputs of a linear regression model. You'll notice that the coefficient on $X_1$ is negative when $X_2$ is not involved, as anticipated, but is positive when $X_2$ is involved. That's because the interpretation of a regression coefficient is the relationship given the other covariates.

lm(y ~ x1, data = dat) %>%
    tidy()
#> # A tibble: 2 x 5
#>   term        estimate std.error statistic  p.value
#>   <chr>          <dbl>     <dbl>     <dbl>    <dbl>
#> 1 (Intercept)   -0.492    0.154      -3.20 1.83e- 3
#> 2 x1            -0.809    0.0549    -14.7  1.33e-26
lm(y ~ x1 + x2, data = dat) %>%
    tidy()
#> # A tibble: 3 x 5
#>   term        estimate std.error statistic  p.value
#>   <chr>          <dbl>     <dbl>     <dbl>    <dbl>
#> 1 (Intercept)   0.0189    0.0540     0.349 7.28e- 1
#> 2 x1            1.04      0.0681    15.3   1.42e-27
#> 3 x2           -2.05      0.0726   -28.2   1.60e-48

Created on 2020-04-27 by the reprex package (v0.3.0)

This concept extends to more than two covariates, as well as continuous covariates.


how to find parameters used in decision tree algorithm

Just type clf after defining it; in your case it gives:

clf
# result:

DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',
                       max_depth=None, max_features=None, max_leaf_nodes=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, presort='deprecated',
                       random_state=None, splitter='best')

i.e. all arguments with their default values, since you did not specify anything in the definition clf = tree.DecisionTreeClassifier().
You can get the parameters of any algorithm in scikit-learn in a similar way.
Tested with scikit-learn v0.22.2
UPDATE
As Ben Reiniger correctly points out in the comment below, from scikit-learn v0.23 onwards, we need to set the display configuration first in order for this to work:
sklearn.set_config(print_changed_only=False)


Can we use Stop Words while using multinomial navies theorem?

The basic task here I get is sentiment analysis of tweets. So for this, we can first extract features and then use a classifier. Stop words are those which are present for grammatical purposes but do not add any value/meaning to the sentence.
So, yes you can remove the stop words from the tweet, extract the features and then pass it to the classifier. You can get already existing stop words from

from nltk.corpus import stopwords
stop_words = stopwords.words('english')

Removing the stop words doesn't always improve the accuracy.
Take a look at this as it is a very similar problem statement as yours. Hope it helps!


The visualisation is in fact correct, the only issue is that the Plotly map simply does not have those countries marked on its map at all.


Better approach to assign values to determine potential fake sentences

I'll suggest to test the sentence or the tweet for polarity. This can be done using the textblob library. It can be installed as pip install -U textblob. Once the text data polarity is found, it can be assigned as a separate column in the dataframe. Subsequently, the sentence polarity can then be used for further analysis.
Polarity and Subjectivity are defined as;

Polarity is a float value within the range [-1.0 to 1.0] where 0
indicates neutral, +1 indicates a very positive sentiment and -1
represents a very negative sentiment.
Subjectivity is a float value within the range [0.0 to 1.0] where 0.0
is very objective and 1.0 is very subjective. Subjective sentence
expresses some personal feelings, views, beliefs, opinions,
allegations, desires, beliefs, suspicions, and speculations where as
Objective sentences are factual.

Data
import pandas as pd

# create a dictionary
data = {"Date":["1/1/2020","2/1/2020","3/2/2020","4/2/2020","5/2/2020"],
    "ID":[1,2,3,4,5],
    "Tweet":["I Hate Migrants",
             "#trump said he is ok", "the sky is blue",
             "the weather is bad","i love apples"]}
# convert data to dataframe
df = pd.DataFrame(data)

Notice, the sentiment column is a tuple. So we can split it into two columns like, df1=pd.DataFrame(df['sentiment'].tolist(), index= df.index). Now, we can create a new dataframe to which I'll append the split columns as shown;
df_new = df
df_new['polarity'] = df1['polarity']
df_new.polarity = df1.polarity.astype(float)
df_new['subjectivity'] = df1['subjectivity']
df_new.subjectivity = df1.polarity.astype(float)

Next, on basis of the sentence polarity found earlier, we can now add a label to the dataframe, which will indicate if the tweet/sentence is fake, not fake or neutral.
import numpy as np
conditionList = [
    df_new['polarity'] == 0,
    df_new['polarity'] > 0,
    df_new['polarity'] < 0]
choiceList = ['neutral', 'positive', 'negative']
df_new['label'] = np.select(conditionList, choiceList, default='no_label')
print(df_new)

The result will look like this;
Result
        Date  ID                 Tweet    sentiment  polarity  subjectivity  label
0  1/10/2020   1       I Hate Migrants    (-0.8, 0.9)  -0.8      -0.8        fake
1  2/10/2020   2  #trump said he is ok    (0.5, 0.5)    0.5       0.5        not_fake
2  3/10/2020   3       the sky is blue    (0.0, 0.1)    0.0       0.0        neutral
3  4/10/2020   4    the weather is bad    (-0.68, 0.66) -0.7      -0.7       fake
4  5/10/2020   5         i love apples    (0.5, 0.6)    0.5       0.5        not_fake

Complete Code
import pandas as pd
import numpy as np
from textblob import TextBlob
data = {"Date":["1/10/2020","2/10/2020","3/10/2020","4/10/2020","5/10/2020"],
        "ID":[1,2,3,4,5],
        "Tweet":["I Hate Migrants",
                 "#trump said he is ok", "the sky is blue",
                 "the weather is bad","i love apples"]}
    # convert data to dataframe
df = pd.DataFrame(data)
# print(df)
df['sentiment'] = df['Tweet'].apply(lambda Tweet: TextBlob(Tweet).sentiment)
# print(df)

# split the sentiment column into two
df1=pd.DataFrame(df['sentiment'].tolist(), index= df.index)

# append cols to original dataframe
df_new = df
df_new['polarity'] = df1['polarity']
df_new.polarity = df1.polarity.astype(float)
df_new['subjectivity'] = df1['subjectivity']
df_new.subjectivity = df1.polarity.astype(float)
# print(df_new)

# add label to dataframe based on condition
conditionList = [
    df_new['polarity'] == 0,
    df_new['polarity'] > 0,
    df_new['polarity'] < 0]
choiceList = ['neutral', 'not_fake', 'fake']
df_new['label'] = np.select(conditionList, choiceList, default='no_label')
print(df_new)


Finding repeating string patterns in thousands of files

It looks like you want new word discover ?

Because thousand is not a big deal

Just build ngrams of filenames , count them would be fine .

You can use Trie to store string counts,  reduce memory cost , I can provide a dict way(in python):

from collections import defaultdict, Counter

# for memory effcient, you would need trie here
# t = Trie()
t = defaultdict(int)

filenames = ["sjkghkjfs <data> skjfs <data> kjskdfjsfkjs <data> sahkj",
            "tretyer erytewr fskjdf <data> trjk",
            "...."]

# some preprocess, tokenize to  sentence , to  words , filter useless one

def ngrams(s, start, end):
    for i in range(start, end+1):
        if len(s) > i:
            for j in range(i, len(s)):
                yield s[j:j+i]

# Suppose you need same string with length 4 ~ 6
for s in filenames:
    for word in ngrams(s, 4, 7):
        t[word]+=1

# Suppose you need most_common 5
print(Counter(t).most_common(5))

Further more

you can

tokenize the filenames at first.
Calculate tf-idf or word entropy to remove the useless words
then do something like above .

And

If you have a group of traget strings ,
word2vec may be a good tool, you can use it to search the strings in similar domain.


NN aren't ideal for regression tasks.

I'd train 10 networks, one for each magnitude.

CNN #k would be a binary classifier predicting whether the magnitude is k or less

In other words, CNN #10 is a function that always returns 1, and CNN#0 is the binary classifier you've already trained.


Performance of CNN based deep models with number of classes

You might want to have a look at the hierarchical classification idea I described in my masters thesis.

In short:

If I had a problem where I needed to distinguish 42 breeds of dogs and 30 breeds of cats, I would very likely just create one classifier for the $42+30=72$ classes. Making a two-step approach (first cat vs dog, then cat-breed / dog-breed) seems not to give better results and for sure is more complicated
Analysis of errors becomes simpler the less classes you have
A single accuracy score for more than a handfull of classes / skewed data is not meaningful. For example, if you have two models with both 90% accuracy, one might be way better than the other one.


Can we specify the number of data generated(minority class) using SMOTE?

SMOTE is an algorithm for generating as many minority samples as you like. Thus, you can generate as many samples as you want.


Not sure if you figured this out, but I've been looking into it recently, and this is what I've found:

Is the 'normal' LSTM assisted by GPU?
is the normal LSTM supposed to be faster running on GPU or CPU?

Like @pcko1 said, LSTM is assisted by GPU if you have tensorflow-gpu installed, but it does not necessarily run faster on a GPU. In my case, it actually slowed it down by ~2x, because the LSTM is relatively small and the amount of copying between CPU and GPU made the training slower. I think with a larger network, it would speed things up. I also found that LSTM only used ~25% of the GPU, while CuDNNLSTM used ~35% of the GPU, but haven't done a thorough investigation to figure out where the difference comes from.

how are LSTM and CuDNNLSTM different?

This github issue talks about how to convert CuDNNLSTM layers to LSTM layers. I found it pretty illuminating in how CuDNNLSTM has 2x weights/biases as LSTM, and how to convert from one to the other.


Regression - Unbalanced Categorical Features

First, it depends on the number of samples and the degree of imbalance:

Small number of samples may cause slightly imbalanced features' categories to be trained very few times within the training stage
High imbalances can cause some categories to only appear in the training or in the validation/test stages

  As a rule of thumb, I usually join the categories up to a point their distribution in the train and validation datasets are similar.

While this can lead to some bias, it avoids problems on handling categorical features when deploying in production or when testing in the test dataset.

Other strategy is to define a threshold (e.g. 5%) in which you join all the features categories which their distribution is below such threshold.


Preprocessing for Text Classification in Transformer Models (BERT variants)

A quick experiment you can do is to once do the preprocessing steps that you usually do and then feed it to the model and get the results. And once feed the dataset as it is to the model to compare the difference.
In my experience doing the preprocessing won't make any difference, based on the dataset it gave me 1 more or less percent difference in accuracy (not a considerable change).
When these models are trained, no preprocessing is done, as they want to learn the context of all sorts of sentences.
A reason that your results are not good enough might because of your label distribution. Most of the time the datasets are populated with only one or two labels and other labels are only a small portion of the dataset. If that's the case you might want to look into oversampling solutions.


How to structure data and model for multiclass classification in SVM?

See the svm documentation here. It does support multiclass problem. The only thing is that your y should be like this [0, 1, 2, 3]. If you do not use one-hot encoding and use the y label as it is, it might work.


Error: ValueError('%r cannot be used to seed a numpy.random.RandomState')

Because you are creating a IsolationForest instance with random_state getting initialised to pandas.DataFrame. The code docs for random_state explain it as

  random_state : int, RandomState instance or None, optional (default=None). If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.


How to control false positives in sequential A/B testing while keeping a low sample size?

=run a maximum of n tests or stop earlier if a good improvement is found

Don't do that. Don't stop before n tests have been run if you like what you've seen so far....

http://www.evanmiller.org/how-not-to-run-an-ab-test.html

In general, however, figuring out how big an N you need given an expected effect size and alpha level is the domain of power analysis. Read more here:

http://www.uta.edu/faculty/sawasthi/Statistics/stpowan.html


The task is doable, but time consuming and not easy. This is how I would plan the work:

Write a PDF scraper that navigates the document and converts all the informations it contains into some standardized textual format.
Label the words/elements of this text that correspond to the informations that you are looking for ("lender name", "lending amount", "outstanding balance", etc.).
Run some NLP model, such as an RNN classifier or CRF, to extract information from text.

Steps 1 and 2 are very time consuming (2 more than 1), but it's certainly doable. It will be a lot of work, especially labeling observations to create the Training set, but a nice thing to put on your CV.


How to define a custom performance metric in Keras?

You have to use Keras backend functions. Unfortunately they do not support the &-operator, so that you have to build a workaround: We generate matrices of the dimension batch_size x 3, where (e.g. for true positive) the first column is the ground truth vector, the second the actual prediction and the third is kind of a label-helper column, that contains in the case of true positive only ones. Then we check which instances are positive instances, are predicted as positive and the label-helper is also positive. Those are the true positives.
We can make this analog with false positives, false negatives and true negatives with some reverse-calculations of the labels.
Your f1-metric may look as follows:
def f1_score(y_true, y_pred):
    """
    f1 score

    :param y_true:
    :param y_pred:
    :return:
    """
    tp_3d = K.concatenate(
        [
            K.cast(y_true, 'bool'),
            K.cast(K.round(y_pred), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    fp_3d = K.concatenate(
        [
            K.cast(K.abs(y_true - K.ones_like(y_true)), 'bool'),
            K.cast(K.round(y_pred), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    fn_3d = K.concatenate(
        [
            K.cast(y_true, 'bool'),
            K.cast(K.abs(K.round(y_pred) - K.ones_like(y_pred)), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    tp = K.sum(K.cast(K.all(tp_3d, axis=1), 'int32'))
    fp = K.sum(K.cast(K.all(fp_3d, axis=1), 'int32'))
    fn = K.sum(K.cast(K.all(fn_3d, axis=1), 'int32'))

    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    return 2 * ((precision * recall) / (precision + recall))

Since the Keras-backend calculator returns nan for division by zero, we do not need the if-else-statement for the return statement.
Edit:
I have found a pretty good idea for a exact implementation. The problem with our first approach is, that it is only "approximated", since it is computed batchwise and subsequently averaged. One could also calculate this after each epoch with the keras.callbacks. Please find the idea here.
An example implementation would be:
import keras
import numpy as np
import sklearn.metrics as sklm

class Metrics(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.confusion = []
        self.precision = []
        self.recall = []
        self.f1s = []
        self.kappa = []
        self.auc = []

    def on_epoch_end(self, epoch, logs={}):
        score = np.asarray(self.model.predict(self.validation_data[0]))
        predict = np.round(np.asarray(self.model.predict(self.validation_data[0])))
        targ = self.validation_data[1]

        self.auc.append(sklm.roc_auc_score(targ, score))
        self.confusion.append(sklm.confusion_matrix(targ, predict))
        self.precision.append(sklm.precision_score(targ, predict))
        self.recall.append(sklm.recall_score(targ, predict))
        self.f1s.append(sklm.f1_score(targ, predict))
        self.kappa.append(sklm.cohen_kappa_score(targ, predict))

        return

To make the network to call this function you simply add it to you callbacks like
metrics = Metrics()
model.fit(
    train_instances.x,
    train_instances.y,
    batch_size,
    epochs,
    verbose=2,
    callbacks=[metrics],
    validation_data=(valid_instances.x, valid_instances.y),
)

Then you can simply access the members of the metrics variable.


You might try to use the metadata of the individual sections for classification:

Examples:

Length -> Abstracts are usually short.
Existence of Figures -> Do not usually appear in abstract of references
Relative Abundance of quotations. The references should have a significant higher share of reference keys than the others.

You could then use the metadata as input for any supervised learning approach. For example a Neural Network or a (boosted) decision tree.


Clustering algorithms for high dimensional binary sparse data

How many ethnic groups did you identify?

If I had to visualize your problem, I'd determine the key influencers for each of the ethnic groups in a Naive Bayes like approach.
These genes (gene combinations?!) (including their values) may strongly correlate to some ethnic group, while not (or inverse) correlate to another.

Place them on top of a pyramid graph. Place bars to the left and right for the correlation values.


Why is Local Outlier Factor classified as Unsupervised if it requires training data with no outliers?

It appears that SciKit-Learn implements two "modes" for LocalOutlierFactor, where one is unsupervised and one is semi-supervised. I think I misunderstood the documentation and implementation at first.

The two modes are "outlier" and "novelty". The sample code provided on the SciKit-Learn website uses Novelty detection mode. When using the "outlier" mode the model does NOT require a training dataset--an unlabeled dataset is used as whole and a score is given to every data point.

  novelty : boolean, default False
  By default, LocalOutlierFactor is only meant to be used for outlier detection (novelty=False). Set novelty to True if you want to use LocalOutlierFactor for novelty detection. In this case be aware that that you should only use predict, decision_function and score_samples on new unseen data and not on the training set.


How to create these two plots from two arrays with numbers?

Changed the values for better representation
# Gathering Dataframe
import pandas as pd
data = [[1, 5], [1, 6], [2, 5], [0,0], [5,8], [20,4]]
df = pd.DataFrame(data, columns = ['xx', 'yy'])
df

#Drawing the plot (Scatter Plot)
from matplotlib.pyplot import *

plott = df.plot.scatter(x='xx', y='yy', title= "Scatter plot between two variables X
and Y");

plott

# (line plot connecting the scatter plots)
plt.plot( 'xx', 'yy', data=df, linestyle='-', marker='o')
plt.show()


Perhaps your repressor is for one time step behind. For e.g. if you are predicting tomorrow's stock price, then the repressor say market volume can be from today.
The other thing you can do is build a separate model that estimates the regressor for the next day and then use that in prophet. Be aware that your regressor is then becomes imperfect since it represents an estimate and not the true value. Depending on your use case, this may or may mot be a problem.


The de-correlation effect is more important than following sequence of trajectories in this case.

Single step Q-learning does not rely on trajectories to learn. It is slightly less efficient to do this in TD learning - a Q($\lambda$) algorithm which averages over multiple trajectory lengths would maybe work better if it were not for the instability of using function approximators.

Instead, the DQN-based learning bootstraps across single steps (State, Action, Reward, Next State). It doesn't need longer trajectories. And in fact due to bias caused by correlation, the neural network might suffer for it if you tried. Even with experience replay, the bootstrapping - using one set of estimates to refine another - can be unstable. So other stabilising influences are beneficial too, such as using a frozen copy of the network to estimate the TD target $R + \text{max}_{a'} Q(S', a')$ - sometimes written $R + \text{max}_{a'} \hat{q}(S', a', \theta^{\bar{ }})$ where $\theta$ are the learnable parameters for $\hat{q}$ function.

It might still be possible to use longer trajectories, sampled randomly, to get a TD target estimate based on more steps. This can be beneficial for reducing bias from bootstrapping, at the expense of adding variance due to sampling from larger space of possible trajectories (and "losing" parts of trajectories or altering predicted reward because of exploratory actions). However, the single-step method presented by DQN has shown success, and it is not clear which problems would benefit from longer trajectories. You might like to experiment with options though . . . it is not an open-and-shut case, and since the DQN paper, various other refinements have been published.


What is the proper order of normalization steps before and after splitting data

Assuming I understand it correctly, I think your process is ok this way but I'm not sure about step 3 "standardization":

Steps 1 and 2 are ok since this cannot leak any kind of information from the test set to the training set.
If the standardization step involves calculating values (e.g. mean and s.d.) over the whole data (including test set) and then standardizing the whole data using these values, then that could be an issue. I'd suggest moving step 3 after the split, so that standardization is based only on the training set and then applied "as is" to the test set.


Going according to your problem, you should set the NaN values to 0. It follows logically as the number of minutes spent with the nurse or doctor will be zero if the patient hasn't visited them yet. And yes as you said filling with mean is not correct as it would not represent the problem correctly.

Also, if you fill the NaN values with big number such as 9999 they might be treated as outliers which is not the case. Hence, purely thinking from a logical point of view you should replace them with 0. But I also suggest you to run trial experiments with some other values such as large values. Machine Learning is a lot about hyperparameter tuning and in this case how you replace your NaN values behaves like a hyperparameter!


What is the best way to compare these small distributions?

To assess the effects of hyperparameters on the neural network results, you need to eliminate confounding variables. Let's say you have network $A_t$ and network $B_t$ corresponding to trial/sample $t$. The randomly initialized weights of $A_t$ should equal $B_t$. If you are using a stochastic optimizer (e.g. SGD), you need to ensure that the random instance selection is the same between training network $A_t$ and $B_t$.
Once you've eliminated confounding variables, then you effectively have paired samples, which you can compare via a Wilcoxon signed-rank test. For small sample sizes, it can be preferable over a paired t-test because 1. you can't verify that the samples are normally distributed and 2. the Wilcoxon test effectively evaluates median rather than mean. The median is more robust against outliers, which are particularly influential if you only have a few samples.


How are 1x1 convolutions the same as a fully connected layer?

Your Example

In your example we have 3 input and 2 output units. To apply convolutions, think of those units having shape: [1,1,3] and [1,1,2], respectively. In CNN terms, we have 3 input and 2 output feature maps, each having spatial dimensions 1 x 1.

Applying an n x n convolution to a layer with k feature maps, requires you to have a kernel of shape [n,n,k]. Hence the kernel of you 1x1 convolutions have shape [1, 1, 3]. You need 2 of those kernels (or filters) to produce the 2 output feature maps. Please Note: $1 \times 1$ convolutions really are  $1 \times 1 \times \text{number of channels of the input}$ convolutions. The last one is only rarely mentioned.

Indeed if you choose as kernels and bias:
$$
\begin{align}
w_1 &=
\begin{pmatrix}
 0 & 1 & 1\\
\end{pmatrix} \in \mathbb{R}^{3}\\
w_2 &=
\begin{pmatrix}
2 & 3 & 5\\
\end{pmatrix} \in \mathbb{R}^{3}\\
b &= \begin{pmatrix}8\\ 13\end{pmatrix} \in \mathbb{R}^2
\end{align}
$$

The conv-layer will then compute $f(x) = ReLU\left(\begin{pmatrix}w_1 \cdot x\\ w_2 \cdot x\end{pmatrix} + \begin{pmatrix}b_1\\ b_2\end{pmatrix}\right)$ with $x \in \mathbb{R}^3$.

Transformation in real Code

For a real-life example, also have a look at my vgg-fcn implementation. The Code provided in this file takes the VGG weights, but transforms every fully-connected layer into a convolutional layers. The resulting network yields the same output as vgg when applied to input image of shape [244,244,3]. (When applying both networks without padding).

The transformed convolutional layers are introduced in the function _fc_layer (line 145). They have kernel size 7x7 for FC6 (which is maximal, as pool5 of VGG outputs a feature map of shape [7,7, 512]. Layer FC7 and FC8 are implemented as 1x1 convolution.

"Full Connection Table"

I am not 100% sure, but he might refer to a filter/kernel which has the same dimension as the input feature map. In both cases (Code and your Example) the spatial dimensions are maximal in the sense, that the spatial dimension of the filter is the same as the spatial dimension as the input.


Site for implementing a research papers without code

Follow Kristian Kersting, a former professor of mine at University of Bonn who was involved in that event. That was actually a workshop for (if I remember right) NIPS.

UPDATE: well … I found it :D


CNN for checking existance of single label

Convolutional models can be used either for classification or not. You cannot train a classifier on a single label, because the loss function of such models is meant to punish the model every time an image is missclassified. If you have a single label, accuracy will be 100% all the time, by definintion. And such a model would be useless.

I think you have two options:

Get a bunch of images that are not dogs, then train a CNN classifier.
Train an Autoencoder for dimensionality reduction on your "dog data". These models can be used for anomaly detection tasks. Once you have trained such a model, when you feed a non-dog image it should be able to recognize it as an anomaly and flag it.

Both methods could work equally good. The first is the simplest one, but if you like playing and experimenting with DL the second is much more fun.


The purpose of the autencoder would be similar to doing an embedding of text to pass latent space features of the input to the LSTM, as opposed to raw inputs. In the case of text, this is usually done to reduce the very large and sparse dictionary (word subword, whatever) space to a   space which is less sparse, significantly smaller, and therefore more computationally efficient.

In your case, it looks like the data is not high dimensional to start with (I am assuming this because of the significant events over time), so you would not get the benefit of mapping the input to a latent space. If your trend over time is high dimensional, then it may be worth it, as the embedding/autencoding may pick up and reduce correlations between trends in a few dimensions.


Am I justified in dropping this independent variable?

If you think it isn't a variable that explains your target variable (churn), you are of course allowed to leave it out - it is in a way then a hand-crafted feature, because you are deliberately telling the model to ignore it (by omitting it).

There are models that would be able to take it into consideration and then decide whether to use it or not, such as component-wise gradient boosting (CWGB) or neural networks. CWGB would simply end up assigning the variable a coefficient of zero if it decides that it doesn't help explain churn rate.

In general, which the percentage split that you gave for males and females, I personally think it could be a valuable feature - even if the model is able to deduce that roughly 14% of all people churn. There might also be correlation between the gender and other features in your dataset that, together, are able to give a stronger signal regarding churn probability.


Residual plots: why do we want to know the error?

Let me try to give an example: Let's say we would like to predict a housing price by looking at the number of bathrooms. The x-axis represents your feature/independent variable (#bathrooms) and your y-axis represents the response/dependent variable. The points represent training samples you collected and know both the independent and dependent variables of.

Now in Linear Regression our goal is to predict the housing price based on a new example of #bathrooms we didn't see yet. For that we have to create a line through the data points that represents the training data the best. So, what qualifies a line as being a good or bad line? The answer is a cost/loss function. In linear regression we often use the squared mean of the error/residuals (MSE):

$$
\frac{1}{n} \sum (y_i - \hat{y_i})^2
$$

where $y_i$ is the true housing price and $\hat{y_i}$ is the predicted value.
Our goal is to minimize this loss.

Plotting the residuals can give us many insights: For example on which training samples we make the highest errors. Also we can see how the residuals are distributed. For more complicated applications we need to check this distribution for the so called Markov Theorem which would go too much in detail.


How to see/change learning rate in Keras LSTM?

In Keras, you can set the learning rate as a parameter for the optimization method, the piece of code below is an example from Keras documentation:
    from keras import optimizers

   model = Sequential()
   model.add(Dense(64, kernel_initializer='uniform', input_shape=(10,)))
   model.add(Activation('softmax'))

   sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
   model.compile(loss='mean_squared_error', optimizer=sgd)

In the above code, the fifth line lr is learning rate, which in this code is set to 0.01. You change it to whatever you want.
For further detail, see this link.
Please tell me if this helped to solve your problem.


Why can't the XOR linear inseparability problem be solved with one perceptron - like this?

Your function

  0,otherwise 1

is not a linear combination of the inputs.

From the definition of single layer perceptronL

  A single layer perceptron (SLP) is a feed-forward network based on a threshold transfer function

Your function has two thresholds:  one for < 1 another for >1


Different accuracy for different rng values

The training errors in this dataset has a huge difference (99% vs 57%). So, maybe the one with the rng(1) split has overfitted your dataset.

  So there is a huge change in accuracy as visible. What does this mean?
  Am I training it wrong?

The huge change might be due to overfitting. (Also, judge the model through validation curves, and then fit a model which balances the bias-variance plot.)


Your network is too large for this much data. Reduce the number of units in the layers, go for simple 'relu' in the layers except the last one where you should use softmax. Consider reducing number of layers.
I would recommend using decision trees libraries like XGBoost, CatBoost, LightGBM etc. for this problem. You will get the best result.


Convert Keras model to C++ using protobuf file and Tensorflow C API

My thoughts are to save Keras model to .pb(protobuf) file and then
load this model with Tensorflow C API.
Do you think this will be possible?

the answer is yes, absolutely. there are many ways to do this and i'll list some here so that you can pick up what suites your needs.

you can use cppflow, which is a c++ wrapper for the tensorflow C API, and actually makes it way easier to do the job than using the C API directly. they also have some examples on how to use it and in fact i think that this is the best solution out there.
you can see this wonderful repo hello_tf_c_api, especially this example.

or you can follow these articles on medium if you wish to use the c API directly:

undocumented tensorflow c api
deploying tensorflow 2.1 as c/c++ executable

there's also these two wonderful github gists that would walk you through how to do this exactly (although i recommend the c approach because it's more straightforward):

Training TensorFlow models in C
Training TensorFlow models in C++


How to predict when an appointment will be scheduled?

Create new features specifying days of the week.
df['Created'] = pd.to_datetime(df['Created'])
df['Scheduled'] = pd.to_datetime(df['Scheduled'])
df['Scheduled_day_of_week'] = df['Scheduled'].dt.day_name()
df['Created_day_of_week'] = df['Created'].dt.day_name()

Create a new column day difference which will be the difference between the two days.
Then you can visualize for each day what would be the day difference.
Similarly you can try for the date.
This process is called Feature Engineering.


Signal correlation - matching specific points

I'm no expert with signal processing but my first attempt to solve the complexity issue would be to downsample the time series. For instance if you resample a 10k points series to say 100 points, methods such as DTW can be applied to get a first approximation of the "global" similarity between 2 signals. This would take much less time than comparing the full time series, and then based on the result (for instance using a threshold) only the pairs of signals with high similarity would be compared fully. In other words, the first step acts as a filter so that only a subset of the pairs need a detailed comparison.


Classifying Data in Orange: What's the difference between 'Features', 'Target Variables', and 'Meta Attributes'

Meta variables are meta data, data about data, not used for statistical inference.
Features or variables or attributes are the measured inputs of the problem domain, the independent variables. The target variable is the dependent variable or the measure we're trying to model or forecast. Not all problems can be or need to be formulated in such a way. Orange was traditionally designed to accommodate machine learning workflows, hence the naming.


How do I assess which sentiment classifier is best for my project?

A couple of important points:

Sentiment analysis is not an exact science. Two people, reading the same text in different contexts will come to different conclusions about sentiment, especially on borderline cases. Perhaps text has complex grammar, or has a metaphor or simile in it where it helps to understand what is actually being compared.
The ground truth for sentiment data sets is established by people. The best you can hope for from any trained ML classifier is that it closely matches the opinion of those people when it predicts on new data.
Your different sentiment analysis tools could well vary in quality, because they will have been built with different technologies at different times, and with different training data.

There are a couple of ways I can think of that you could assess the classifiers for your purpose:

1. Test against your own labelled test data

It is important to source this yourself to ensure it does not overlap with training data used by any of the models you want to compare (if it did overlap then it gives a big advantage to any model that trained using it). You will need to collect ground truth data on sentiment for all the texts.

Before you test, it is a good idea to pick a metric that you care about for your intended use. Accuracy is not the only metric, but it is probably fine for your purpose. You might choose other metrics if there are different costs associated with incorrect classifications - e.g. if assigning negative sentiment needs to be done cautiously.

Then run each classifier across your test set and calculate the metric. The classifier with the best metric is your best guess at the one you should use.

If you have a large enough test set, you can split it into parts and get a measure of the estimate error in your metric. This will help you see whether differences in the test metrics are significant.

This can be the best approach, especially if you can source data that you know matches your project goals. However, it would take a long time and a lot of effort.

If you are in a hurry, you could just search for and use someone else's dataset e.g I found these ones on a quick web search: a Kaggle competition and a free sample hosted by Niek Sanders (I have no idea of the quality of these - you should sample them and see if the data is useful for you). There is a risk though that these data sets were used to train one of your classifiers, so it would give you a false high rating.

2. Read up on associated reports and papers

Each of the sentiment analysers should explain the model class and data used to train it in documentation. Quite often you can find papers comparing the different approaches and quoting accuracy scores on a standard data set. If you are lucky, you will find enough comparisons that you get some sense of which algorithms are considered "cutting edge" and which are out-dated.

3. Collect text that is classified differently by algorithms you are considering, and get feedback

A variation of (1), you can do this at any stage, and it might work well in the context of a live product. For all the data that is classified, log how different classifiers respond to it, and where there is discrepancy, save it for later assessment. You will need some people to read and help classify the text (ideally without seeing what the classifiers thought), and you can collect metric data over time and use the classifier that does best once you have collected enough examples to make a clear decision.

Ultimately what matters is the results of using your classifier in a project. Just labelling data has no inherent purpose. It is the consequences of those labels that matter. So you need to be driven by results. If you use one classifier and get feedback that the product is not performing well, you may be able to trial another one. If your product is used by many people, and enough are giving feedback of quality, you could even A/B test a couple of classifiers in production.