How to creat a plot for the accuracy of a model

Your context is different than the one provided in the link. There, the author has made a neural network in Keras and has plotted the accuracy against the number of epochs. One epoch is when an entire dataset is passed both forward and backward through the neural network once. So, he is calculating accuracy after every epoch while the weights vary to fit data based on the loss function. (Thus, the accuracy increases as the number of epochs increases.)

In your case, you are performing a linear regression which fits the data and generates an equation. There is no feedback system. Accuracy here can be defined based on your need. In this case (Predicting sons height based on their father's), you can define accuracy as how accurate your predictions were. Take an error function like MAE (Mean absolute error). Lesser the error rate is more accurate is your model. MAE is an accuracy measure here.

For general classification tasks, accuracy is the number of instances you predicted correctly divided by the total number of instances. (In the link, author used default keras accuracy metric defined somewhat like this)


xgboost or lightgbm to handle Binomial problems

I do not think this is possible with the main (as I can tell) xgboost library, because this essentially sounds like a regression on rates/proportions situation (a dependent variable bounded in [0,1]).

An easy way to have implemented this would have been to allow the Beta distribution (see how it is bounded) as an objective function in xgboost, but it is not an option at this point.


Cleaning input data with pd.get_dummies()

Categorical features need to be converted to numerical values. They are various ways to do that. I would recommend reading this blog and this one to learn what are the advantages and disadvantages of choosing each. The first method you showed is called one-hot encoding that you can get easily by pd.get_dummies as you mentioned, and frequently used by people in algorithms like XGBoost. One of the biggest downfall of the first method OHE, is increasing often unnecessarily the dimensionality of your feature space. Other cons could be not being a good representative of feature space as well as problem of missing features in test data. The second method is LabelEncoding, usually used in target variables or when you have very few categorical features.

The problem surfaces when you start having a large number of categorical features (>50 % of feature space) with each feature having many sub levels. Here you need more clever way of feature encoding like Target-based mean encoding (smoothing, you can find some kernels in Kaggle like this). If you are Python user, I have found a fairly new package under scikit-learn-contrib offering a wide range of Categorical Encoding Methods. This Kdnuggets post also compares some of the aspect of using different methods.


Yes, this is a how-long-is-a-piece-of-string question. I think it's good to beware of over-engineering, while also making sure you engineer for where you think you'll be in a year.

First I'd suggest you distinguish between processing and storage. Storm is a (stream) processing framework; NoSQL databases are a storage paradigm. These are not alternatives. The Hadoop ecosystem has HBase for NoSQL; I suspect Azure has some kind of stream processing story.

The bigger difference in your two alternatives is consuming a cloud provider's ecosystem vs Hadoop. The upside to Azure, or AWS, or GCE, is that these services optimize for integrating with each other, with billing, machine management, etc. The downside is being locked in to the cloud provider; you can't run Azure stuff anywhere but Azure. Hadoop takes more work to integrate since it's really a confederation of sometimes loosely-related projects. You're investing in both a distribution, and a place to run that distribution. But, you get a lot less lock-in, and probably more easy access to talent, and a broader choice of tools.

The Azure road is also a "big data" solution in that it has a lot of the scalability properties you want for big data, and the complexity as well. It does not strike me as an easier route. Do you need to invest in distributed/cloud anything at this scale? given your IoT-themed use case, I believe you will need to soon, if not now, so yes. You're not talking about gigabytes, but many terabytes in just the first year.

I'd give a fresh team 6-12 months to fully productionize something based on either of these platforms. That can certainly be staged as a POC, followed by more elaborate engineering.


How to retrieve the clustering results of rpart

I found an answer to my question using the package partykit, I post it to help others.

tree_party <- as.party(tree) #coercing the rpart.object into partykit

tree_fit <- fitted(tree_party) #a two-column data.frame with the fitted node numbers and the observed responses on the training data.

tree_pred <- predict(tree_party, newdata = data, type = "node")


Machine Learning applied to database design

This is such an interesting question. I suppose that it is possible but you would have to answer some more questions before you can actually get help with modeling something.

Are you looking for it to learn SQL or NoSQL?
You'd have to make a distinction between something that can learn relational database design versus something that learns how to be a DBA and work in a particular language. For example, relational databases are based on theory (and relatively straightforward) but how you implement certain things in Oracle or SQL Server (as examples) will vary greatly. Or maybe you're looking for a particular type of design like data warehousing (star patterns, etc). Whichever approach you choose would have a profound effect on the type of model you are going to build.
There are some pitfalls that you would have to account for. A relationship based on text columns is acceptable design, but a relationship based on integer hashes of those same text columns is much better. How a model would account for something like this is unknown to me.
Relating to the item above, you would have to come up with some metric for the success of your model. Is it the levels of relational design that it can reach? Is it some hardware performance benchmark? Is it some level of cognition that your model can reach for extremely complex designs?

I think that once you answer these types of questions you will be in a much better position to start model development.


Combining Datasets with Different Features

You can use R to do that.

The smartbind function is the perfect way to combine datsets in the way you are asking for:

library(gtools)

d1<-as.data.frame(rbind(c(1,7,3),c(4,8,4))))
names(d1)<-c("featureA","featureB","featureC")

d2<-as.data.frame(rbind(c(3,4,5,6),c(9,8,4,6)))
names(d2)<-c("featureA","featureC","featureD","featureE")

d3<-smartbind(d1,d2)


Neural Networks for Unsupervised Learning

Yes, there are others. The most important dimensionality reduction technique in Deep Learning is Autoencoders. They are neural networks with a "funnel" structure, that shrinks the size of the signal, forcing the Network to learn how to represent the same information with less nodes.

Autoencoders are much more common than SOMs in practical DL. While SOMs are used mostly for data visualization (usually they reduce a dataset in a 2D representation), Autoencoders give you absolute freedom in the number of factors that you can extract. This makes them more useful for any research or production purpose.

A good theoretical introduction can be found here.

Please also take a look at this practical implementation in TensorFlow 2.0.


What will be the input_shape of tf.keras.layers.Conv3D be for these inputs

From Keras document,

Input shape
5+D tensor with shape: batch_shape + (channels, conv_dim1, conv_dim2, conv_dim3) if data_format='channels_first' or 5+D tensor with shape: batch_shape + (conv_dim1, conv_dim2, conv_dim3, channels) if data_format='channels_last'

Assuming data_format='channels_last' and num_of_videos may be used to decide batch_size(which is not required while defing input_shape),
input_shape=[ 100, 100, 37, 3 ]


Feeding XLM-R embeddings to neural machine translation?

You only got the indices in the XLM-R's vocabulary. This is the input to XLM-R, you need to actually run the model. By calling

model(tokens_tensor)

you get a tuple of tensors that are outputs of the model. Check the documentation for the outputs are.


Why do we pick random features in random forest

The idea of random forests is basically to build many decision trees (or other weak learners) that are decorrelated, so that their average is less prone to overfitting (reducing the variance). One way is subsampling of the training set. The reason why subsampling features can further decorrelate trees is, that if there are few dominating features, these features will be selected in many trees even for different subsamples, making the trees in the forest similar (correlated) again.

The lower the number of sampled features, the higher the decorrelation effect. On the other hand, the bias of a random forest is the same as the bias of any of the sampled trees (see for example Elements of Statistical Learning), but the randomization of random forests restrict the model, so that the bias is usually higher than a fully-grown (unpruned) tree. You are correct in that you can expect a higher bias if you sample fewer features. So, "feature bagging" really gives you a classical trade-off in bias and variance.


How to determine whether a bad performance is caused by data quality?

First, it sounds like your choice of model selection is a problem here.  Your outputs are binary-valued, not continuous. Specifically you may have a classification problem on your hands rather than a traditional regression problem.  My first recommendation would be to try a simple classification approach such as logistic regression or linear discriminant analysis.

Regarding your suspicions of bad data, what would bad data look like in this situation? Do you have reason to suspect that your $X$ values are noisy or that your $y$ values are mislabeled? It is also possible that there is not a strong relationship between any of your features and your targets.  Since your targets are binary, you should look at histograms of each of your features to get a rough sense of the class conditional distributions, i.e. $p(X_1|y=1)$ vs $p(X_1|y=0)$. In general though, you will need to be more specific about what "bad data" means to you.


Can anyone help with me how to train new data with already saved pickle file?

Load the pickle file and your new data. Fit the loaded model on the data
model = pickle.load(open(pickle_file,'rb'))
model.fit(x_new, y_new)


I think your biggest problem is the lack of biases. Between the input layer and the hidden layer, you should not only transform by the weights but should also add a bias. This bias will shift your sigmoid function to the left or right. Take a look at this code (I made some adaptations).

What is important:

Added biases.
Altered your error_w such that they are correct.
Made some good random starting points for biases (np.random.rand(width) * 15. - 7.5) such that all biases are random points on the desired x-scale.
Made a plot that shows the initial guess and final.

Let me know if some parts are not clear:

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1 / (1 + np.math.e ** -x)

def sigmoid_deriv(x):
    return sigmoid(x) * (1 - sigmoid(x))

def guess(x):
    layer1 = x
    z_2 = layer1_weights * layer1 + layer1_biases
    a_2 =sigmoid(z_2)
    z_3 = np.dot(a_2, layer2_weights) + layer2_biases
    # a_3 = sigmoid(z_3)
    a_3 = z_3
    return a_3

x_data = np.random.rand(500) * 15.0 - 7.5
y_data = [np.sin(x) for x in x_data]

ETA = 0.05
width = 10

layer1_weights = np.random.rand(width) * 2. - 1.
layer1_biases = np.random.rand(width) * 15. - 7.5
layer2_weights = np.random.rand(width) * 2. - 1.
layer2_biases = np.random.rand(1)* 2. - 1.

error_all = []

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()

epochs = 500000

for loop_iter in range(epochs):
    # data init
    index = np.random.randint(0, 500)
    x = x_data[index]
    y = y_data[index]

    # forward propagation
    # layer 1
    layer1 = x

    # layer 2
    #TODO add the sigmoid function here

    z_2 = layer1_weights * layer1 + layer1_biases
    a_2 =sigmoid(z_2)

    # layer 3
    #TODO remove simgmoid here (not that is really matters, but values at each layer are after sigmoid
    z_3 = np.dot(a_2, layer2_weights) + layer2_biases
    # a_3 = sigmoid(z_3)
    a_3 = z_3

    # error
    error = .5 * (a_3 - y) ** 2  # L2 loss

    # backpropagation
    # error_wrt_layer3 * layer3_wrt_weights_layer2

    # error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)

    delta = (a_3 - y)

    error_wrt_layer2_weights = delta * a_2
    error_wrt_layer2_biases = delta

    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1

    # error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1
    error_wrt_layer1_weights = delta * np.dot(sigmoid_deriv(z_2), layer2_weights) * layer1
    # error_wrt_layer1_weights = 0

    error_wrt_layer1_biases =  delta * np.dot(sigmoid_deriv(z_2), layer2_weights)

    # a = 0
    # while a ==0:
    #   a*0

    # update the weights
    layer2_weights -= ETA * error_wrt_layer2_weights
    layer1_weights -= ETA * error_wrt_layer1_weights
    layer2_biases -= ETA * error_wrt_layer2_biases
    layer1_biases -= ETA * error_wrt_layer1_biases

    error_all.append(error)

    if loop_iter % 10000 == 0:
        print(error)

# plt.plot(error_all)
# plt.show()

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()


Small amount of training data set for naive Bayes classifier for binary classification

Actually in machine learning more data equals more accuracy but as you mentioned in the question you had 300 sample dataset.So, the classifier has the little room to decide whom it should select but if you have less number of classes and features you may get better results.When I'm doing my project based on sensor data I used just about 100 sample records from a sensor and it actually predicted very well and got the accuracy of 77 and it has grown once I increased the dataset.but don't train it with the noise like outliers and unwanted features they affect your accuracy a lot


How do attention mechanisms in RNNs learn weights for a variable length input

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size.

For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight $\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\alpha_{iT_x})$ is calculated for target element $y_i$  as follows
$$\alpha_{ij}=\frac{\text{exp}(e_{ij})}{\sum_{k=1}^{T_x}\text{exp}(e_{ik})}$$
where $e_{ij}$ is calculated by neural network $a$ that receives hidden state $s_{i-1}$ of decoder (decoder generates the target sequence element-by-element) and hidden state $h_j$ of encoder (encoder distills the source sequence into hidden states $h_j$, where $h_j$ is the concatenation of $j$-th hidden states of forward and backward RNNs) and outputs
$$e_{ij} = a(s_{i-1}, h_j).$$
In other words, network $a$ recieves a vector of size $|s| + |h|$ and outputs a number. Because of this network, the attention matrix $\boldsymbol{\alpha}$ is free in size.

Attention weights are then used to calculate context $c_i$ for target $y_i$ as follows
$$c_i = \sum_{j=1}^{T_x} \alpha_{ij}h_j,$$

and so on and so forth.

Consequently, for the next (source, target) pair with lengths ($T_{x_2}$, $T_{y_2}$), the size of $\boldsymbol{\alpha}$ would also be $T_{y_2} \times T_{x_2}$ with no problem.


Poor performance of SVM after training for rare events

This I can say because I ran the trained model on the entire dataset
  again (re-trained it) an predicted on the same dataset.

You seem to be making a fundamental mistake here. If you train and test on the same data, your performance will not be representative of how the model can perform on unseen data points.

Make sure that you train and test on different datasets. If the difference in performance persists, make sure that the validation dataset is representative of your training set. If it's not, this might explain why the model is not performing well on the validation set.


Softmax activation predictions not summing to 1

Softmax doesnt work as an LSTM activation. You have to add a dense layer using a softmax activation after the LSTM layer. I wouls suggest using another activation in the LSTM like relu.


Machine learning model to predict the best candidate

I think it would make more sense to train a model to grade (regression) each candidate, them from candidates of a particular set you can use the best candidate from its "grade".

Also, you should try changing the information from raw cloud of points to more meaningful geometric form irfomation:

Number of vertices/segments
Segments length  mean and variance
Skewness
Size and direction of major and minor axis
Center position
Moments of area (first,second,third...)

Update

To apply a regression model you will need to generate grades for the training set and that might be a challenge. For that I will propose a few heuristics to generate this:

Since you have about 10k samples, you could assign to every candidate the probability of been the best candidate in any set (for example, if he is the best candidate in 10 sets you can give him a grade $\frac{10}{10,000}$.

You could try clustering the samples and assigning a grade to every cluster as the probability of a candidate in that cluster been the best candidate by $\frac{N_{best}}{N_{cluster}}$, where $N_{best}$ is the number of best candidates in any set in that cluster and $N_{cluster}$ is the number of candidates in that cluster
You can assign to each best candidate a grade like $1$ if he is best candidate in every set it appears and $1-e^{-\alpha N}$ for every $N$ times it appears in a set without been the best candidate. You will have to tune the decay rate $\alpha$ like any hyperparameter.


Finding Patterns in Error Reports - How to Approach?

You need data for when the error occurred and for normal operation as well, so that you can gain insights from comparing these two groups of log data in analyses and visualizations, e.g. box plots.

You also should consider to regard a time span of log data from before the error happened as possibly related to the occurance of the error. So, you should group your log data into the following two groups by adding a boolean feature to each log entry or data set row: (1) a specific error happened within the next x seconds / minutes / hours / days and (0) a specific error did not happen within the next x seconds / minutes / hours / days.

It is always a good idea to visualize your data, because it helps you to understand your data, e.g. to see if your data is "dirty" or if your assumptions about a specific feature behavior are correct or not as well as related behavior of two or three features. In chapter 4 of the "Python Data Science Handbook" by Jake VanderPlas, the author shows how plot data using python and matplotlib.

In a next step, you can do preprocessing tasks like e.g. to clean your data so that you can later apply statistical methods or machine learning to it to find feature behavior related to error occurrances.

And here is a link to the free online course at edX "Analyzing Data with Python" which you could take.


Why is Random Forest feature importance biased towards high cadinality features?

Random forest (as almost any other algorithm) is prone to selecting variables which can lead to a one-to-one relationship with the $Y$ variable.

Why? Because you are leading them to be overfitted. If your variables have high cardinality, it means they form little groups (in the leaf nodes) and then your model is "learning" the individuals, not generalizing them. The more "cardinal" the variable, the more overfitted is the model.

For example, if you have social security number as variable (biggest cardinality possible), this variable will for sure have the biggest feature importance. Because in the leaf nodes you will find every individual with his social sec. number and his output. But the capacity of generalization of the model is zero.

Almost every task in data science looks that this doesn't happen.


Can I use survival analysis model to predict probability of an item sold

You can. You should be able to create a survival function for each type of item to give a breakdown of the probability of it being sold by that point in it's lifecycle.
An issue you might run into is that the interpretation of the survival function is not the same as your outputted probability in the classification problem. In the classification problem you are given a window: "I predict that this item will be sold within X days" (determined by your "reasonable amount of time" window). The survival analysis curve however will only tell you the probability of your item being sold/not-sold at this point in time relative to it hitting the shelves. If someone asks you "when will this item be sold?" or "will this item be sold this week" you might have an easier time answering this question via the classification framework in comparison to the survival analysis approach.
In my experience: if you're interested in a predictive model the classification framework has more direct business application (depending on your selected window size). If you are simply looking to describe the lifecycle of your products & communicate a point regarding shelf life to some target audience you can probably get by with survival analysis.


Detecting outlier with combining two vectors

Autoencoder Solution

You could try an autoencoder. The autoencoder would take an input vector and it would try to recreate it as an output. So you take your input, and measure the distance between the input and the predicted output variable, using your metric of choice (euclidean should work but try various). Larger distances can be thought of as more abnormal. So you can stack rank your observations from weirdest to most normal.

Make sure that you are only training the autoencoder on normal data though. This would of course assume that you have more than the 13 samples that you are looking at. If not this probably isn't going to work very well, just because of the small sample.

KDE Solution

The idea is to use Kernel Density Estimation to generate a non-parametric joint density of your data set. You then find what the probability of finding a value that extreme would be. Here is some code using python's sklearn package:

from sklearn.neighbors.kde import KernelDensity
import numpy as np
X=np.matrix([[1.3269, 1.3354, 1.3318, 1.3282, 1.34666, 1.3460, 1.36084, 1.3526, 1.3539, 1.3510, 1.3480, 1.3479, 1.34893],[0.0352, 0.0992, 0.1570, 0.1431, 0.1634, 0.1629, 0.1046, 0.1655, 0.1635, 0.1642, 0.1658, 0.1666, 0.15735]])
kde = KernelDensity(kernel='gaussian', bandwidth=.45).fit(X.T)
score=kde.score_samples(X.T)
prob=np.exp(score)
print(prob/prob[6])

This code shows that the observations in the lowest probability density areas are observations 1,2 and 7. Of course this would work better with a larger sample, and you need to fuss with the bandwidth to calibrate it, but that this should do it.


Why is the reported loss different from the mean squared error calculated on the train data?

That's because the square loss is defined as 0.5*MSE.

See definition here:


How do I add together multiple columns based on header names?

Well after much searching and a lot of errors I have the below working code

df['Tier 1'] = df.groupby(df.columns.str.contains('Performance'),axis=1).sum()[1]


Difference between Gradient Descent and Normal Equation in Linear Regression

To train a model, two processes have to be followed. From the predicted output, the error has to be calculated w.r.t the real output. Once the error is calculated, the weights of the model has to be changed accordingly.

Mean square error is a way of calculating the error. Depending upon the type of output, the error calculation differs. There are absolute errors, cross-entropy errors, etc. The cost function and error function are almost the same.

Gradient descent is an optimization algorithm or simply update rule, used to change the weight values. Some of the variations are Stochastic gradient descent, momentum, AdaGrad, AdaDelta, RMSprop, etc. More about Optimization algorithms


Why $L2$ loss is strictly convex if number of samples $N$ is larger than input dimension $d$?

I think an additional assumption of linear independent inputs is necessary to have a strictly convex loss function.
Linear independence with $N>d$ gives you $Xv \neq 0 ~ \forall v \neq 0$, which implies that $X^\top X$ is positive definite, as $||Xv||_2 > 0 $.


Translating a Business Metric into an ML Accuracy Metric

The purpose of this algorithm appears to be about reducing cost. So a metric which includes the financial risk associated with defaults should be preferred over one that simply measures number of defaults.

A weighted false_positive * loss_on_default metric would seem to be ideal, where you set the probability cutoff of positive class (auto-approve) so that 80% of loans were approved regardless of absolute probability of the prediction. Conceptually this is similar to picking a point on the ROC curve and assessing the algorithm at that point.

This metric could not be used directly as an objective function, you'd probably use want something like logloss weighted per sample by the loss_on_default in order to train a model.

I did a quick search for similar metrics, and could not find anything (it is also not something I have done before, so do please test the idea carefully, it is mostly conjecture on my part). There are plenty of metrics in e.g. SciKit learn however that take an array sample_weight that does something similar - e.g. Sklearn's Zero-One loss is pretty close except it will include false negatives as well.


Regression - How Random Forest and Gradient Boosting really works?

Random Forest:  Build a decision tree.

Sample N examples from your population with replacement (meaning examples can appear multiple times).
At each node do the following

Select m predictors from all predictors
Split based on predictor that performs best via some objective function
Go to the next node, select another m predictors and repeat

Combine all trees as an average or weighted via some scheme.

Gradient Boosting

One key note is that random forest trees essentially indepedent of each other.  Boosting algorithms add a certain depedency to the model.

Initialize a model by finding the minimizer of a certain objective function
For each iteration

Compute the partial derivative of -$L(y_{i}, F(x_{i}))$ with respect to $F(x_{i})$ for all i to n
Fit a tree $h_m$ to the the result from above.
solve $\lambda_m =arg min \sum_{i=1}^{n}L(y_{i}, F_{m-1}(x_{i}) + \lambda h_{m}(x_{i})$

Update model via  $F_m(x) = F_{m-1} + \lambda_m h_{m} (x)$


K-means is the wrong algorithm for this problem.

It assumes dense, continuous input data, which you haven't.


You could use doc2vec to create vector representations of each document. Once you have all the vector representations you can use standard unsupervised clustering techniques like k-means, hierarchical clustering, or K-SOM.

The doc2vec model you create will be able to compute cosine similarity between two documents and also find the n most similar documents to a given document and provide a similarity score for each.


How does Performance function classify predictions as positive or negative? Package:ROCR

I cant access an R console at the moment to check, but I'm quite certain the cutoff is 0.5: if your glm model does prediction, it first produces real values and then applies the link function on top. To the best of my knowledge, you can't change it inside the glm function, so your best bet is probably to check ROC, find what the optimal threshold is and use that as cutoff.


Eigen Decomposition of Data Matrix for PCA

Some Intro

Eigen-Decompostion does lots of things among them one thing is interesting. Eigen-Decomposition usually captures the characteristic of the matrix it is applied to. For instance is the matrix is the matrix of similarities, the result of eigen decomposition will be a clustering (see Spectral Clustering).

PCA

The idea behind PCA is to find coordinates on which the intrinsic variance of data is maximized. Well ... according to Intro, if I have a matrix encoding joint variance (co-variance) of all pairs of variables, then I can use eigen decomposition to simply capture this property for me! This is what PCA does.


Learning time of arrival (ETA) from historical location data of vehicle

Based on what I figured out from your problem:

1

You can easily convert your data to a graph using Networkx, igraph or any other tool/library/software. Then what you need is a Shortest Path Algorithm (Dijkstra is widely used and implemented in all graph/network analysis softwares). Once you created the graph you can simply calculate the average estimated time.
For turning the problem into a Learning Problem, you can use historical time estimations for different paths and assign a weight to an edge proportional to the property of that edge (e.g. traffic jam probability, time conditions) and try to predict the ETA for a new query.

2

You can also turn it into a Network Science Problem and use Graph Theoretc approaches to approach the question. You can start with statistical analysis of nodes and edges attributes e.g. passing time distribution, shortest path length distribution, probabilistic modeling of traffic jam and so on to see if some meaningful insight leads you the next step.
Another idea is to use graph clustering algorithms to extract most connected parts of the town and go through the analysis of them i.e. calculate the ETA for different parts instead of whole the data and assign the estimated time to the members of corresponding cluster and reduce the computational complexty if your algorithm.

3

The last but not least is having a look at ArangoDB. It's a new database model which is based on graphs and you can run queries on millions of edges in an amazing speed! all what you need is a bit of javascript knowledge and even if you don't have it you can use AQL language designed for arangoDB. The interesting point is that it uses JSON files as the standard data format so you are already half way through ;)

Hope i could help :)
Good Luck!


Although your question is not very specific so I'll try to give you some generic solutions. There are couple of things you can do here:

Check sparseMatrix from Matrix package as mentioned by @Sidhha
Try running your model in parallel using packages like snowfall, Parallel. Check this list of packages on Cran which can help you runnning your model in multicore parallel mode.
You can also try data.table package. It is quite phenomenal in speed.

Good reads:

11 Tips on How to Handle Big Data in R (and 1 Bad Pun)
Why R is slow & how to improve its Performance?


Regression coefficient(s) when explanatory & response variables are time series

You should also have a look at the Cross Validated forum on StackExchange a lot questions asked there are related to time series analysis.

What you are looking for seems like a "Dynamic Regression or "Intervention Analysis" https://www.otexts.org/fpp/9/1 (with examples in R)

An here https://onlinecourses.science.psu.edu/stat510/node/72 you will find a procedure which does an ordinary linear regression but investigates the autocorrelated errors caused by the time series structure of your data (also with an example in R)


So the only real reason for using pooling is to reduce to the size of the feature representation thus leading to smaller memory and computational footprint as the networks deeper. yes, cost efficiency but also:

Generalisation: We get rid off small--unimportant details when we combine several values into one representative one. Hence what you really get is reduced chance of overfitting


How to use NLP to determine the normal words in the text

You can probably train a classification model to classify normal words from bad words, using character subsequences as features in the words. For example, exe can be a feature for distinguishing a normal word from a bad word in your case.


In calculating policy gradients, wouldn't longer trajectories have more weight according to the policy gradient formula?

wouldn't longer trajectories get more weight?

Not necessarily. Gradient $\triangledown_{\theta}$ could be negative or positive (1D analogy), therefore, larger number of gradients could have a smaller weight, which makes sense. A consistent short trajectory is more informative (has more weight) than an inconsistent long trajectory with sign-alternating policy gradients.

  Why would it work like that?

If we are comparing two consistent trajectories, where most gradients are in the same direction, this formula makes sense again. A long consistent trajectory contains more useful information (more steps that confirm each other) than a short one. In real life, compare the informativeness of a successful week to a successful year for your policy learning.


After a Feature Scaling do i have the same initial information?

I assume by mean normalization, you mean scaling each feature by subtracting the mean and dividing by standard deviation:

$$
x_{\text{scaled}} = \frac{x - \bar{x}}{\sigma}
$$

where $x$ is a feature.

Even though you are changing all of the values of $x$, they are each being scaled by the same amount - initially, a translation by a constant (subtracting the mean), and then scaling by a constant (dividing by standard deviation).

Here's a two-dimensional, randomly generated dataset generated with scikit-learn's make_blobs function (left) and the scaled version using the above equation for the $x$- and $y$-coordinates (right):

The $x$- and $y$-values for each point have changed, but they are still all in the same place relative to each other. If you look closely, you can see that the structure of the data is identical, even though it has been scaled a bit, and you could go back to something that looks more like the original by simply 'zooming in' on the data. Because the structure of the data is the same, we say no information was lost.

Now consider a transformation where we only take the $x$-value, and set all $y$-values to $0$:

The structure of the data has changed, and there is no way to return to the original data by scaling or stretching space uniformly, so we say information was lost here. This is an extreme example, but hopefully it illustrates the point.

One way to think about it is to think if it is more or less difficult for a classifier to distinguish between the classes after the transformation. In the first case, we can draw a line that perfectly separates the two clusters just as easily with the original data or the normalised data, but in the second case, there is no such line that separates the transformed data.

By the way, if you normalise each example rather than each feature (as asked in your comment), for this data, you end up with something that looks like this:

where all points land on either $(-1,1)$ or $(1,-1)$. This makes sense, because normalisation makes the range of the values span from $-1$ to $1$. When there are only two dimensions, one of them has to become $-1$ and the other has to become $1$. Hopefully it's fairly obvious that information is lost here, and it's generally not a good idea to do this.

This is quite a hand-wavy explanation and doesn't really cover any actual information theory concepts, but hopefully it gives you some intuition for this. If you want to dive deeper into the mathematical side of things, have a look at the Wikipedia article for information theory.


(pre-trained) python package for semantic word similarity

The spaCy Python package might work for you.  It allows you to easily "install" large pre-trained language models, and it provides a nice high-level interface for comparing word vectors.

To install spaCy:

pip install spacy

Then you need to download a language model.  I believe these models are trained on Common Crawl, which is a massive dataset.  You should choose one of the medium or large models, because the small models do not ship with word vectors.

python -m spacy download en_core_web_md

Using spacy models to compute word similarity is a breeze:

import spacy

# load the language model
nlp = spacy.load('en_core_web_md')

word1 = 'cat'
word2 = 'dog'

# convert the strings to spaCy Token objects
token1 = nlp(word1)[0]
token2 = nlp(word2)[0]

# compute word similarity
token1.similarity(token2)  # returns 0.80168

Here's an example that's more similar to the one in your question:

import spacy

nlp = spacy.load('en_core_web_md')
token = lambda word: nlp(word)[0]  # shortcut to convert string to spacy.Token
score_words = lambda w1, w2: token(w1).similarity(token(w2))

score_words("pilot", "airplane")      # 0.5998
score_words("student", "university")  # 0.7238
score_words("cat", "dog")             # 0.8017
score_words("cat", "airplane")        # 0.2654
score_words("student", "apple")       # 0.0928


Estimating the battery capacity using current power consumption and battery percentage

Firstly, a common chemical battery discharging from 95% to 35% in 1 second is undergoing a very deep discharge and its discharge profile will be very different from a slow steady discharge, e.g. 95% to 35% within 10 hours. Secondly, inferring from a very small snippet of the charge process (2% change) may work only if you know exactly which charge profile is being exhibited by the battery during that charge cycle, so that you can identify at which point of the charge cycle is the battery right now.

For most chemical batteries, the current consumption by the load determines the discharge profile and hence the capacity at a given point of time in the future. Since this is usually non-linear, and the slope varies at different points of the discharge profile, a linear regression would not be a good fit. I would recommend trying out a polynomial regression fit on the discharge profile. Multivariate adaptive regression splines should also work well. Similarly, the rate of charging determines the charge profile/characteristic curve.

You could fit multiple models, one each for each type of charge profile (e.g. deep discharge, slow discharge, etc.) and then select the appropriate model based on the current status of the battery, to predict the future state.

To understand this process for a lead acid battery, refer to these -

http://scubaengineer.com/documents/lead_acid_battery_charging_graphs.pdf and,
http://batteryuniversity.com/learn/article/how_to_measure_state_of_charge.


You can extract features from timeseries sliding window, eg. mean, std, etc etc, and then use these as features into your model for anomaly detection


Use strsplit().

If you want to remove the space between the two words (along with 'NSW'):

city_clean <- unlist(c(strsplit(city, " NSW")))

Output: [1] "Sydney"      "Newcastle"   "Liverpool"   "Broken Hill"

It wasn't clear to me whether you wanted to keep the space or not. If you want to keep the trailing space after the city name

city_clean <- unlist(c(strsplit(city, "NSW")))

Output: [1] "Sydney "      "Newcastle "   "Liverpool "   "Broken Hill "

If you prefer to use regex, here's another solution:

city_clean <- gsub(" NSW", "", city)

Output: [1] "Sydney"      "Newcastle"   "Liverpool"   "Broken Hill"


Should I only use scalar labels with Keras LSTM?

Can you post a model summary using:

model.summary()

Also, elaborate on how exactly the Y_train dataset works with the X_train?  It's not clear how the 25 time steps from X_train data correspond to the Y_train 5 outputs.


How to integrate time series data into normal features for machine learning?

Assuming: DF1 and DF2 look something like:

So going to make some assumptions about what you want to do:

(1) you want a classification model like: I want to classify buyers/non buyers based on their overall attributes as well as their historical activity for a given time window:
- then you first have to create features based on the transaction data frame DF2, which would rollup transactions and performance information to customer id level for each customer. Some metrics that you can create from the detail transactions table DF could be recency of visits,  frequency of visit, average purchase amount, recency of conversions etc.., so you would have customer id's that did not buy anything over the time period and customer that bought at least once over the time window.

join the first table and the rolled up data by customer id to have all features and label (buyer,non buyer) given an appropriate time window

(2) if you want a model for classification at each timestamp for each customer id then most likely your first data frame contains some sort customer characteristics and demographics that do not change overtime or at least over the chosen time window. In that case you join the two data frames by customer id. Here each customer could be buyer or non buyer depending on the each timestamp


How to create a multiple layer perceptron with layers of specific sizes in keras?

There may be quite a lot of reasons for this. As I understand you try to repeat someone's results, and all you know is the architecture. Your keras architecture seems correct, but you should also take into account:

Data - did the authors used data augmentation?
Regularizations, such as dropout, L1, L2, batch normalization - that can significantly influence your results.
Other hyper-parameters, like optimizer, batch size etc. Did you do a grid/random search of those?

If you took all this into account, provide us with more complete training code, so we don't have to guess.


How to find splits in data so that each split has equal weighting according to function f

A classic optimization problem! You can use Linear Programming/Optimization to find a good split. Every of the n samples s $\in S$ has weight f(s) and we want to divide them into m folds. You can use a trick to linearize the L1 objective or you can use Quadratic Programming for a L2 objective function. The Quadratic Programming model is easier to define in this case. Let's define $x_{ij}$ as a binary decision to put sample i in fold j and $\mu$ is the ideal, mean weight per fold. Then this is our objective function:

min $\sum_{j=1}^m(\sum_{i=1}^nx_{ij}f(i)-\mu)^2$

Under the following constraints:

$\sum_{j=1}^mx_{ij}=1$ for all $i\in \{1..n\}$ to ensure exactly one assignment per sample

$x_{ij} \in \{0, 1\}$ to turn it into binary decision varibales

Depending on the size of your dataset and the solver you use this can be a heavy optimization, but there are a lot of greedy heuristics that will get you close fairly fast.


Can we access HDFS file system and YARN scheduler in Apache Spark?

HDFS

Spark was built as an alternative to MapReduce and thus supports most of its functionality. In particular, it means that "Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc."1. For most common data sources (like HDFS or S3) Spark automatically recognizes schema, e.g.:

val sc = SparkContext(...)
val localRDD = sc.textFile("file://...")
val hdfsRDD  = sc.textFile("hdfs://...")
val s3RDD    = sc.textFile("s3://...")

For more complicated cases you may need to work with lower-level functions like newAPIHadoopFile:

val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat],
      classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],
      classOf[org.apache.hadoop.hbase.client.Result])
val customRDD = sc.newAPIHadoopRDD(conf, classOf[MyCustomInputFormat],
      classOf[MyCustomKeyClass],
      classOf[MyCustomValueClass])

But general rule is that if some data source is available for MapReduce, it can be easily reused in Spark.

YARN

Currently Spark supports 3 cluster managers / modes:

Standalone
Mesos
YARN

Standalone mode uses Spark's own master server and works for Spark only, while YARN and Mesos modes aim to share same set of system resources between several frameworks (e.g. Spark, MapReduce, Impala, etc.). Comparison of YARN and Mesos may be found here, and detailed description of Spark on YARN here.

And, in best traditions of Spark, you can switch between different modes simply by changing master URL.


If I have negative and positive numbers for a feature, should MinMaxScaler be -1 to 1?

In my opinion, it depends on the importance of the meaning of negative sign.

If the sign of the value indicates the direction of the value, then it will be better to retain the negative sign. For example, +1 indicates moving rightward and -1 indicates moving leftward.

In the other hand, if the the negative sign only means it is smaller than 0, it will be fine to normalize them to 0 to 1. For example, the variable recording temperature, negative values only means it is lower than positive values.


Assuming $\tau = (\mathbf{s}_1,\mathbf{a}_1,...,\mathbf{a}_T)$ is a sampled trajectory from $\pi_\theta(\tau)$ with return $r(\tau)=\sum_{t=1}^{T}r(\mathbf{s}_t,\mathbf{a}_t)$ then we define the expected reward (our cost function) as:

$$J(\theta)\equiv E_{\tau\sim \pi_\theta(\tau)}[r(\tau)] = \int \pi_\theta(\tau)r(\tau)d\tau$$

where $\pi_\theta(\tau)$= $\pi_\theta(\mathbf{s}_1,\mathbf{a}_1,...,\mathbf{a}_T)=p(\mathbf{s}_1)\prod_{t=1}^{T}\pi_\theta(\mathbf{a}_t|\mathbf{s}_t)p(\mathbf{s}_{t+1}|\mathbf{s}_t,\mathbf{a}_t)$.

Notice that the p.d.f $\pi_\theta(\tau)$ here  is a joint probability distribution over states and actions. Thus the expectation comes from:

$$E[X]=\int_{-\infty }^{\infty} \int_{-\infty }^{\infty} xf(x,y)dxdy$$

and this is the reason that you need to integrate over the volume of states and actions. $p(\mathbf{s}_1)$ is a distribution over initial states and independent of the integration over actions so it can get out of the 2nd integral. Taking the log of the joint distribution $\pi_\theta(\tau)$, substituting into $J(\theta)$ and differentiating both sides of the first equation,  you get an equation w.r.t the $\theta$ parameters and quantities that you can compute so you can estimate the gradient of $J(\theta)$.

I am not very familiar with your notation (i think you found it in a master thesis) but this is the general approach.


shape of theano tensor variable out of keras Conv2D

You can't get the shape of a theano tensor, because it is not fixed. The output of the convolutional layer is just a symbolic variable and its shape depends on whatever you put into the layer as input.

You can get the shape of the output for a specific input by making a theano function for the output of the layer, and feeding a numpy array through the function:

import numpy as np
input = np.ones(28*28).reshape(1, 1, 28, 28).astype('float32')
fn = theano.function([input_img], x)

print fn(input).shape
>>> (1, 16, 28, 28)


Can you try this:

df1 = df.filter(df.AppName=="23954ec32332dfgd")
print df1.count()


Where do I start? I have a massive dataset (from web scraping) and want to predict y from 20 variables

Since you are really new to all this, here are some suggestions.

Download and install R and RStudio Desktop.

Look at the book "Introduction to Statistical Learning". The book comes with code examples (in R) which are ideal to start learning. For you it will be useful to have a look at Chapter 2 to understand what statistical learning is all about and also to get a good idea of how to use R.

However, what is really relevant for you is Chapter 3 (linear regression) and Chapter 7 (moving beyond linearity).

Now you can get started. Here are some examples how regression can be done in R:

# Install a package if needed
# install.packages("ISLR")

# Lead packages
library(ISLR)
library(gam)
library(Metrics)

# Look at data
head(mtcars)
# Look at one variable in our dataframe
summary(mtcars$mpg)

# Make a test/train split
set.seed(123)
smp_size <- floor(0.75 * nrow(mtcars))
train_ind <- sample(seq_len(nrow(mtcars)), size = smp_size)
train <- mtcars[train_ind, ]
test <- mtcars[-train_ind, ]

# Linear regression
# Estimate mpg (y) dependent on ...
reg1 = lm(mpg~disp+wt+qsec+hp, data=train)
summary(reg1)

# Regression with polinomials
reg2 = lm(mpg~poly(disp,2)+poly(wt,2)+poly(qsec,2)+poly(hp,2), data=train)
summary(reg2)

# Generalised additive models (regression splines)
reg3 = gam(mpg~s(disp,3)+s(wt,3)+s(qsec,3)+s(hp,3),data=train)
par(mfrow=c(2,2))
plot(reg3)

# Predict mpg
pred1 = predict(reg1, newdata=test)
pred2 = predict(reg2, newdata=test)
pred3 = predict(reg3, newdata=test)

# See how well our model fits
mae(test$mpg, pred1)
mae(test$mpg, pred2)
mae(test$mpg, pred3)


You've passed an objective function reg:linear in space; there is no such objective function.  The objective reg:squarederror that you pass to cv should be fine.


Noisy behavior in deep learning feed-forward net

The overal logarithmic shape of your f1 score graph indicates that learning is effective and cost is heading towards some minimum. That's good. I'm assuming that the noise you're referring to is the instability of the graph after approximately 3k iterations: cost dropping and rising in a zig-zag manner.

This often hints at the learning rate being too large. Back propagation finds the right gradient but you take too big step and end up climbing rather than descending along the edge of the cost function. It's especially evident when a graph seems to oscillate around some middle value. You haven't mentioned what exact value of learning rate you're using but try to reduce it. A good starting point would be 0.01 but it depends on many factors so try to experiment.

Another issue might be a batch size: that is, how many examples contribute to the calculation of a gradient. If it's too large you might end up with an average gradient pointing in the wrong direction. And then even a small step (i.e., low learning rate) won't help. And it might again manifest itself in a zig-zag pattern. If batch size is one of the parameters try to decrease it.

The least likely issue might be the encoding architecture of your network. And especially the modest number of 8 neurons on the last layers. In this case individual neurons might have a considerable impact on the final output. And even little adjustments resulting from a single step of back propagation could potentially flip the sign of that neuron's activation value, impacting the results of other examples. Try increasing the number of neurons on the last layers. I'd personally suggest trying an architecture of 50x50x50.

Hope this helps!


Neural Network for Multiple Dependent Outputs

In many cases, at the end of a neural network you could find a Softmax layer, which outputs probabilities, so they add up to 1. It seems exactly what you're looking for. Please find more information about softmax function here or here.
Hope it helps.


I am able to load the data set like so:

 dev <- read.table("iran_it_status_1394_detail_data_jadi_net.tsv",
                   header=TRUE, sep="\t", blank.lines.skip = TRUE,
                   na.strings="",
                   stringsAsFactors=FALSE, skipNul = TRUE, fill=T, quote="")

Note the removal of the encoding (so that the function "finds" lines in the file), the fill attribute (to allow for a ragged table with empty cells), and the elimination of quotes (apparently there is a misquoted line somewhere near line 585).

This results in a table full of encoded characters - you'll need to know more about the source data to figure out how to work with it, but if you open the file up in a raw text editor (e.g.: Sublime text) you might get some clues:


Python and Titanic competition how to get the median of specific range of values where class is 3

If I understand the problem correctly, you want to fill all missing values in the Fare column by the median value of the Fare column where Pclass=3. This can be achieved by putting the extra row filter test['Pclass']=3 on median of the fare column, see below.

test['Fare'] = test['Fare'].fillna(test.loc[test['Pclass']=3,'Fare'].median())


Why do Isolation Forest implementations turn it into a supervised learning problem (with random values for the target?)

Extra-random Trees needs a target variable, so Isolation Forest generates a random target (sklearn, solitude).  At prediction time, no y values are used, and the ExtraTrees doesn't actually make a prediction; instead, the samples are propagated to the leaves and the depth is extracted (sklearn).
As for the tree-building process, sklearn at least doesn't make use of the y values, because the ExtraTrees model has max_features=1 and splitter='random' (source).  I'm not so sure about solitude, since it has mtry=ncol-1 (source); maybe further down, using splitrule='extratrees' takes care of that?  Otherwise, the splits chosen will try to optimize on the random y, though since those are random it maybe doesn't matter (certainly I wouldn't call it a supervised model, anyway).


is perceptron based implementation of Logical conjunction (AND) regression or classification

AND operator gives you answer of True or False, it is binary in nature.

Hence, it is classification in nature.


OpenAI has not released the weights of GPT-3, so you have to access it through their API. However, all other popular models have been released and are easily accessible. This includes GPT-2, BERT, RoBERTa, Electra, etc.
The easiest way to access them all in a unified way is by means of the Transformers Python library by Huggingface. This library supports using the models from either Tensorflow or Pytorch, so it is very flexible.
The library has a repository with all the mentioned models, which are downloaded automatically the first time you use them from the code. This repository is called the "model hub", and you can browse its contents here.


I think you need to come up with a way to treat the data such that you're thinking in days, not hours, right? The peaks look they're just at 24, 48, 72, 96, (1 day, 2 days, 3 days, 4 days) etc, and are pretty much normally distributed around those peaks.

I think a good test might be to try a categorical approach to start, to see how well you can predict which 'peak' the ad belongs to (is this an ad in the 24-hour normal distribution? the 48-hour?). If you can figure out which peak that ad belongs to then you might be able to identify features that indicate whether it's more likely to be on the short- or long-side of the hump. If you have bad results putting the ads into categories that might tell you something too.

If you do try this, be careful of how you measure performance as your dataset will be unbalanced by the higher occurrence rate of quickly-pulled ads.


review: gradient descent, epochs, validation in neural network training

Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.

Typically, validation is done only at the end of the epoch. So no batch should remain unused. For example, this is the behavior of Keras https://keras.io/models/model/#fit

validation_freq: Only relevant if validation data is provided. Integer or list/tuple/set. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g.  validation_freq=2 runs validation every 2 epochs. If a list, tuple, or set, specifies the epochs on which to run validation, e.g. validation_freq=[1, 2, 10] runs validation at the end of the 1st, 2nd, and 10th epochs.

  hyper parameters: number of ConvNet filters, size of ConvNet filters,
  number of layers won't be trained or tuned?

Yes, such hyper parameters are set when model is defined.  These are not modified during training.

However : Systems like Google AutoML are trying to change this. These systems try to learn model hyper parameters during training.


LSTM model prediction is almost constant

It could be a number of different reasons but when I had that problem in the past it was usually due to too high of a learning rate or the optimizer. I recommended either dropping the initial learning rate or going with vanilla SGD. Occasionally I saw problems with Adam particularly if you have no warmup. You might want to try more general hyperparameter search techniques as well and iterate over things like batch size, forecast length, and lr etc


Is it ok to trust regression predictions when none of the coefficients are statistically significant?

It depends on the goal of the project.
Statistical significance is more important for publishing in academic journal. For applied projects, statistical significance is less important.
Even if a model is not statistical significant, it can have value. The model may summarize the data and identify relationships. The model maybe practically important, the effect maybe of value to practitioners.


Any pythonic way to auto determine imbalance class problem, specially in multiclass scenario?

Depending on your dataset, I would say you can set the threshold yourself.
For example, a 50% split is a perfect balance so by definition any deviation away would be an imbalanced problem. That being said, this is quite typical, so it would make more sense to set your threshold to be somewhere around 80% imbalance or higher.
Count your classes and set a threshold, e.g.
threshold = 80
imbalance_measure = (minority_class/majority_class)*100
if imbalance_measure >= threshold:
    print("Imbalanced problem")
else:
    print("Balanced problem")

In fact, if you data is suitable you can test this yourself through cross-validation
Some further relevant documents here


classification algorithms for anomaly detection when there is class imbalance

Here is a pretty detailed summary on Handling Imbalanced Classes. Since you mentioned R, you might want to take a look at unbalanced package.


Is there a way to add new feature to existing machine learning model and retrain it?

So the question asks if you can retrain the model after adding another feature.
Answer is yes but you need to instantiate a new model which can accept a 3-feature/dimensional input.
This is due to the mechanics of neural networks whereby in the 1st layer, the features in your original model would be multiplied by a ($2 x n$) matrix where $n$ represents the number of hidden neurons in the second layer of the model.
Therefore by multiplying a ($3 x 1$) input with the weight matrix will throw an error.


Can a decision tree learn to solve a xOR problem?

Yes, a decision tree can learn an XOR.

  I have read online that decision trees can solve xOR type problems...

Often things are phrased not carefully enough.  A neural network can perfectly sort a list of integers, but training one to do that would be quite hard.  Your image shows that a tree can easily represent the XOR function, but your question is how to learn such a tree structure.

  My question is how can a decision tree learn to solve this problem in this scenario. I just don't see a way for any metric (Information gain, Gini Score, ...) to choose one of the splits in image 2 over any other random split.

Indeed, the first split is probably quite random, or due to noise (if you go for $\operatorname{sign}(x\cdot y)$ with continuous $x,y$ instead of the discrete $x,y$ and XOR).  But, as long as your algorithm makes the plunge with one of those first splits, the next splits are obvious and your tree will make them.

  Is it possible to solve the presented problem with a decision tree?

Here's a notebook (github/colab, suggestions welcome) demonstrating that yes, a (sklearn) decision tree can learn $\operatorname{sign}(x\cdot y)$ (perhaps with some errors when points are extremely close to 0); but it also goes on to show some of the difficulties, e.g. when variables other than $x,y$ are available to the tree to split on.  Up-shot: noise variables can wreck that first split I mentioned above, and even useful variables can make the tree lose track of the XOR.

  Would using a random forest solve the problem in any way?

Probably not the basic problem, but it looks like it helps with, e.g., the noise variables above.


Dropout in other machine learning models

Random forests could be thought of as using a kind of dropout-esque technique as each split node only considers a random subset of the features, effectively 'dropping out' the other ones. Also, sometimes in large tree ensembles, each tree is only given a random subset of features to begin with, akin to dropout on the input layer of a neural network.


Is there a different between one hot encoding and labelling categorical as scalar value and then normalise the batch?

The second option is wrong. If you label Male, Female, Undefined as [0, 1, 2] you are treating categories as numbers. That is not correct. The Network in this way would learn something such as Undefined > Female, which doesn't make sense.

You need one hot encoding to treat those variables: they are not a numerical values that you can put on a scale.

About the Age question: if that's a numerical variable (expressed in years, I guess?) then you can feed it directly into the Neural Network without problems.


Face detection for different poses more robust than MTCNN?

If anyone is looking for something that is more robust than MTCNN, try this but you need GPU with Cuda I believe.
It says Tesla P40 on requirements but that is for training only.
I could infer on new images with the provided parameters on a GTX 1080 with Cuda.
It's called DSFD from Tencent in China.  Their results on some well known benchmark datasets surpass MTCNN.
I can concur that it detected the above-mentioned side faces (not profile but lying sideways).
It is on Github:  DSFD Tencent
There is a demo.py where you can designate your own image as an arg on command line.  If you figure out how to use this with CPU only, please let me know.


This is a hyperparameter. And so this is usually approached with some sort of hyperparameters search with cross-validation, like GridSearchCV or RandomizedSearchCV.
https://scikit-learn.org/stable/modules/grid_search.html


What is the value of AIC criterion if RSS is 0?

RSS = 0 implies that the model is a perfect fit since there was no residual. The limit of the log of 0 is $-\infty$, and since lower AICs are better, and this model is perfect,it makes perfect sense that the AIC should be a negative number such that no number can be lower.


A good answer to this question has to rely on the specific dataset / domain.

The questions I would ask myself are (in this order):

Can I solve my classification task without those features? → Just remove those features
Do I know of a relationship between features I know and the (partially) missing feature? → Find hard rules to fill those missing ones
Otherwise: Train a classifier / regression model to fill those missing features


What does it mean when the coefficient of the squared term is negative in regression?

In this context probably the plain English way to put it is that 'returns increase with additional exposure to the ad, but there is a tapering effect at the upper end of exposures. Looking at this picture of a parabola here (i.e a graph of y= -ax^2+bx+c):

Most likely your returns data are between 0 and the peak so you don't actually see  a decline, just a reduction in the rate of increase.


1. Why are the q-values of different actions very close to each other for a given state ?

I'm going to explain this with a small example.

Consider the game of "Catch". Fruits(circular) keep falling from the top of the screen (vertically) and the agent(square) needs to just align itself to the fruit to get the reward. There are three actions that it can take : move left , stay , move right. Let's say that a2 refers to not moving the paddle, a3 refers to moving right and a1 refers to moving left. The image on the screen is to be used as the state in the MDP formalism.

Let's say we take a sub-optimal action a3(move right) and move to the next state. Then the best action in that state would be to move left (a1) and then execute the optimal action. So, the only cost difference between the actions a2 and a3 will be the two steps wasted to go and come back.

If there is no negative reward for taking the sub-optimal action, then the agent has no incentive to choose the optimal action. So, the negative rewards for taking the sub-optimal action should be high enough that the agent is discouraged from doing it. I've tried to put this intuition mathematically here. This could explain why q-values are so close to each other.

Then, the optimal Q* function satisfies the following :

2. Is there a bound on this gap ?
The bound that I've shown is approximate. The technical term for this difference is "Action Gap". A much better analysis is available here : Increasing the Action Gap: New Operators for Reinforcement Learning

3. Why are the q-values of different states very close to each other for a given action ?

This should be fairly obvious. Consider two different cases. In the first one, the fruit falls from the top left corner and in the second one, the fruit falls from the top right corner. The agent in both cases is at the center. Then, consider the action of moving left. This action will give a high q-value in the first case and a low q-value in the second case.

4. How to solve the action gap problem ?

Let's define the average reward in a state to be $V$. $V$ is approximately the average of Q-values of all the actions in a given state. If we now subtract this $V$ from each of the $Q$ values, we get a quantity which describes how good the action is compared to the average of all the action. This is called as the advantage of the action. In some sense, this is approximately the action gap. So, instead of predicting the q-values, we are directly predicting the action gap itself.

Link : A3C paper


VC-dimension of the infinite set of convex bodies?

For a binary class of data and any number of instances, as you have an infinite number of convex bodies, you can put all instances of a class inside the body and leave others. In this way, you can shatter any set with any arbitrary size. Hence, the VC dimension of an infinite number of convex bodies is infinite.


Do I need to engineer lagged features when creating an LSTM for time series forecasting?

TL;DR

No, you don't have to include lagged variables when using an LSTM.

Long Answer

In an LSTM architecture, your neurons have not only an input gate and an output gate, but they also have a forget gate which causes them to remember values over multiple time intervals. So, you don't have to include any lagged variables in your feature set, since you can expect the LSTM to discover relevant time-depended relationships on its own.
This is not only the case for LSTM's but also for other RNN-architectures.


You can't assume the loss wouldn't drop with additional cycles, but it's generally safe to assume there are diminishing returns. This is why stopping rules are defined in terms of both number of iterations and change in the loss function. You want to stop when the gains become vanishingly small. At that point, increasing your accuracy from, say, 90.1% to 90.2% probably have little practical benefit, and performance on a hold-out sample probably won't differ much. But, that is an empirical question you can actually test. Run a model to near completion then run another with more stringent stopping criteria and compare on a hold-out.


Your issue might occur if you didn't tell that the second row in the table should also be considered a header line.
To address this, try to reset the header at the beginning.
import pandas as pd

df = pd.read_csv('YOUR/FILE/DIRECOTRY.csv', skiprows=1) // ignore the 2nd row (0-indexed)
df.rename(columns={0:'Index'}, inplace=True) // optional
df.groupby(['Occupation'])


generate a report in R where the value in each Column is to be derived from different tables in sybase databasea

Here is a good overview of recommended ways:

http://r4ds.had.co.nz/r-markdown-formats.html#r-markdown-formats

I suggest using the package flexdashboard (heading "Dashboards"). Interactivity can be provided by htmlwidgets you can find an overview here:

http://www.htmlwidgets.org/

As an example, install:

install.packages('flexdashboards') # the dashboard
install.packages('plotly') # for interactive graphs
install.packages('DT') # for interactive tables
install.packages('tidyverse') # for data transformation etc

now create a dashboard as explained on

https://rmarkdown.rstudio.com/flexdashboard/

and fill it with your graph or tables. Here I use the starwars dataset provided in the package dplyr

---
title: "Single Column (Fill)"
output:
  flexdashboard::flex_dashboard:
    vertical_layout: fill
---

### Chart 1

```{r}
library(tidyverse)
library(DT)
datatable(starwars %>% select(-films, -vehicles, -starships))
```

### Chart 2

```{r}
g <- starwars %>%
  ggplot(aes(height, mass, color = gender)) +
  geom_point() +
  ggtitle("Mass vs. Height by gender")

library(plotly)
ggplotly(g)
```

output will be a html(interactive) file with table and graph:


Not aware of research per se but I did this in C# before. The concept you need to know about to recognize only certain patterns in speech is "Grammar".

Grammars in terms of speech or text recognition consist of multiple rules and word tokens that can be identified by your speech recognizer.

As further inspiration for your idea maybe look at SpeechRecognizer and the Grammar class in the VB.net Framework.

ie:
https://docs.microsoft.com/en-us/previous-versions/office/developer/speech-technologies/hh361633(v=office.14)?redirectedfrom=MSDN


You can use the dataframe's .values method to access raw data once you have manipulated the columns as you need them.

E.g.

train = pd.read_csv("train.csv")
target = train['target']
train = train.drop(['ID','target'],axis=1)
test = pd.read_csv("test.csv")
test = test.drop(['ID'],axis=1)

xgtrain = xgb.DMatrix(train.values, target.values)
xgtest = xgb.DMatrix(test.values)

Obviously you may need to change which columns you drop or use as the training target. The above was for a Kaggle competition, so there was no target data for xgtest (it is held back by the organisers).


Using a decision tree with 3 dimensional input points

To use a single vector (p1x, p1y, p1z, p2x, p2y....) per person would be the most easy and obvious way.

Maybe, your prediction would also improve by including some other transformations of these features. But usually trees are eager to use all the information you feed them efficiently. So just a 9-dimensional vector would do.


There are two terms which relate to the number of examples while learning. Epoch and iteration. During each epoch, you feed all the examples in your training set and update the network. You can feed the data simultaneously, not for large-scale tasks, or batch by batch. Each turn you pass each batch is called iteration. Consequently, each epoch may have multiple iterations. The reason we have to pass the data multiple times is that we don't know the height of each local optimum along each axis, feature. Consequently, we use a learning rate to limit our steps to small ones and take multiple steps to get close to the desired point.

Due to the request of one of our friends I add the explicit definitions:

Epoch: It means how many times the entire dataset has been passed through the network.
Iteration: It means for each epoch, how many times you have passed the chunks of the dataset. These chunks of data are called batch and the number of times you pass them through the network is called the number of iterations.


training neural net with multiple sets of time-series data

Yes, this is a straightforward application for neural networks.  In this case yk are the outputs of the last layer ("classifier"); xk is a feature vector and yk is what it gets classified into.  For simplicity prepare your data so that N is the same for all.  The problem you have is perhaps that in the case of time series you won't have enough data: you need (ideally) many 1000's of examples to train a network, which in this case means time series, not points.  Look at the specialized literature on neural networks for time series prediction for ideas on network architecture.

Library: try Pylearn2 at http://deeplearning.net/software/pylearn2/  It's not the only good option but it should serve you well.


Periodically executing a scraping script with Python

If you are on a Unix-based OS, use a scheduler like cron to run your script periodically. cron is the most common one, and you can find many tutorials for it. I guess this is the part you are unfamiliar with.

When it executes your script, make it check the time like so: fname=datetime.datetime.now().strftime('%Y-%m-%d-%H.csv'). Open the appropriate file in append mode (with open(fname, 'a') as csv_file) ; if it does not exist it will be created. That's all there's to it!


That is the real-world! in text analysis you usually end up with a extremely high-dimensional representation. First of all, I always recommend TF-IDF over original BoW (CountVectrizer).
To end up with a low-dimensional space, you may use this naive approach which is not written efficient in computational way but the approach gives you an idea where you need to go. It will be more clear when I add explanations to the notebook soon.


How to find out what portions of an image is helping CNN to classify it

Yes, the typical approach is to obtain the saliency map of the input, which are "heatmaps" of the contribution of each pixel to the final classification.
In this free online book about Explainable ML, you can find the most relevant approaches to obtain saliency maps, like vanilla gradients, together with other pixel attribution techniques.
Here you can find an end-to-end tutorial on how to implement saliency maps with Keras.


Which parameters are hyper parameters in a linear regression?

I like the way Wikipedia generally defines it:

  In machine learning, a hyperparameter is a parameter whose value is
  set before the learning process begins. By contrast, the values of
  other parameters are derived via training.

On top of what Wikipedia says I would add:

Hyperparameter is a parameter that concerns the numerical optimization problem at hand. The hyperparameter won't appear in the machine learning model you build at the end. Simply put it is to control the process of defining your model. For example like in many machine learning algorithms we have learning rate in gradient descent (that need to be set before the learning process begins as Wikipedia defines it), that is a value that concerns how fast we want the gradient descent to take the next step during the optimization.

Similarly as in Linear Regression, hyperparameter is for instance the learning rate. If it is a regularized Regression like LASSO or Ridge, the regularization term is the hyperparameter as well.

Number of features: I would not regard "Number of features" as hyperparameter. You may ask yourself whether it is a parameter you can simply define during the model optimization? How you set the Number of features beforehand? To me "Number of features" is part of feature selection i.e. feature engineering that goes before you run your optimization! Think of image preprocessing before building a deep neural network. Whatever image preprocessing is done is never considered hyperparameter, it is rather a feature engineering step before feeding it to your model.


Why don't means of color channels average to equal flattened array mean

You just understood numpy wrong (God bless those who understood it right). The correct code should have been:

trainImgsArr = np.random.rand(17611, 224, 224, 3)
grandMean = np.mean(trainImgsArr)
meanByChannel = []
for i in range(0, 3):
    avg = np.mean(trainImgsArr[:, :, :, i]
    meanByChannel.append(avg)

Basically I am doing splicing of the array,taking all pixels of a colour channel from all images and averaging them. Whereas you did something else entirely (your results are not reproducible). Axes (0, 1, 2) does not mean the colour channels it only means the index you are referring to and performing the required operation.

Ignore the convolution part but check the subscripts of h_n-1 you can access the those positions by varying the last argument in trainImgsArr.shape.


Database for a trie, or other appropriate structure for recommendation engine

Have a look at Apache Mahout. Last version features also user-item-based recommenders.


Target Encoding: missing value imputation before or after encoding

If you want to do TargetEncoder you have to impute the missing values first.

First of all you should convert your categorical features into int, using LabelEncoder or OrdinalEncoder. I used a huge numeric value (my choice : 8888) in order to fill the NaN values, before running OrdinalEncoder. Then transform your matrix to int, it will be more efficient.
For the imputation of missing values, you may use different strategies :

2.1. . Fill with the most frequent value in each features (column) (if you did the previous stage you do not need this). If you perform TargetEncoding, imputing with most frequent generally works well.

2.2. . Fill using IterativeImputer. You can provide in the argument estimator the kernel you want to use (I tested RandomForestRegressor - for numeric features). Warning: IterativeImputer uses only regression kernels. If you did the previous stage, do not forget to add the argument : missing_values=8888.

You can apply your TargetEncoder now.


Not sure at all since I am not very familiar with these old concepts.
But I think what you call LLM (not a very common concept it seems) are algorithms that solve linearly separable data classification. SVM are algorithms that look for an hyperplane that maximize the margin with data and thus can be considered LLM. Quick reminder how a SVM solves a linearly separable problem :

We escape LLM's grasp when trying to solve a nonlinear classification problem, as represented on the left of the following picture.

SVM can still solve these type of classification with the kernel trick, that applies a non-linear function to turn the classification into a linear problem.
Hope this can help your understanding of the difference, all images are from Wikipedia. My apologies for not knowing how to resize these images :(


How to fill in missing value of the mean of the other columns?

Using a transform as follows could work:

df["budget"] = df.groupby("genres")["budget"].transform(lambda x: x.fillna(x.mean()))

The mean calculation uses only the non-null values in its calculation. So the mean of each group's non-null values are imputed to that same group's null values.

See also this question on Stack Overflow.


It is my understanding that the dataset gets updated / extended every year. It started with Phoenix, then they added Las Vegas, Madison, Waterloo and Edinburgh, and so on, for a total of 11 cities by 2017.

Basically, this year's dataset should also include all cities from previous years.


I am afraid the that besides learning rate, there are a lot of values for you to make a choice for over a lot of hyperparameters, especially if you’re using ADAM Optimization, etc.
A principled order of importance for tuning is as follows

Learning rate
Momentum term , num of hidden units in each layer, batch size.
Number of hidden layers, learning rate decay.

To tune a set of hyperparameters, you need to define a range that makes sense for each parameter. Given a number of different values you want to try according to your budget, you could choose a hyperparameter value from a random sampling.
Specifically to learning rate investigation though, you may want to try a wide range of values, e.g. from 0.0001 to 1, and so you can avoid sampling random values directly from 0.0001 to 1. You can instead go for $x=[-4,0]$ for $a=10^x$ essentially following a logarithmic scale.
As far as number of epochs go, you should set an early stopping callback with patience~=50, depending on you "exploration" budget. This means, you give up training with a certain learning rate value if there is no improvement for a defined number of epochs.
Parameter tuning for neural networks is a form of art, one could say. For this reason I suggest you look at basic methodologies for non-manual tuning, such as GridSearch and RandomSearch which are implemented in the sklearn package. Additionally, it may be worth looking at more advanced techniques such as bayesian optimisation with Gaussian processes and Tree Parzen Estimators. Good luck!

Randomized Search for parameter tuning in Keras

Define function that creates model instance

# Model instance
input_shape = X_train.shape[1]

def create_model(n_hidden=1, n_neurons=30, learning_rate = 0.01, drop_rate = 0.5, act_func = 'ReLU',
                act_func_out = 'sigmoid',kernel_init = 'uniform', opt= 'Adadelta'):
    model = Sequential()
    model.add(Dense(n_neurons, input_shape=(input_shape,), activation=act_func,
                   kernel_initializer = kernel_init))
    model.add(BatchNormalization())
    model.add(Dropout(drop_rate))
    # Add as many hidden layers as specified in nl
    for layer in range(n_hidden):
    # Layers have nn neurons model.add(Dense(nn, activation='relu'
        model.add(Dense(n_neurons, activation=act_func, kernel_initializer = kernel_init))
        model.add(BatchNormalization())
        model.add(Dropout(drop_rate))
    model.add(Dense(1, activation=act_func_out, kernel_initializer = kernel_init))
    opt= Adadelta(lr=learning_rate)
    model.compile(loss='binary_crossentropy',optimizer=opt, metrics=[f1_m])
    return model

Define parameter search space

params = dict(n_hidden= randint(4, 32),
              epochs=[50], #, 20, 30],
              n_neurons= randint(512, 600),
              act_func=['relu'],
              act_func_out=['sigmoid'],
              learning_rate= [0.01, 0.1, 0.3, 0.5],
              opt = ['adam','Adadelta', 'Adagrad','Rmsprop'],
              kernel_init = ['uniform','normal', 'glorot_uniform'],
              batch_size=[256, 512, 1024, 2048],
              drop_rate= [np.random.uniform(0.1, 0.4)])

Wrap Keras model with sklearn API and instantiate random search

model = KerasClassifier(build_fn=create_model)
random_search = RandomizedSearchCV(model,  params, n_iter=5, scoring='average_precision',
                                   cv=5)

Search for optimal hyperparameters

random_search_results = random_search.fit(X_train, y_train,
                                          validation_data =(X_test, y_test),
                                          callbacks=[EarlyStopping(patience=50)])


In short, the precision-recall curve shows the trade-off between the two values as you change the strictness of the classifier.

There is a great explanation here, using the classification of images of airplanes and geese as an example.

  A good way to characterize the performance of a classifier is to look at how precision and recall change as you change the threshold. A good classifier will be good at ranking actual airplane images near the top of the list, and be able to retrieve a lot of airplane images before retrieving any geese: its precision will stay high as recall increases. A poor classifier will have to take a large hit in precision to get higher recall. Usually, a publication will present a precision-recall curve to show how this tradeoff looks for their classifier.


Is there an Orange server to deploy the models developed in Orange 3

No. Orange is a stand-alone, desktop application and is not a tool for designing analysis pipelines that would on their own run on the servers. Instead, it fosters interactive visualizations and exploratory data analysis that all require human input.


The learning_rate in TensorFlow code is the sum learning rate of a batch or the learning rate of a data?

It is the learning rate that you set on the optimizer for your problem. It requires a fair bit of experimentation to figure out what learning rate works best for your problem. It is a more of an art than a science, though there exists heuristics on an optimal range of values to select from.
For most problems, the default learning rate should be a good starting point.


Should we use discrete or continuous input for decision trees

Discrete is the way to go. The reason is simple if you visualize a decision tree it involves drawing a decision boundary based on a set of constraints which are in the form of features. It would be much easier to draw these decision boundaries based on discrete features compared to its continuous counterpart. If the values are continuous it becomes difficult for the classifier to effectively draw this boundary and might have some skew in its results.

There is a useful video series in Udacity, Intro into Machine learning, please refer the decision tree section they show a really good visualization on how decision trees work.

Link: https://in.udacity.com/course/intro-to-machine-learning--ud120

Please check that out it may help you understand better.


Ordered elements of feature vectors for autoencoders?

For a fully-connected network the precise order of features does not matter initially (i.e. before you start to train), as long as it is consistent for each example. This is independent of whether you have an auto-encoder to train or some other fully-connected network. Processing images with pixels as features does not change this.

Some caveats:

To succeed in training, you will need the pixel order to be the same for each example. So it can be randomly shuffled, but only if you keep the same shuffle for each and every example.

As an aside, you will still get some training effect from fully random shuffling the variables, because for example writing an "8" has more filled pixels than writing a "1" on average. But the performance will be very bad, accuracy only a little better than guessing, for most interesting problem domains.

To visualise what the auto-encoder has learned, your output needs to be unscrambled. You can actually input a (same shuffle each example) scrambled image and train the autoencoder to unscramble it - this will in theory get the same accuracy as training to match the scrambled input, showing again that pixel order is not important. You could also train autoencoder to match scrambled input to scrambled output and visualise it by reversing the scrambling effect (again this must be a consistent scramble, same for each example).

In a fully-connected neural network, there is nothing in the model that represents the local differences between pixels, or even that they are somehow related. So the network will learn relations (such as edges) irrespective of how the image is presented. But it will also suffer from being unable to generalise. E.g. just because an edge between pixels 3 and 4 is important, the network will not learn that the same edge between pixels 31 and 32 is similar, unless lots of examples of both occur in the training data.

Addressing poor generalisation due to loss of knowledge about locality in the model is one of the motivations for convolutional neural networks (CNNs). You can have CNN autoencoders, and for those, you intentionally preserve the 2D structure and local relationships between pixels - if you did not then the network would function very poorly or not at all.


What is the State of the art method for full body gesture recognition in images

If you are planing to train a model which is using images taken by a single camera, I encourage you to take a look at this paper. They use a camera to track a person and recognize gestures involving arm motion. In this paper, two alternative methods for gesture recognition are compared: a template based approach and a neural network approach.

However, another way which I suggest is using depth cameras such as Intel Realsense or Microsoft Kinect cameras. This paper uses depth cameras for full body gesture recognition. I think using these cameras take your project a step ahead and more accurate in initial states.


Compare between similar and dissimilar couples of instances

You can compute a similarity score between each couple of instances (diff in features) and then you can check if the distribution of the difference for each group (similar and dissimilar) is significantly different using the Kolmogorov-Smirnov test.


Why do machine learning engineers insist on training with more data than validation set?

The reasoning will be: "The more data for training the better". Then you have to keep in mind that the validation/hold-out set has to resemble how it should work on production/testing.  The theory is that the larger the training data, the better the model should generalize.
The validation set can be much smaller, on extremely big dataset you can make it even 0.01% of the data, and there should be no problem.
In basic cases you don't even need to do the K-fold, this makes the training more expensive and only for hyperparameter search and inside the training set it needs to be done.
For your case, you can consider the split you want. Just keep the balance of the training data to be as larger as possible and validation data to resemble the best the production environment as possible.


Gradient of NN output with respect to inputs

Yes there is. This is called "back-propagation to the input". I would like to invite you to read this awesome blog which relies on lucid. You will see some code on their notebooks.


Is is possible to build a recommender system with this dataset?

It is definitely possible to build a recommender system with the dataset you have.

Although there are a few questions.
1. What is your objective from the recommender system?
2. What approach do you want to take with each recommendation?
3. What is the final goal of the recommendation?

Novice Approach:
A very simple approach could be to recommend similar products, for e.g. if a person is looking at a phone A (on website) you suggest an alternative as phone B purely based on the similarity of the product and nothing else.

There are three steps to Recommender System:
1. Query Identification - What to Recommend?
2. Candidate Generation - How many to Recommend?
3. Ranking - In what order to Recommend?

If you answer these questions you will have a fairly good recommender system.


Rational business people don't pay for accuracy, they pay to either

save money on a profitable process (thereby making it more profitable), or by
creating new money (creating new profitable processes).

So any project that is undertaken has to be couched in terms that
   reflect this. The first step is always understanding which of the two
   processes you are working on, and you should have a clear idea of how
   that might be achieved, while keeping in mind that as you make
   progress the details of how you do that might change.

If you can improve the accuracy of a process, you can probably make money for the firm and the business people will invest in your progress. The only rational reason a business person could have for insisting on 99 percent accuracy and reject 90 percent is if they already had a way of doing it that was better than the 90 percent.  If that is the case they are of course justified in their position.

Understanding and presenting the business case of the projects you are working on in terms that the Business People understand is part of the maturing process of any Engineer. It is not unique to Data Science at all, though Data Science has some unique aspects (like lower maturity but higher probability of fortuous serendipity discovery - at least in today's environment.

A relevant process that is close to Data Science that makes this step explicit can be found here:https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining

But most Enterprise Architecture Frameworks are similarly applicable.


What could be the problem leading to the result that a model can never perfectly overfit?

I would guess there are a few samples that aren't necessarily from the same distribution as the rest. I would try identifying outliers and removing them.


How to determine number of Neurons setup in Convolutional Neural Networks?

A typical best practice is to read related papers or check github projects with information about similar architectures and how other ML engineers structured them. Usually, you can see some certain patterns in CNN architectures like increasing the depth along the pipeline (or number of channels) while decreasing width and height of results in each layer. Since the search space is huge and time so limited, it is usually better to rely upon already made architectures, and use, if possible, transfer learning to shorten training cycles. On the other hand, if you are doing pure research and want to create your top-notch, new state-of-the-art CNN architecture, you will need to carry out a very big effort on CNN architecture engineering as well as hyperparameter tuning, luckily on a systematic basis.


How to reshape data for LSTM training in multivariate sequence prediction

This makes sense. It should work for input and first couple of layers. For output layers, you can have a softmax if you need to generate only next record in sequence.

Following Keras code has an example that :

Accepts multi-dimensional inputs  (Each sample is a Sequence of video frames)
Predicts next few frames of video ( Multi dimensional since each pixel is a feature)

https://github.com/keras-team/keras/blob/master/examples/conv_lstm.py


Should unique vectors (SIFT descriptors) be used in K-Means Clustering?

It does make a difference.

If you have many duplicates, you can merge them into weighted vectors. It's straightforward to add to existing code.


Are CNNs insensitive to rotations and shifts in images?

If you use max-pooling layers, they may be insensetive to small shifts but not that much. If you want your network to be able to be invariant to transformations, such as translations and shifts or other types of customary transformations, you have two solutions, at least as far as I know:

Increasing the size of data-set
Using spatial transformers

Take a look at
What is the state-of-the art ANN architecture for MNIST and Why do convolutional neural networks work.

Thanks to one of our friends, another way is to use transfer learning after data-augmentation.


How does one individually extract elements from a tensor?

First recognize that your input array has shape (1,2), so a[0]*a[1] will not work.

Following is the code that is working for me-

def mul(a):
    return a[0][0]*a[0][1]

a = tf.constant([[2,3]])
b = tf.contrib.eager.py_func(func = mul, inp=[a], Tout = tf.int32)
with tf.Session() as sess:
    sess.run(b)

If you want to pass custom functions, check this link- https://www.tensorflow.org/api_docs/python/tf/py_function


How do I draw the following Keras model?

You should Go along with Tensorboard, it looks more professional than the rest and it is easier to understand the model.
May I ask your exact purpose for the visualization? You could check others material with the similar purpose and go along with the general choice! Many would go for draw_convnet or PlotNeuralNet if the final use-case was to publish a blog. It also depends on which libraries and softwares you are using!


k-means and perceptron generalization ability

Comparing K-means and perceptrons doesn't make sense, they are different types of algorithms.

K-means doesn't have a generalization ability at all, since it is an unsupervised algorithm, and generalization is a property of supervised learning algorithms.

If you are changing it to behave like a supervised algorithm, than it will have poor generalization ability compared to the perceptron, because K-means will have a tendency to overfit the data since it is forced to assign a cluster to each of its data points.

If you want to compare perceptron's performance to other algorithms, you should compare it to k-NN (K nearest neighbors), Random Forests, or Support Vector Machines.


How does training a ConvNet with huge number of parameters on a smaller number of images work?

First of all, you are ignoring the dimensionality of the problem. Images are very high-dimensional. Let's say an image has a resolution of $256 \times 256$, which means each image has $65,536$ pixels. ImageNet images are RGB, so each image has 3 channels, resulting in $196,608$ pixels per image. Now, the whole dataset ($1.3$m images) has more than $255$ billion pixels associated with its images.

Because filters work at pixel-level, the information it gains from one image is more than what a regular ML algorithm would gain from a single training example.

As you can see, the kernel (whose weights you want to update, and accounts to 9 parameters in this case) is much smaller than the input image. It would be a mistake to consider a single image (of around $200,000$ pixels as we say previously) as only capable of updating one parameter.

Secondly, the input of each layer changes from layer to layer, because of the sequential nature of the network. The second layer sees the output of the first,and so on... This means that the second layer's filters won't see the same image as the first and in fact, as training progresses, their input will also gradually change (because the first layer's filters will be getting more effective).

By the time it reaches let's say the $M^{th}$ layer, the input will have changed a lot from the original image, so it would be a mistake to consider that the original image is updating the totality of the parameters in the network.

Thirdly, data augmentation is also a thing. By making simple random transformations to each image (flips, shifts, scales, rotations, brightness/contast adjustments etc.) the network is tricked into thinking that this is a totally new image. This can exponentially increase the size of the dataset.

Finally, you should take a look at where the parameters in a network are. The VGG19 architecture does have more than $143$ million, but around $124$m of those are from the last 3 FC layers. In fact the first of the three layers has around $103$m parameters on its own! This is a very inefficient network design and that the research community has strayed far from recently. A more representative network you should look at is the ResNet architecture. For example a 50-layer ResNet (much deeper than the 19-layered VGG) consists of just around $25$m parameters, while achieving a similar -if not better- performance.