Are "Gradient Boosting Machines (GBM)" and GBDT exactly the same thing?

Boosting is an ensemble technique where predictors are ensembled sequentially one after the other(youtube tutorial. The term gradient of gradient boosting means that they are ensembled using the optimization technique called gradient descent (Boosting Algorithms as Gradient Descent.
Given this, you can boost any kind of model that you want (as far as I know). Moreover in the scikit learn library, gradient boosting, its under the ensemble folder. You can boost any kind of model (linear, svm) its just that decision trees achieve normally great results with this kind of ensemble. Same way that you can do a bagging with any kind of estimator, but if you do it with a decision tree and add a couple technichalities more, you can call it Random Forest.
From scikit learn documentation: GB builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage a regression tree is fit on the negative gradient of the given loss function.

But are Gradient Boosting Machines (GBM) and GBDT the same thing? Are they just different names?

Gradient boosting machines are a kind of ensemble and Gradient Boosting Decision Tree the particular case when a tree is used as the estimator.

Apart from GBM/GBDT and XGBoost, are there any other models fall into the category of Gradient Boosting?

You can use any model that you like, but decision trees are experimentally the best.
"Boosting has been shown to improve the predictive performance of unstable learners such as decision trees, but not of stable learners like support vector machines (SVM)." Kai Ming TingLian Zhu, Springer


So the question is count the number of duplications by user ID and other variables.
For this you can use the df.groupby().size() method. This will return the counts of unique entries in with a particular value in a column.

Here is the documentation for this function: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html


Since Neural Networks can learn any non-linear function, I would go for an Autoencoder for dimensionality reduction first, and then run a k-Means clustering on the encoded features. If a regular (non-linear) pattern in the data exist, an Autoencoder that is "deep enough" should be able to account for that.

For the same reasons (i.e. non-linear patterns) I would avoid linear techniques such as PCA.

If this suits you, you can check my Notebooks. I implemented Autoencoders in TensorFlow 1.x here, and in TensorFlow 2.0 here.


Alternative to VGAM for Zero Truncated Negativ Binomial GLM in R

I think the answer is "no". countreg does zero-inflated and zero-truncated count distributions, but only Poisson and NB. actuar gives pdf, cdf, et cetera, but doesn't do regression.


Welcome to the forums.

My understanding is that you're wanting to use the previously trained model to label new data points? If so, you'll be wanting to use .predict(X). From sklearn's documentation they say.

  All supervised estimators in scikit-learn implement a fit(X, y) method to fit the model > and a predict(X) method that, given unlabeled observations X, returns the predicted
  labels y. (Source)

Another note, is that you can't pass direct strings to a model - you'll need to preprocess your data like you did for your training set. Here is a good example of building a classifier and using it to predict new points.

Let me know if you have any questions of I've misunderstood.


How do I use Convolutional Neural Nets to classify if there are rain drops or soil on the camera lens?

So neural nets can be quite powerful for end-to-end training. So if you have a dataset that contains both clean and dirty images, and also the classes you are trying to detect in the first place, it should be fairly straightforward to try to train you net.

I'd proceed as follows.

adjust the network to produce a logits vector that outputs not only the object classification but also a binary classifier for whether your image is clean or dirty.
then adjust the labels for each image to indicate the classification label and the cleanliness of your image.
I'd then train and test on the dataset to see if a model emerges.

The convolutional parts of your network should converge to features that are good at
A. classifying the subject of the image, and
B. detecting whether the camera is dirty or not.

Its a good idea and I might use it myself!


Normalisation is a very blurry concept. It is quite misunderstood most of the times. I will take the specific case of Neural Nets and explain it:

Purpose: Normalisation is done so that the Neural Network weights converge faster. In CNN's and Deep Neural Nets this is of particular help especially in CNN this helps to prevent exploding/vanishing gradients.

The most common explanation for normalisation I have come across is that if you have 2 features, one of them has a significant larger scale than the other e.g house price and house area then the feature with a larger scale will dominate the output. This is quite incorrect according to me, since when you back-propagate through the Neural Net the weight updates are directly proportional to the activations, so larger activation means larger feedback and hence weights get reduced faster and become smaller until w1*house price = w2*house area approximately this relation holds true. Yes it will lead to more oscillations (intuitionally since learning rate also becomes multiplied with a larger scale) but it will ultimately probably converge.

So the best 3 reasons for using normalisation are:

If scale of a feature is large the weights connected to that feature will have larger oscillations resulting in slower convergence and if deep NN is used probably no convergence, whereas normalisation helps in making the values small -1 to 1 so the gradient updates are also small resulting in faster convergence.
The best intuition for normalisation can be found from this video of Stanford and its subsequent video. Since we know weight updates is directly proportional to inputs it will also take the sign of the inputs (or exact opposite sign) always. Now we know house price and area is always positive (in our Universe at least!). So the weight updates will always have definite sign either both positive or both negative (for error being propagated from a single next layer node). But the weight updates may have a optimal direction in the 4th quadrant, so the weight updates will follow a zig zag pattern which will result in loss of efficiency.
Finally when you are dealing with Deep Neural Nets like CNN if you do not normalise pixels it will result in exponentially large/vanishing gradients. Since generally softmax/sigmoid is used in the last layer, it squashes the outputs. If you have a large output, generally due to un-normalized data, it'll result in wither exact 0 or exact 1 output, which is fed into a log function and BAM! overflow. The error becomes inf or NaN in python. So inf error means exploding gradients and NaN means gradient cannot be calculated. So you fail to train from the start and it continues inf is generally followed by NaN which will continue to eternity. This can probably be remedied by using higher floating point precision but it will result in higher memory and processor consumption ultimately inefficiency.

TL;DR: Normalisation is used for faster weight convergence. Issues faced by un-normalised data are larger weight oscillations, weight updation in non optimal direction, overflow in precision in Deep Neural Nets.


how to compare between kmeans and hierarchical clustering results

In general, even using k-means more than once will create slightly different clusters (that s if you do not set a random seed).
Ideally, the profiles of the clusters and the distribution of data points into them will be the similar. If this is the case then it shouldn't really matter which one you choose. If not then, I suggest you examine carefully what caused any major differences.
If you really want to chose, then:

you can use a metric like total within distance or silhouette (ex. select the clustering solution with the lowest total with distance or with maximum average silhouette),
you can use your judgement to select the clustering solution that makes more sense business wise.

As mentioned before, ideally, metrics like total within distance or silhouette should not differ much. Hence, it all comes down to using the second criterion.
As for the part of the question about any benefits from using multiple algorithms,

if you are referring to k-means and hierarchical clustering, you could first perform hierarchical clustering and use it to decide the number of clusters and then perform k-means. This is usually in the situation where the dataset is too big for hierarchical clustering in which case the first step is executed on a subset.
in general, since not all clustering algorithms are suitable for every case it is useful to use multiple algorithms. For example k-means and hierarchical clustering are good at detecting clusters that have a spherical/ball shape. They would perform poor on more complex shaped data. A problem that DBSCAN does not have (though DBSCAN has other disadvantages, see discussion at Wikipedia)


Deep learning test loss curve won't go down

Your learning curves suggest overfitting.
In this respect you should try

increasing dropout
adjust l1/l2 regularisation to higher levels
inspect class ratios in training and validation tests ("learning on easy predicting on difficult" imbalances)

Additionally, you may want to

add batchnorm layer
try different activations
check to what extend weights/bias of each layer get updated during training iterations, in case of vahishing/exploding gradients


Time horizon T in policy gradients (actor-critic)

In this case, I think it doesn't matter when you reach $s_{spec}$, but how the q-value gets updated because of taking an action at that state.
Therefore there shouldn't be different q-values for each possible $t\in \{0, ..., T\}$, only q-values for each possible actions.
I'm sure it does make a difference for being at a state at a specific timestep, but it's the agents job to learn this by using the RL algorithms (like policy gradient method in the lecture).

In regards to $T$ being fixed or not, horizon $T$ can be infinite or fixed to a finite number.
For example, if $T$ is fixed to $10$, the agent should learn a policy that maximizes the total discounted rewards in the finite amount of time, but it may not be the most optimal policy. When $T$ is infinite, there is more time to explore and figure out the most optimal policy.

The closest method I know that takes notice to when the state-action pair was encountered is Experience Replay that is used in DQN.

I'm also learning Reinforcement Learning right now! I recommend Deep RL Bootcamp since they give you labs in Python which are really intuitive.


How to install XGBoost or LightGBM on Windows?

Follow this XGBoost installation guide: https://xgboost.readthedocs.io/en/latest/build.html

If you are using python3 then make sure that you run: pip3 install xgboost

To fix the problem with lightgbm on windows try installing OpenSSL first, refer this: https://www.cloudinsidr.com/content/how-to-install-the-most-recent-version-of-openssl-on-windows-10-in-64-bit/n


Should I make single machine learning model for predicting price of house or 6 different models, given 6 datasets for different cities?

Both options would work. It would depend on which method you want to use, what you want to do afterwards, or computing capacity.
E.g. developing a model for each city might allow you to contrast and compare the outcomes more simply. But you can also do multivariate predictions to predict for each city even once you have combined your dataset.
Also if your dataset is very large it might make sense to treat the cities independently to make the computations slightly cheaper.


Probably you can use multiple decision trees - each for slice of dataset. Then combine the results from each decision tree. You can also weigh the results from each decision tree. Eg the dataset for 2015 can be given more weight than dataset for 2014. I do not see much harm in spliting dataset and training different trees as it helps to account for the predictor values in a more accurate manner. If you could more details about the data or the "variable" you mentioned, people would be able to help more.


Arguments in python fast.ai function that are not in the function definition?

Those parameters are taken care of by **kwargs in the function definition.
You can look how this is dealt with by the code in the fastai repo and looking for **kwargs in the function definition.
Some explanations about how this works can be found on SO: https://stackoverflow.com/questions/3394835/use-of-args-and-kwargs.


What to consider before learning a new language for data analysis

Personally going to make a strong argument in favor of Python here. There are a large number of reasons for this, but I'm going to build on some of the points that other people have mentioned here:

Picking a single language: It's definitely possible to mix and match languages, picking d3 for your visualization needs, FORTRAN for your fast matrix multiplies, and python for all of your networking and scripting. You can do this down the line, but keeping your stack as simple as possible is a good move, especially early on.
Picking something bigger than you: You never want to be pushing up against the barriers of the language you want to use. This is a huge issue when it comes to languages like Julia and FORTRAN, which simply don't offer the full functionality of languages like python or R.
Pick Community: The one most difficult thing to find in any language is community. Python is the clear winner here. If you get stuck, you ask something on SO, and someone will answer in a matter of minutes, which is simply not the case for most other languages. If you're learning something in a vacuum you will simply learn much slower.

In terms of the minus points, I might actually push back on them.

Deepening your knowledge of one language is a decent idea, but knowing only one language, without having practice generalizing that knowledge to other languages is a good way to shoot yourself in the foot. I have changed my entire favored development stack three time over as many years, moving from MATLAB to Java to haskell to python. Learning to transfer your knowledge to another language is far more valuable than just knowing one.

As far as feasibility, this is something you're going to see again and again in any programming career. Turing completeness means you could technically do everything with HTML4 and CSS3, but you want to pick the right tool for the job. If you see the ideal tool and decide to leave it by the roadside you're going to find yourself slowed down wishing you had some of the tools you left behind.

A great example of that last point is trying to deploy R code. 'R''s networking capabilities are hugely lacking compared to python, and if you want to deploy a service, or use slightly off-the-beaten path packages, the fact that pip has an order of magnitude more packages than CRAN is a huge help.


Extending DTW 1-NN Classification to On-line Scenario

Hmm
Yes and no.
This is called "early classification". There as about two dozen papers that do this, but at least 20 are flawed...

See https://www.youtube.com/watch?v=d_qLzMMuVQg
http://www.cs.ucr.edu/~eamonn/UCRsuite.html


Do xgboost and random forests in general handle multiple splits of the same numeric feature in a single branch?

Yes, it can handle multiple splits withing the same branch.
A decision tree model can use the same feature as many times as optimally needed.
You can see an example in the sklearn doc.
The petal-length feature is used multiple times in the same branch.


How does Xgboost learn what are the inputs for missing values?

The procedure is described in their paper, section 3.4: Sparsity aware split-finding.

Assume you're at your node with 50 observations and, for the sake of simplicity, that there's only one split point possible.

For example, you have only one binary feature $x$, and your data can be split in three groups:

Group $B$: 20 observations such that $x=B$,
Group $C$: 20 observations such that $x=C$,
Group $M$: 10 observations such that $x=?$

The algorithm will split based on $x$, but does not know where to send the group $M$. It will try both assignments,

$$(B,  M), C \qquad \text{and} \qquad B, (C, M),$$

compute the value to be assigned to the prediction at each node using all the data, and chose the assignment that minimizes the loss.

For example, if the split $(B,  M), C$ is chosen, the value of the left node will have been computed from all the $B$ and $M$ samples.

This is what is meant by

  Automatically "learn" what is the best imputation value for missing values based on reduction on training loss.

Comment question:

  how can group M be calculated if I don't know what are the values for them?

The value we compute is not based on the feature $x$, which we don't know, but based on the known label/target of the sample. Let say we are doing a regression and trying to minimize the mean square error, and let say we have the following:

For group $B$, the mean of the target is $5$.
For group $C$, the mean of the target is $10$.
For the missing value group $M$, the mean of the target is $0$.

Note that even if the value of $x$ is missing, we know the value of target of the sample, otherwise we could not use it to train our model.

In this case, the split would be made on $(B,M), C$, the value assigned to the right node containing samples $C$ would be $10$, and the value assigned to the left node containing samples in $(B, M)$ would be the mean of the target for the whole group. In our example, the mean would be

$$\frac{|M|}{|B| + |M|}\text{mean}(M) + \frac{|B|}{|M|+|B|}\text{mean}(B) = \frac{10}{30}0 + \frac{20}{30}5 = 3.\overline{3}$$


In Neural Nets, why Use Gradient Methods as Opposed to Other Metaheuristics?

It would be a waste of information; the gradient is available, so use it and save time.
There is reason to believe that the local optima are good; see, for example, Choromanska et al. (notes).
Over-optimizing for the training set leads to worse generalization, so sometimes we deliberately don't even try by stopping early. Probably the best free lunch in machine learning.


What is difference between Nearest Neighbor and KNN?

Not really sure about it, but KNN means K-Nearest Neighbors to me, so both are the same. The K just corresponds to the number of nearest neighbours you take into account when classifying.
Maybe what you call Nearest Neighbor is a KNN with K = 1.


What is the best way to normalize a set of datasets

If you know strict bounds on the sensor output, that would be better than normalizing by the min/max of the dataset. Even if the bounds are not necessarily strict, but simply reasonable, that would suffice. For example, if there are no theoretical bounds on a temperature sensor, you might reasonably impose strict bounds given prior knowledge about its environment (e.g. if a temp sensor was placed in NY, you might assume strict bounds as -30C to 50C)
If you were to normalize by the min/max of the training data, what do you expect to happen if the deployed model encounters a value outside this range? If, for example, your training set had min 5 and max 30, how would you normalize an input of -10? It would be much more intuitive and reliable to shift up to a nonnegative domain, (i.e. subtract your strict minima), then scale to a value in [0,1] via the strict range.
Also, depending on your neuronal activation functions, consider centering your data in addition to scaling it.


Is Gradient Descent central to every optimizer?

No. Gradient descent is used in optimization algorithms that use the gradient as the basis of its step movement. Adam, Adagrad, and RMSProp all use some form of gradient descent, however they do not make up every optimizer. Evolutionary algorithms such as Particle Swarm Optimization and Genetic Algorithms are inspired by natural phenomena do not use gradients. Other algorithms, such as Bayesian Optimization, draw inspiration from statistics.

Check out this visualization of Bayesian Optimization in action:

There are also a few algorithms that combine concepts from evolutionary and gradient-based optimization.

Non-derivative based optimization algorithms can be especially useful in irregular non-convex cost functions, non-differentiable cost functions, or cost functions that have a different left or right derivative.

To understand why one may choose a non-derivative based optimization algorithm. Take a look at the Rastrigin benchmark function. Gradient based optimization is not well suited for optimizing functions with so many local minima.


python sklearn decision tree classifier feature_importances_ with feature names when using continuous values

I think you are mixing two different things here.

The feature_importance_ - this is an array which reflects how much each of the model's original features contributes to overall classification quality.
The features positions in the tree - this is a mere representation of the decision rules made in each step in the tree. A feature position(s) in the tree in terms of importance is not so trivial.

There are some potential heuristics for understanding the relation between the two. If a feature dosn't appear in the tree it has 0 importance and generally the higher the feature is in the tree the more important it is (assuming its being compared to another feature on the same branch).


What technologies are fastest at performing joins on large datasets?

More importantly than the technology is the type of join you are using. For instance if the join keys are sorted, you can use sort merge joins and use join orders to get a better performance.

That being said, you can use in memory solutions for fastest joins if the size of your intermediate results will not blow up your cluster memory. Look at Spark SQL or Mem-SQL for instance.


why naive is needed in Naive Bayes ,what happens if naive is not included in Bayes theorem?

I do not think your formulation is correct. What you have described are just conditional distributions for each word in the sentence but not the joint conditional distribution, given a specific class.

In your case, we have by Bayes rule:

$$ Pr(spam | X) \propto Pr(X | spam) \times Pr(spam) = Pr(you, won, lottery, for, 1million | spam) \times Pr(spam).$$

Now, $\hat Pr(spam) = \frac{\text{# of spam occurences in train}}{\text{total observations in train}}$, the MLE that uses none of the features.

In Naive Bayes, we make the assumption that:

$$Pr(you, won, lottery, for, 1million | spam) \times Pr(spam) = Pr(you | spam) * Pr(won | spam) * Pr(lottery | spam) * Pr(for|spam) * Pr(1million | spam) * Pr(spam).$$ That is, we assumed that each of the features are conditionally independent given a class. If we do not wish to make this assumption, we simply leave $Pr(you, won, lottery, for, 1million | spam)$ as is in the above expression and posit some sort of multivariate/joint conditional distribution.

The reason why we make this assumption is due to the difficulty in determining what the conditional joint distribution of the features is given a particular class. Is the joint distribution a multivariate Poisson? A complex mixture of many? Some other multivariate counting distribution? What is the dependence structure, and how should we represent it? What if we have a thousand features? How do we represent the dependence structure now in a reasonable way? Is the dependence structure so complicated that representing it reasonably is more annoying and/or time-consuming than just assuming independence?

These questions are not trivial and are very difficult, if not impossible, to answer. Thus, we make the independence assumption to simplify the problem. We may not know the joint conditional distribution but perhaps we can derive and/or visualize the marginal conditional distributions for each feature. This, of course, comes at the cost of assuming independence between features which is highly unlikely to be true (especially for word frequencies) but nevertheless occasionally effective for some problems (kind of).


Random Forest in R with only character variables

You can check the class of the class(df$dependent). You are expecting it to be numeric.

To convert multiple columns to factors, you can do something like this

factor_cols <- c("col_1","col_7"),
df[factor_cols] <- lapply(df[factor_cols], as.factor)

If you keep the customer id,  then you will have a problem when applying your model to a new customer.


counter vector fit transform cosine similarity memory error

It's not clear to me what is your data and what you are trying to do with it, but from what I gather you are trying to calculate cosine similarity for each pair in a cartesian product, right?

If yes then you might want to use "blocking" to reduce the number of comparisons, see https://datascience.stackexchange.com/a/54582/64377.


Something simlar to the blue fitted line can be obtained using Holt-Winters model. Check HoltWinters() function from stats package in R.

You can find some examples here and here.


Where can I learn the complete mathematics involved in LDA?

Welcome to DS.SE!

I believe the Latent Dirichlet Allocation paper by Blei, Ng, and Jordan (all big names in AI/ML/NLP) is a point to start. They provided the math foundation as well as multiple examples.

See: Blei, David M., Andrew Y. Ng, and Michael I. Jordan. "Latent dirichlet allocation." Journal of machine Learning research 3.Jan (2003): 993-1022.


How to define a suitbale number of centroids for fuzzy c-means?

You can try the exact same heuristics you would use for k-means.

But usually, you would do this:

try some parameters by experience
run the algorithm
carefully inspect the result
go back to 1 and try other parameters until you have found something interesting or are tired of trying

There is no "correct" solution in clustering.

It's about discovering some new pattern and you cannot out the "new" into an optimization equation. The point is to try, and try again. It is not a drop-in replacement for classification.


Can colored images have more than 3 channel values?

Yes, Images can have more than 3 channels. Satellites routinely record multiple frequencies at once (for instance infrared). Normal monitors can't render that outright and you'll have to project those channels back to RGB. A simple way to do that is false colors.

These three false-color images demonstrate the application of remote sensing in precision agriculture: The left image shows vegetation density and the middle image presence of water (greens / blue for wet soil and red for dry soil). The right image shows where crops are under stress, as is particularly the case in fields 120 and 119 (indicated by red and yellow pixels). These fields were due to be irrigated the following day.

But convolutional nets take inputs (for instance RGB) and transform them to higher dimensional representations by applying several convolutions over the data. That is what you are referring to now. You can think of those bands as different filters that highlight features that we originally detected in the RGB-channels. They live inside the pipeline, and are then projected to a (typically lower dimensional) target (in this case 2)

So You can also input multi-band pictures, convolve those to even more channels.


Collaborative Social Network Visualization

I think Gephi, an open-source visualization tool, would help you a lot. Actually, as I know, the InMaps and its community detection algorithm are same as the Gephi's.


What is the difference between float64 and double in TensorFlow?

Taking a look at tensorflow's dtypes.py, there's this line:

double = float64

So double is exactly the same at float64.


Can you recommend a machine learning challenge that is suitable for novices?

You've already answered yourself by tagging kaggle.

Let me share the two competitions that I have every new hire in my team go through- and then have them keep improving their solurion for the next six months. These are a very well curated set of problems for a budding data scientist!

Titanic: Machine Learning from Disaster
https://www.kaggle.com/c/titanic
This one helps bridge the gap between an analyst and a data scientist, as well as eases you into the world of code from Excel.
Digit Recognizer
https://www.kaggle.com/c/digit-recognizer
The MNIST dataset has prepared almost every data scientist for the real world of dirty data, in a sweet, cushy manner. It makes very real the fact that not always will we have structured and organized data, but it should not deter us from gaining insight from it!

Happy Coding!


Using text classification for system calls

The problem seems to arise in this line.

predictions = Dense(N_CLASSES, activation='softmax')(average)

Here, N_CLASSES has the value of 2. Meaning you have 2 classes. But the y_train vector has shape ( 1, ).

You need to convert the y_train vector to a one hot vector. Here's any example.

If y_train = [ [ 0 ] , [ 1 ] ] then its one hot vector would be [ [ 1 , 0 ] , [ 0 , 1 ] ]. The one hot vector has shape ( 2 , ) which is needed in the above code snippet.
So make this edit.

y_train = keras.utils.to_categorical( y_train , N_CLASSES )


Large Scale Personalization - Per User vs Global Models

The answer to this question is going to vary pretty wildly depending on the size and nature of your data. At a high level, you could think of it as a special case of multilevel models; you have the option of estimating a model with complete pooling (i.e., a universal model that doesn't distinguish between users), models with no pooling (a separate model for each user), and partially pooled models (a mixture of the two). You should really read Andrew Gelman on this topic if you're interested.

You can also think of this as a learning-to-rank problem that either tries to produce point-wise estimates using a single function or instead tries to optimize on some list-wise loss function (e.g., NDCG).

As with most machine learning problems, it all depends on what kind of data you have, the quality of it, the sparseness of it, and what kinds of features you are able to extract from it. If you have reason to believe that each and every user is going to be pretty unique in their behavior, you might want to build a per-user model, but that's going to be unwieldy fast -- and what do you do when you are faced with a new user?


You could use X and lin_reg.predict(X_poly), but

The plot will put those points in the order they appear in X, and since you're using a line connector it will appear to jump all over the place.  You could fix this by using a scatterplot instead.
It's vaguely disingenuous to use the training set's x-values for plotting the fitted curve; using np.linspace to get equally-spaced x-values is preferable (even if your original X was randomly generated and should fill the space reasonably well).


About A technique mentioned on ESL to transform an unsupervised into supervised

Fascinating problem, welcome to the site! In the second edition, this is covered in section 14.2.4 Unsupervised as Supervised Learning. The idea here is to design a classifier (that's supervised learning) between the desired but unknown distribution, and a known reference distribution (14.10), then to use this classifier along with the reference distribution to estimate the unknown distribution at any point (14.12). You can see that when the classifier deems a point to have come from $g$ ($\hat \mu \to 1$), the density $g_0 \hat \mu /(1-\hat \mu)$ increases. The converse is also true. This is what you would expect.

Mass refers to the probability measure of the sample. The idea is to redress by difference in sample sizes by weighting the probability of each sample inversely by its size (or equivalently, the size of the other sample). That's why the density of the mixture is the average of the components. A mixture density is simply a distribution formed by a mixture or sum (of two distributions in this case). Finally, $\mu$ is commonly used in statistics to denote the mean. The conditional mean of $Y$ is simply the probability that $x$ is drawn from $g$ (because the $Y=0$ term does not contribute to the expectation), thus

$$\mu(x) \equiv \mathbb E(Y|x) = \mathrm P(Y=1|x) = \frac{g}{g+g_0}(x)$$

This equation comes from Bayes' rule, after using $g = P(x|Y=1)$ and $g_0 = P(x|Y=0)$. This approach shows the power of using density ratios. In fact, there's a whole book devoted it.


Clustering is an unsupervised approach wherein unlike supervised learning its devoid of not knowing the class label or the response or dependent variable. In clustering, the idea is to determine patterns in data such that it can be grouped into cohesive clusters. Such groups will have data points that are similar to each other whilst dissimilar to other data points in other groups. Now, clustering algorithms can broadly be classified into two types, namely, partitional and hierarchical. Some well-known partitional clustering algorithms in the literature are k-means (that works for numerical data only), k-modes (that works for categorical data only), and the k-prototypes (that can work for mixed data (i.e. both numerical and categorical data)).
A fundamental problem associated with partition-based clustering algorithms is not knowing the initial number of groups. And as such researchers using this class of algorithms often fall into this trap by failing to address this fundamental issue. Thus their results are not reproducible.
In literature, there exist several approaches to have proposed to solve this issue of Estimating the possible number of clusters. A well-known approach is the Elbow method proposed by researchers (Kaufman & Rousseeuw, 2009). I fact a group of researchers developed an R package called NbClust that can be used to estimate the number of clusters using the elbow method. This package contains ~30 algorithms for determining the number of clusters.
Recommendation when building a partition-based clustering model
I would recommend the following steps for result reproducibility

setting a random seed value
choose a method to determine the number of clusters
run the experiment at least 5 to 10 times, note the result for each run, and then average the values. Report the averaged value as the final result.

Reference
Kaufman, L., & Rousseeuw, P. J. (2009). Finding groups in data: an introduction to cluster
analysis (Vol. 344): John Wiley & Sons


Implementing sklearn.linear_model.SGDClassifier using python

I think that this is the same issue as in this question.

The shape of $X$ must be (n_samples, n_features) as explained in the
SVC.fit docstring. A 1-d array is interpreted as a single sample (for
convenience when doing predictions on single samples). Reshape your $X$
to (n_samples, 1).

That means you should use numpy.reshape to reshape the $X$ column. If the data frame has n rows, you should use
X_new = X.reshape(n, 1)

Then use the fit method with $X_{new}$. Note: you probably don't need to do this if you use two or more $X$ columns for your model fitting.


Value extraction from a python dataframe [ problem statement specific ]

The problem with your code is that it considers the numerics as column names ie., 0,1,2,3, etc., upto 31 rows and 24 hours column  but u must loop through each of the column name value which is stored as dictionary so u need two looping:

This loops and creates a new list which stores each and every attribute of the json
This adds the attributes to the DataFrame
now perform your operation at this dataframe you might get an answer

Google drive shared location of edited ipython notebook


Why is there a trade-off between bias and variance in supervised learning? Why can't we have best of both worlds?

The tradeoff between bias and variance summarizes the "tug of war" game between fitting a model that predicts the underlying training dataset well (low bias) and producing a model that doesn't change much with the training dataset (low variance).

What statisticians/mathematicians a while ago realized is that any model can be made to perfectly fit the dataset at hand (i.e. have zero bias). Look at this picture, for instance from the Wikipedia page on overfitting:

The graph depicts two binary classifiers that are trying to distinguish between the blue class and the red class. The classifiers have been trained on all of the blue and red observations shown in the graph to generate their boundaries. Notice that the green line has zero bias; it perfectly separates the blue and red classes whereas the black line clearly has a few errors on the training set and therefore has higher bias.

The problem? Observations in datasets are realizations of random variables that follow some unknown probability distribution. Thus, observed data will always have some sort of noise, containing observations that don't actually represent the underlying probability distribution. These anomalies can be seen in the graph above in which there are red and blue observations that appear to be "crossing" the implied boundary between the two classes. Clearly, the majority of observations are easily distinguishable in this simple example except for a few cases. These few cases can be viewed as noise, which is inherently random and therefore, cannot be predicted.

The green line, as a result, is essentially fitting random noise which by definition is unpredictable and non-representative. The result is that if we were to train both classifiers again using a new training dataset (from the same process), the "green boundary model" will be expected to generate a very different boundary compared to what is shown above, since it was influenced by data that did not represent the actual underlying data generating process. This is overfitting = high variance. The model associated with the black line, which has refrained from fitting the noise of the data, should be expected to remain relatively stable since it was not influenced by data that was not actually representative of the population.

What does this all mean? The model associated with the black line is closer to reality (the "truth"), and therefore, if we were to pull new observations from "reality" that neither classifier has seen it will, on average, be expected to produce lower overall error despite having higher bias. This is the bias-variance tradeoff, which is really just a way of stating the tradeoff between underfitting and overfitting. I don't have another graph to illustrate this, but imagine if I straight up drew a diagonal line from the bottom left corner to the top right corner, and called this another classifier. Now, my "model" has zero variance; in fact, it is completely independent of the training dataset. But now, I haven't even fit the training dataset. I now have zero variance for tons of bias and large overall error.

The decomposition of MSE into bias, variance, and irreducible error encapsulates the tradeoff. When you hope to fit a model that generalizes well to the population, simply having a model with low bias is not enough and minimizing variance is also important, if not more important. But, if we are to take variance into consideration, we now have to sacrifice some ability to predict the training set well (through regularization) in hopes of obtaining a model that is closer to the truth.

Almost all ML methods use regularization to effectively trade bias (again, worse performance on the training dataset) for hopefully, lower variance. Some examples:

L1/L2/elastic nets introduce extra terms to the unbiased maximum likelihood estimators for the regression coefficients in GLM's. The introduction of these terms make the estimators biased, but the idea is to hopefully penalize in an effective way such that we get bigger gains in reducing variance, which in turn results in lower overall error.
Support vector machines (see the picture above) have a cost parameter that controls how smooth or "wiggly" the boundary is, which as you can see in the graph above trades bias by purposely missing some observations in the training dataset for a model that generalizes better to the population overall.
Neural networks have dropout layers that effectively introduce bias by simply removing hidden units from the model at random. The goal here is to purposely sack parts of the model that have learned from the training dataset in hopes of preventing complex co-adaptions that are highly specific to the training set and not to the general population.
Decision trees can be controlled through pruning (cost complexity, removing branches that do not significantly reduce in sample error/lead to large enough information gains) and through limiting the depth of the tree. The result of both methods? Again, higher bias (removing rules near the bottom of the tree that have potentially been created to better predict non-representative noise observations, leading to sparse terminal nodes) but hopefully, lower variance.


What is a suitable Tensorflow model to classify images into foggy/not foggy?

From the sounds of the problem you could probably do some thing with extracting some features from the images such as how many edges they have, brightness (to get day/night), average color values. Then using a more simpler classification algorithm such as SVM, KNN, Decision Tree, Random Forest.

The Tensorflow ready-to-use models look to be very complicated models with a large number of layers so it will take a long time to train and run. It will also be very hard to train them from scratch (Retraining a pretrainid network could help with that though). So I think they might be a bit overkill. Also note that those models were probably made with the imageNet dataset in mind which has 1000 classes where you have just 2.

Its very hard to know what will work without seeing the images or being able to trying it first.

I would start with simpler faster methods before trying slower more complicated methods.
So in order try the feature extraction plus classifier if that is unable to learn a good relationship then move on to a basic CNN if that doesnt work move on to the more complicated CNN models.

With respect to the objects within rather than the overall appearance. For a CNN using different pooling layers can effect this change this. eg. Max Pooling will get the max value of a filter so can be largely effected by a small part of the image where as Average Pooling will be better at looking at the whole image.


How to load numerous files from google drive into colab

One possible option would be operate directly on the zip files using zipfile.ZipFile.
Counting the number of items in a zip file:
from contextlib import closing
from zipfile import ZipFile

with closing("/content/gdrive/My Drive/path.zip") as zip_file:
    count = len(zip_file.infolist())


How to continuously train a model with a stream of new incoming data

This is a form of continual learning / life-long learning, which tends to be very challenging because retraining on new data make traditional models "forget" what they had previously learned. Depending on the frequency by which you receive new data and the time it takes to train the model, it is often infeasible to retrain on all data every time you receive a new sample.
w.r.t. LSTMs, you don't need to change your input size because you can partition your time series data according to a preset and constant input size. For example, if you have 100 ordered samples, you could have the LSTM receive input of 10 samples and predict/regress the following 11th sample. So you'd have 90 partitioned samples of constant size. If you had another ordered sample (101), then you'd have 91 partitioned samples of size 10.
Continual learning is an active research field, so you can find many recent works such as this one from a couple months ago on adversarial CL.
For a review of CL in robotics, you can see this paper.


Any clear winner for Data Science in Scala?

Not sure anybody have worked with all these tools, so I'm going to share my experience with some of them and let others share their experience with the others.

MongoDB addresses problems that involve heterogeneous and nested objects, while data mining mostly works with simple tabular data. MongoDB is neither fast with this type of data, nor provide any advanced tools for analysis (correct me if you know any). So I can think of a very few applications for Mongo in data mining.

Hadoop is a large ecosystem, containing dozens of different tools. I will assume that you mean core Hadoop features - HDFS and MapReduce. HDFS provides flexible way to store large amounts of data, while MapReduce gives basis for processing them. It has its clear advantages for processing multi-terabyte datasets, but it also has significant drawbacks. In particular, because of intensive disk IO during MapReduce tasks (that slows down computations a lot) it is terrible for interactive development, iterative algorithms and working with not-that-big datasets. For more details see my earlier answer.

Many algorithms in Hadoop require multiple MapReduce jobs with complicated data flow. This is where Scalding gets shiny. Scalding (and underlying Java's Cascading) provides much simpler API, but at the moment uses same MapReduce as its runtime and thus holds all the same issues.

Spark addresses exactly these issues. It drops Hadoop's MapReduce and offers completely new computational framework based on distributed in-memory collections and delayed evaluations. Its API is somewhat similar to Scalding's with all MR complexity removed, so it's really easy to get started with it. Spark is also the first in this list that comes with data mining library - MLlib.

But Spark doesn't reinvent things like basic linear algebra. For this purpose it uses Breeze. To my opinion, Breeze is far in quality from scientific packages like SciPy, Octave or Julia, but it is still good enough for most practical use cases.

Mahout relies on Hadoop's MapReduce and thus is terrible for iterative algorithms. Spire and Saddle look cute and probably have their niche, but seem to be much less well-known than Breeze. I couldn't find much information about H2O, so it doesn't look like a big player here (comments from people who used it are welcome).

Some quick summary. Spark seems to be the most simple, flexible and fast-growing project for large-scale data processing. It facilitates a number of new projects (e.g. Shark or Spark SQL) and penetrates into existing (including Cascading and Mahout). Spark knows how to utilize HDFS API, and thus scales to terabytes of data easily. And for data mining downshifters who don't want to bother with a cluster setup there's always pure Breeze.


Ok. So what I understood is, that for cosine metrics, I can use both: Sum of squared distances from centroids to vectors that belong to clusters, where the distance can be calculated as euclidian or as cosine (cosine would be probably more precise, but more complicated(thanks to dot product)). The squared distance is only used as optimization, so we don't have to calculate the square root in both euclidian and cosine distance formula.


I have tried SMOTE for categorical response variable scenario but never tried on Continuous response variable but I think that should not matter. You can run oversampling or undersampling based on the predictor variables. If you are using R, you can use DMwR library and use the SMOTE function. In Python you got to use imblearn.over_sampling.SMOTE


Confusion with euclidean distance implementation

First of all, you need to convert your categorical columns to numerical if you like to take them into account (e.g. column $T_m$).
You need to normalize your features as there is variance in scale (e.g. column $G$ vs $BLK$)
You probably need to reduce dimensionality as k-means MIGHT get stuck in Curse of Dimensionality.

The rest is fine. Euclidean distance generalizes for any number of dimensions.
$$\sqrt{(x_{1} - x_{2})^2 + (y_{1} - y_{2})^2 + (z_{1} - z_{2})^2 + ...}$$

So, when you calculate your k-means, you will get $k$ points each of them in $d$ dimensions ($d$ is the number of columns). Then for each data point you calculate the distance above for data point and the center i.e.

$$\sqrt{(x_{data} - x_{center})^2 + (y_{data} - y_{center})^2 + (z_{data} - z_{center})^2 + ...}$$

Python does it for you easily.


First of all, I would get acquainted with some core concepts of Machine Learning before building websites:

Common algorithms, differences between them (Naive Bayes, KMeans, Regression, Support Vector Machines, Decision Trees, you name it...)
Supervised vs. unsupervised
Features and feature selection
Classification evaluation (precision, recall, accuracy)

Machine learning is a huge field which is applied in many different areas. I don't know how proficient you are at this stage, but I would certainly play around with these things first in order to get a solid understanding of what you're dealing with. After that, the options are virtually limitless in a world of Big Data (forgive me for using the annoying buzz-word). What is truly valuable is someone knows how to do machine-learning rather than what (because there's enough to go about).


How to build a classifier for determining if two pairs are a match?

Generally your approach looks good to me. Here are a few comments/suggestions:

You didn't mention how the set of labelled pairs is obtained. This part can be tricky when doing record linkage among a large set of N addresses, since it's very difficult to manually annotate all the N*N pairs. Bootstrapping is a common approach afaik.
The way the data is obtained may also have consequences about the proportion of positive/negative cases. In general the proportion positive cases is very low, and this can cause the usual problems of class imbalance. It's important not to train the model with a positive/negative ratio completely different from the one in the test set/in production.
It's often useful to diversify the similarity measure: instead of using only Jaro-Winkler, you can think of cosine-TFIDF, Levenshtein edit distance, and a lot of variants. In the same logic it's useful to vary the levels of comparison, e.g. characters, characters bigrams/trigrams, words.


Is there a proper way of comparing weight coefficients of two logits?

Of course, this makes complete sense, as long as all the other variables are the same. In fact, if you think of logistic regression as a linear regression with the log-odds of the output variable (as in http://www.wright.edu/~thaddeus.tarpey/ES714glm.pdf), then it is trivial that comparing the weights is the same as comparing them in linear regression, which makes perfect sense.


Over fitting and association with regularization

The short answer is that regularization penalizes model complexity. It does this by adding some function of your parameters to your basic cost function as a regularization term. This forces a compromise between a good fit (low error) on your training data and model simplicity by driving the parameters closer to zero. This reduces overfitting because large parameters (in terms of magnitude) are the main culprit behind overfitting.

For example, L1 regularization uses the sum of the absolute values of your weights, while L2 regularization uses the squares of your weights. More detailed comparison here.

For mathematical details, have a look at this answer.


Features standardization - Multilayer perceptron

Generally, preprocessing parameters are fit only on the training subset, because otherwise you could overfit your data and overestimate quality of your model on the test subset.

With feature standardization, however, overfitting is not so dangerous, so I assume you can preprocess all your dataset at once safely.

The best practice, however, is to do all the processing in one pipeline and apply cross-validation to the whole pipeline:

from sklearn.pipeline import make_pipeline
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score, GridSearchCV
scaler = StandardScaler()
model = MLPClassifier()
pipeline = make_pipeline(scaler, model)
scores = cross_val_score(pipeline, X, y)

If you use scikit-learn cross-validation, incorporating pipelines would be simple.


Can we use GAEs with A3C asynchronous reinforcement learning?

Sure can:

https://github.com/ikostrikov/pytorch-a3c

The reason is the GAE paper was released after A3C, so the paper implementation, which is what most people will look at, doesn't use GAE.

Relevant lines here: https://github.com/ikostrikov/pytorch-a3c/blob/master/train.py#L95-L98


Can we use difference-in-differences with a biased A/B test?

Why are you hesitant to start over? A/B testing is double-edged sword because it is a relatively straightforward exercise (as these things go) but it has the power to completely pivot your business in a very short amount of time. Plus keep in mind that the thresholds you are seeking to surpass can be "tempermental" (for the lack of a better word) where a z-value of (for example) 1.92 can't be, "oh, that's close enough . . . ".

As a data scientist, I would never want to sign my name to an experiment that I didn't have complete confidence in both the design and execution. If there is anything that can affect your final scores, you should absolutely consider scrapping it and starting over.


When doing logistic regression you start calculating a bunch of probabilities $p_i$ and your target is maximize the product of those probabilities (as they're considered independent events). The higher the result of the product the better is your model.
As we are dealing with probabilities we are multiplying numbers between 0 and 1, therefore, if you multiply a lot of those numbers you would get smaller and smaller results. So we need a way to move from probabilities multiplication to a sum of other numbers.
Then is when $ln$ function enters in to play. We can use some of this function properties such as:

$ln(a b) = ln(a) + ln(b)$.
When our prediction is perfect i.e. 1, the $ln(1) = 0$.
$ln$ lower than 0 are growing negative numbers e.g. $ln(0.9) = -0.1$ and $ln(0.5) = -0.69$.

So we can move from maximizing the multiplication of probabilities to minimizing the sum of the $-ln$ of those probabilities. The resulting cross-entropy formula is then:

$$
- \sum_{i=1}^m y_i ln(p_i) + (1-y_i) log (1-p_i)
$$

If $y_i$ is 1 the second term of the sum is 0, likewise, if $y_i$ is 0 then the first term goes away.
Intuitively cross entropy says the following, if I have a bunch of events and a bunch of probabilities, how likely is that those events happen taking into account those probabilities? If it is likely, then cross-entropy will be small, otherwise, it will be big.


One option is to strip the output from the .ipynb file. Then the git diff would only track the cell data. One package that strips the output Jupyter Notebook is nbstripout.


Multiply weights after using dropout in training - PyTorch

PyTorch handles this with scaling the output of the dropout layer at training time with this probability:

So it handles this itself after applying model.eval(). You can lookup their code here:


AttributeError: module 'tensorflow.python.keras.utils' has no attribute 'to_categorical'

Use keras>=2.2 and tensorflow >=1.14 to resolve the issue.


It turns out that the red shape in each cluster indicates the cluster centroid, which is the shape that all the other shapes in the cluster are compared with. There is an offset occasionally because it's a centroid, and there may be a some variance within the cluster. For that reason, the standard deviation set in the k-shape algorithm can affect the centroid, with larger standard deviations contributing to less "exact" centroids.


Using RNN (LSTM) for predicting one future value of a time series

I recently did a lot of reading and code writing on LSTMs so I'll try to pitch in and answer the question, though I'm not yet familiar with Theano Keras or deeplearning4j. From a quick scour of the Internet it seems that the meaning of batch_size is program-dependent, so it might be a good idea to check the online Help files. From what I gather though, it can refer to a couple of different things:
1) the count of cases fed into the training algorithm for processing before the next examination of inputs and/or outputs.
2) the number of iterations of the training algorithm over the last set of input before the next examination of inputs and/or outputs.
3) the size of sequences fed to the LSTM algorithms.

With that in mind it seems almost certain that Theano and deeplearning4j are referring to #1 and/or #2, given the following links. On the other hand, I've run across journal articles recently, where batch size referred to sequence size, so watch out for a potential mismatch between the software and academic terminology:

☻ This post at CrossValidated indicates that in deeplearning4j and Keras the batch_size should be set to 1 for online learning. So in other words, batch_size controls the number of training cycles before the next input/output check.

☻ In Keras, the batch_size apparently sets the number of training cycles to execute or # of cases to train on before checking the inputs and/or outputs again, according to this Github page.

☻ This Github package also seems to use a different variable to denote sequence length.

☻ From Tony Kanh's answer at this StackOverflow page, in TensorFlow the batch_size determines the size of the output vector. This is probably identical to the number of items processed in each training cycles, since multiplying it by the num_steps and size (probably the sequence size?) determines the dimensions of the output.

☻ For closely related neural nets like Bidirectionals, the batch size is apparently equivalent to the sequence size, not the number of cases fed into the training algorithm. See p. 5, Berglund, et al., "Bidirectional Recurrent Neural Networks as Generative Models," available at the Cornell University Library website without a paywall.

As I said I haven't used Theano and deeplearning4j yet, but I hope that at least provides a starting point to find the correct answer.


I'm not sure that it going to work.  Take a look at the links below.

https://spark.apache.org/docs/latest/ml-classification-regression.html
https://spark.apache.org/docs/latest/ml-clustering.html

Find an algo from something in there and run it.  It should work fine.


What does a deep learning model look like in file?

Yes you are right. It depends on which frameworks you are using for training the model.

For example, if you use tensorflow framework, the trained model will be saved into several files. You can save and load model into two ways:

Checkpoint
Savedmodel (freeze)

Please check this link. Here

In keras, Model weights are saved to HDF5 format. This is a grid format that is ideal for storing multi-dimensional arrays of numbers. you can save and load your model to file:

Save Model to JSON.
Save Model to YAML.

The JSON format of the model looks like the following:

    {
   "keras_version":"2.0.2",
   "backend":"theano",
   "config":[
      {
     "config":{
        "dtype":"float32",
        "bias_regularizer":null,
        "activation":"relu",
        "bias_constraint":null,
        "use_bias":true,
        "bias_initializer":{
           "config":{

           },
           "class_name":"Zeros"
        },
        "kernel_regularizer":null,
        "activity_regularizer":null,
        "kernel_constraint":null,
        "trainable":true,
        "name":"dense_1",
        "kernel_initializer":{
           "config":{
              "maxval":0.05,
              "minval":-0.05,
              "seed":null
           },
           "class_name":"RandomUniform"
        },
        "batch_input_shape":[
           null,
           8
        ],
        "units":12
     },
     "class_name":"Dense"
  },
  {
     "config":{
        "kernel_regularizer":null,
        "bias_regularizer":null,
        "activation":"relu",
        "bias_constraint":null,
        "use_bias":true,
        "bias_initializer":{
           "config":{

           },
           "class_name":"Zeros"
        },
        "activity_regularizer":null,
        "kernel_constraint":null,
        "trainable":true,
        "name":"dense_2",
        "kernel_initializer":{
           "config":{
              "maxval":0.05,
              "minval":-0.05,
              "seed":null
           },
           "class_name":"RandomUniform"
        },
        "units":8
     },
     "class_name":"Dense"
  },
  {
     "config":{
        "kernel_regularizer":null,
        "bias_regularizer":null,
        "activation":"sigmoid",
        "bias_constraint":null,
        "use_bias":true,
        "bias_initializer":{
           "config":{

           },
           "class_name":"Zeros"
        },
        "activity_regularizer":null,
        "kernel_constraint":null,
        "trainable":true,
        "name":"dense_3",
        "kernel_initializer":{
           "config":{
              "maxval":0.05,
              "minval":-0.05,
              "seed":null
           },
           "class_name":"RandomUniform"
        },
        "units":1
     },
     "class_name":"Dense"
  }

   ],
   "class_name":"Sequential"
}


Using RNN (LSTM) for Gesture Recognition System

Based on my understanding, this seems to boil down to a multi-class classification problem (each set/sequence of hand gestures eventually corresponding to a word/symbol). An RNN/LSTM/GRU architecture is a great place to start, but will be computationally expensive and rather slow to train. However, whether you absolutely need one or more of these depends on how your data is structured, and whether you can get away with treating your images as a set, rather than a sequence.

Here are some potential ideas:

Start by setting up a CNN layer for each "RUN_*" folder, along with
dropout and/or flattening between each layer.
Start by setting up a single long vector representing all stages of
the sequence, then add a couple of layers of CNNs, also with
dropout and/or flattening.

This will take some exploration and tuning on your end to see what works the best, and you will need to choose with evaluation metrics you want to optimize for (e.g. precision/recall)

The reason I'm suggesting a CNN is because, in certain cases, CNNs can provide similar performance to a recurrent model, but at a fraction of the cost. Unfortunately, it's tough to tell with little to no information about you data.


Why cant RNN learn long term dependencies=?

There is no explicit notion of memory (like gates in lstm and gru)

Gates are a way to optionally let information through, ommiting this functionality we will just be updating weights that will in process fade away hence longer memory is hard to learn.


Anomaly score calculation for multidimensional data set

One very simple yet efficient way, is auto-regression, which means, you train a regressor on past data, and predict the future, if the prediction is off by too much - then you call it an anomaly.

to make it more formal, if we denote $\vec{v}_{t0}$  as your data at timestamp $t_0$, we train a regression model $R$ that learns from $\{v_{t_0},\dots,v_{t_{n-1}}\}$.

Your anomaly score is a function of $R({v_{t_0},\dots,v_{t_{n-1}}})-v_{t_n}$


Encoding with OrdinalEncoder : how to give levels as user input?

I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though.
OrdinalEncoder is capable of encoding multiple columns in a dataframe. So, when you instantiate OrdinalEncoder(), you give the categories parameter a list of lists:
enc = OrdinalEncoder(categories=[list_of_values_cat1, list_of_values_cat2, etc])

Specifically, in your example above, you would just put ['low', 'med', 'high'] inside another list:
end = OrdinalEncoder(categories=[['low', 'med', 'high']])
enc.fit_transform(X.loc[:,['animals']])
>>array([[0.],
         [1.],
         [0.],
         [2.],
         [0.],
         [2.]])
# Now 'low' is correctly mapped to 0, 'med' to 1, and 'high' to 2

To see how you can encode multiple columns with their own individual ordinal values, try this:
# Sample dataframe with 2 ordinal categorical columns: 'temp' and 'place'
categorical_df = pd.DataFrame({'my_id': ['101', '102', '103', '104'],
                               'temp': ['hot', 'warm', 'cool', 'cold'],
                               'place': ['third', 'second', 'first', 'second']})

# In the 'temp' column, I want 'cold' to be 0, 'cool' to be 1, 'warm' to be 2, and 'hot' to be 3
# In the 'place' column, I want 'first' to be 0, 'second' to be 1, and 'third' to be 2
temp_categories = ['cold', 'cool', 'warm', 'hot']
place_categories = ['first', 'second', 'third']

# Now, when you instantiate the encoder, both of these lists go in one big categories list:
encoder = OrdinalEncoder(categories=[temp_categories, place_categories])

encoder.fit_transform(categorical_df[['temp', 'place']])
>>array([[3., 2.],
         [2., 1.],
         [1., 0.],
         [0., 1.]])


Can I use an ANN to translate image output from one sensor to simulate output from another sensor?

This is very much possible. There is a function which can map the images from the higher resolution pictures to the lower dimensional ones; and a neural network can be trained to learn that function.

However, to train a neural network to do this you will need thousands of images from both cameras. Then you can feed the pictures taken with your higher resolution camera as the input to the network and then compute the loss of the network at the output with the corresponding lower resolution images.

If you do not have so many images, there has been work on taking images and applying some sort of filter to change their appearance. These techniques are often called style transfer, you can find some tutorial here and code which I have tried and can confirm works here. It might be hard to get a representative image to use as the style image using your old camera. You can try an average of a few pictures, or a picture of a white background, you would have to try things out, I do not know what would work in this case.

If you share examples of your data we can help you more.


How to store and analyze classification results with Python?

Sacred is a python library developed by the IDSIA lab that "facilitates automated and reproducible experimental research". It is available through pip as sacred.

For a related discussion see reddit.


Reading accuracy in Keras from the saved model

According to the official Keras website, you have to use:

keras.models.load_model(filepath)

Example:

model = load_model('my_model.h5')

This will load your saved H5 model to 'model' and then you can try:

model.evaluate(x,y)

To return the loss value & metrics values for the model in test mode.


According to the documentation, the value is not exactly the area under curve, it is
$$\text{AP} = \sum_n(R_n - R_{n-1})P_n.$$
which is a rectangular approximation.

For your specific example, i.e.

      R     P
  1   0.0   1.0
  2   0.5   1.0
  3   0.5   0.5
  4   0.1   0.66

it is calculated as
$$\begin{align*}
\text{AP} & = \overbrace{(0.5 - 0.0)\times1.0}^{(R_2 - R_1)P_2} + \overbrace{(0.5 - 0.5)\times 0.5}^{(R_3 - R_2)P_3} + \overbrace{(1.0 - 0.5) \times0.66}^{(R_4 - R_3)P_4} \\
&= 0.5 + 0.00+ 0.33 = 0.83
\end{align*}$$
which is the area under the red curve as illustrated below:

compared to $$\text{AUPR}=0.5 + \overbrace{\frac{0.5 + 0.66}{2}  \times 0.5}^{\text{trapezoid area}} = 0.79$$
which is the area under the blue curve.


Ethically and Cost-effectively Scaling Data Scrapes

For many APIs (most I've seen) ratelimiting is a function of your API Key or OAuth credentials. (Google, Twitter, NOAA, Yahoo, Facebook, etc.) The good news is you won't need to spoof your IP, you just need to swap out credentials as they hit there rate limit.

A bit of shameless self promotion here but I wrote a python package specifically for handling this problem.

https://github.com/rawkintrevo/angemilner

https://pypi.python.org/pypi/angemilner/0.2.0

It requires a mongodb daemon and basically you make a page for each one of your keys. So you have 4 email addresses each with a separate key assigned. When you load the key in you specify the maximum calls per day and minimum time between uses.

Load keys:

from angemilner import APIKeyLibrarian
l= APIKeyLibrarian()
l.new_api_key("your_assigned_key1", 'noaa', 1000, .2)
l.new_api_key("your_assigned_key2", 'noaa', 1000, .2)

Then when you run your scraper for instance the NOAA api:

url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations'
payload= {  'limit': 1000,
        'datasetid':  'GHCND',
        'startdate': '1999-01-01' }

r = requests.get(url, params=payload, headers= {'token': 'your_assigned_key'})

becomes:

url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations'
payload= {  'limit': 1000,
            'datasetid':  'GHCND',
            'startdate': '1999-01-01' }

r = requests.get(url, params=payload, headers= {'token': l.check_out_api_key('noaa')['key']})

so if you have 5 keys, l.check_out_api_key returns the key that has the least uses and waits until enough time has elapsed for it to be used again.

Finally to see how often your keys have been used / remaining useage available:

pprint(l.summary())

I didn't write this for R because most scraping is done in python (most of MY scraping).  It could be easily ported.

Thats how you can technically get around rate limiting.  Ethically ...

UPDATE The example uses Google Places API here


Does the bias of an artificial neuron adjust or remain constant during training?

Neurons values are something like this

$N_i = w_{i1}x_{i1} + w_{i2}x_{i2} + ... + w_{in}x_{in} + b_i$

the bias is $b$ here

but it also can be written like

$N_i = w_{i0} + w_{i1}x_{i1} + w_{i2}x_{i2} + ... + w_{in}x_{in}$

where bias is $w_{i0}$

it is the weight of index 0

when mentioned weights in optimization it is by default include bias


Backpropagation: Relevance of the error signal of a neuron

Let me explain one thing very clearly "NEURAL NETWORK ARE VERY SIMPLE STRUCTURES AND YET VERY EFFICIENT ALGORITHM WHICH GIVES AMAZINGLY GREAT RESULTS AND IF YOU SEE CLOSELY NEURAL NETWORKS ARE ACTUALLY BIG COMPOSITE FUNCTIONS WHICH REQUIRE YOU TO CALCULATE GRADIENTS SO THAT YOU CAN OBSERVE HOW THE OUTPUT VARIES ACCORDING TO YOUR INPUT WEIGHTS(That is what is backpropagation achieved using gradient descent) THAT IS SIMPLY THE GOAL I.E. TO CALCULATE DERIVATIVES OF THE TWO EQUATIONS YOU HAVE MENTIONED (2) AND (3)".

Now as you stated you can calculate its value and reuse it i.e. eq(1) because eq (1) here gives you the error at a neuron j in layer 'l' with respect to that neuron's weighted sum of inputs it is useful and necessary for calculation of weights when the loss is calculated at the'Lth layer(last layer)' it is then backpropagated to the L-1 th layer where gradient is calculated and then weights are updated it is the eq(1) that allows you to compute the gradients of a neuron j in layer 'l' and then the weight matrix of that neuron j is updated so essentially eq(1) allows you to use eq(2) and eq(3) to optimize your weights in order to minimize your loss function.

One more thing if you calculate its derivative that is of eq(1) its partial derivative it will be: first-rate of change of cost function w.r.t to activation function * rate of change of activation function w.r.t to weighted sum of inputs * rate of change of weighted sum of inputs w.r.t to weights of your neuron so this is essentially a chain in itself which gets longer and longer as the loss is backpropagated through l-1,l-2,l-3.... layers and so on, thus eq(1) is actually calculated using the first two in the chain stated above which is then used to calculate the rate of change of weights w.r.t to your loss function. This is why whenever you read about backprop this equation is bound to be seen as it is kind of connection mechanism that allows you to relate your loss with neurons in the subsequent layers during backpropagation and subsequently update their weights. Hence according to me, eq(1) in itself is the true essence of Backpropagation using gradient descent.


Are Correlithm Objects used for anything in the industry?

Looks like lots of marketing around a specific case of what everyone was already doing to sound bigger than it is.  So, are people implementing mathematical frameworks in software to extract and use the structure within data?  Absolutely.  Are people using the word 'Correlithm' when doing it?  Not as far as I've seen.


plotting a decision tree based on gridsearchcv

While I don't have a module named graphviz I can still try to help. Reading the documentation for GridSearchCV, I can see that there's a attribute called best_estimator_ that provides the estimator that was chosen by the search. By applying .best_estimator_ to your sample code, it seems be working fine.

clf.fit(iris.data, iris.target)
dot_data = export_graphviz(clf, out_file=None, filled=True, rounded=True,
                                class_names=['0','1','2'])
-------------------
AttributeError: 'GridSearchCV' object has no attribute 'tree_'

With .best_estimator_

clf.fit(iris.data, iris.target)
dot_data = export_graphviz(clf.best_estimator_, out_file=None, filled=True, rounded=True,
                                class_names=['0','1','2'])

I get no error. I hope this helps


Does Sampling size matters in Multi classification Model

It is always better to keep sample sizes close each other. The problem you are facing is Imbalanced Classification.
There are lots of methods you can apply such as upsampling/downsampling, synthetic data generation (check SMOTE).

Model:

I would first convert the model to binary classification such that:

model 1 predicts: A or not A
model2 predicts B or not B in "not A" group

Note: Another option three x or not x classifier for three classes)

Then I would apply those sampling techniques (see above) to models. If you have high number of observations try to use downsampling to make class weights as 50:50, if you cant do that try SMOTE to generate data from minority class.

Metrics:

Note that if you don't have balanced dataset, try not to use accuracy as a performance metric. Consider area under ROC, F1 and precision/recall according to your case.

Hope it helps!


Spatial Transformer Networks vs Deformable Convolutions

The main difference, as they mentioned in the paper, is that STN has a global parameter to transform the features. That is, it computes one set of parameters to transform the input. DC computes a 2D offset map for each location in the input, so for each location in the input features there is a separate transformation.


If you are looking for a statistical trick, I don't know, but
Recently Andrew NG team recently published about NGBoost.

NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a modular boosting algorithm for probabilistic predictions.

In this Towards Data Science toy example you can see how to use the Python API:
Quoting the TDS author:

NGBoost’s one of the biggest differences from other boosting algorithms is can return probabilistic distribution of each prediction. This can be visualized by using pred_dist function.

In their web they say "NGBoost can be used with any base learner, any family of distributions with continuous parameters, and any scoring rule." So it seems pretty flexible to do whatever you consider.


Error Analysis for misclassification of text documents

It seems to me that both of your questions could be answered by storing the retrieved neighbours on your test set and giving them a thorough analysis. Assuming you are using a unigram + tf-idf text representation and a cosine similarity distance metric for your K-NN retrieval, it would be trivial once you have a classified document to display the K neighbours and analyze their common unigrams and their respective tf-idf weights in order to see what influenced the classification. Moreover, doing it on your misclassified documents could help you understand which features caused the error.

I'd be interested to know if there is a more systematized approach to those issues.


What is the best approach for send time optimization?

Generally, the best method to find optimal send time is A/B testing. Send the same emails/notification at different times to different people. Then see if one group tends to open the emails more than another group.


This was meant as a comment but it is too long.

The fact that your test set has a different range might be a sign that the training set is not a good representation of the test set. However, if the difference is really small as in your example, it is likely that it won't affect your predictions. Unfortunately, I don't think I have a good reason to think it won't affect a SVM in any circumstance.

Notice that the rationale for using MinMaxScalar is (according to the documentation):

  The motivation to use this scaling include robustness to very small
  standard deviations of features and preserving zero entries in sparse
  data.

Therefore, it is important for you to make sure that your data fits that case.

If you are really concerned about having a difference range, you should use a regular standardization (such as preprocessing.scale) instead.


How to treat patients without events in time-to-event analysis?

Generally, I would pick a value that is vastly outside the range of values that are non-imputed in this column since that'd probably cause more extreme (noticeable) errors in case they get included. So rather 9999 than -1. However, as you probably already attempted, the "most correct" solution would be to set it to NaN since the column is just not defined in these cases. If at any step during your analysis, some computation is done on the TimeSinceLastEvent column, anything but NaN would technically lead to incorrect results.
What is your downstream analysis strategy? Can you maybe split the data into two subsets (one with heart attacks, one without) and analyse them separately?


Updated: Measuring distance from camera to object

Perhaps you could search and find tons of materials. This is a well-established task, for example this tutorial, of more advanced methods combining object detection with distance measurements, see Object-Detection-and-Distance-Measurement, and references at the bottom. In a nutshell, when you know details about your camera focal length and the details about the object in the image, it is a simple formula to estimate the distance base on those details. I am guessing that you may need to do some calibration, but I haven't done this one myself, so I wouldn't know for sure.

Earlier I have had: Measuring distance between objects

One idea that might be worth exploring is to segment your image, for example use the recent package by Facebook Detectron2, that seems to be capable of segmenting people out-of-the-box. Once segmented (filtered only for humans), for each segmentation calculate the center of segment (Centroid) or something similar assigned to each person. And given that would be (x,y) coordinate, you can easily calculate distance from a person to all neighboring people.

For the center of segment , OpenCV is quite handy. See this post Find the Center of a Blob (Centroid)! Or this great tutorial to estimate distance between objects in an image with OpenCV!

In a nutshell: Image Segmentation -> Centroid -> Distance


I too am working on same problem, found these below links very useful in getting started on oversampling and under sampling-

https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/

https://en.wikipedia.org/wiki/Oversampling_and_undersampling_in_data_analysis


How to avoid overfitting in Reinforcement Learning

In RL this is known as the exploration-exploitation trade-off, so i don't think you can avoid this regularizing the neural network, but rather the action policy it learns. The agent you are training is learning to solve a specific task and its policy is "over-fitting". You could tackle this in several ways, one of which is to encourage exploration by using a distribution over the Q Values such as Softmax

$$p(a|s) = \frac{\exp(\beta Q(a,s)}{\sum_{a'} \exp(\beta Q(a',s))},$$

or by applying a $\epsilon$-greedy action selection, where you choose a random action with probability $\epsilon$ and otherwise the max Q Value. There are a lot of other ways to achieve this, just to quick "exploration strategies for dqn" search on google scholar.


So how can I couple my clusters to a feature label from my feature matrix?

Principal Components are not intuitive features. What seems common here is to cluster users based on PCs and investigate clusters based on original features afterward i.e. extract different clusters and plot data based on different subsets of features and use different colors for different clusters. It might give you some intution.

These two paths can be seen in many PCA results where the information varies with e.g time.

For instance in your case a beginner user visits less than an old one and the number of new users are most probably higher than old ones so data will be more dense around origin and far it gets from the origin less dense it becomes. Such a phenomenon affects your PCs as well so you'll see some paths along different lines in PC space.

Hope it helps :)


Auto.arima with xreg in R, restriction on forecast periods

Using xreg suggests that you have external (exogenous) variables. In this, a regression model is fitted to the external variables with ARIMA errors.

When forecasting you need to provide future values of these external variables. In practice, these are often forecasts or could be known. For example, if you're trying to predict Sales and you use Advertising spend as an external variable, you may know the advertising spend for the upcoming year.

auto.arima then produces forecasts for the length of xreg, therefore disregarding h.

Based on your comments below, I've provided an example script demonstrating this based on the Sales example above.

library(forecast)

# Generate sample data
sales <- sample(100:170, 4*10, replace = TRUE)
advertising <- sample(50:70, 4*10, replace = TRUE)

# Create time series objects.
sales_ts <- ts(sales, frequency = 4, end = c(2017, 4))

fit <- auto.arima(sales_ts, xreg = advertising)

# If we pass external_regressor into the forecast, h will be disregarded and we will
# get a forecast for length(external_regressor)

wrong_forecast = forecast(fit, h = 4, xreg = advertising)

length(wrong_forecast) # Will be 40

# To forecast four quarters in advance, we must provide forecasted external regressor data
# for the upcoming four quarters, so that length(new_regressor) == 4.
# In reality, this data is either forecasted from another forecast, or is known. We'll randomly generate it.

upcoming_advertising <- sample(50:70, 4, replace = TRUE)

correct_forecast <- forecast(fit, xreg = upcoming_advertising)

length(correct_forecast$mean) # Will be 4

The key things to note are:

If we forecast with the same regressors as we did when generating the forecast, h will be disregarded and a forecast will be generated for the length of xreg in your case, 10 years.

As such, we must provide new data for xreg for the length of time we wish to forecast - in your case, 4 quarters.


Using random forest when not all the observations have data

In theory decision trees (and random forests) are able to deal with missing values in the data. But whether a particular implementation of the algorithm allows this (and how to use it with this implementation) depends on the specific package.


How is it possible for RNN to do sentiment analysis?

RNNs do not learn to predict sentiment. They learn correlations between the input data and the target labels. If they see that every time the input contains the word "bad" they have to generate the label "negative", then they will learn it. If they see in the training data that the previous phenomenon happens always except when there is a "not" before "bad", then they will learn it. Depending on how the data distribution is, they may generalize the negation to any combination of verbs and adjectives, or maybe not, and they only handle negation appropriately when the input data is very very similar to the training data.
Your question seems to aim at understanding the "internal dynamics" of the RNN when it is predicting sentiment. While there is some research in that direction, I think they can still be considered as "black boxes", in that we do not actually understand the functions modeled by neural networks and therefore their outputs are not "explainable"


Forecasting an individual based on a representative group

I think you are doing a supervised classification problem.  You have labelled data — previous customers, which you could label as different groups (e.g. high spend / low spend) along with predictors about the customer — their details (e.g. age).

You could think of each of these customer details representing dimensions on a plot, wth your task being to find out where different groups of customers are clustered along these dimensions.  Once you know that you can predict what group a new customer will likely belong to, based on their details.

There are many methods for solving this sort of problem, including but not limited to support vector machines, gradient descent and nearest neighbour.  Which one you choose will depend on the amount and sort of data you have.  A really good guide if you are starting out is this flowchart in the SciKit learn documentation, which isn't exhaustive, but gives a useful overview.


What is the best solution to replace NaN values?

You are right in saying that replacing with a simple mean, mode... is a common but unreliable imputation strategy in many cases.
You have in scikit learn some utilities for imputation of missing values (have a look at https://scikit-learn.org/stable/modules/classes.html#module-sklearn.impute) using for instance the knn imputer as an additional strategy.
Take into account you cannot assume your feature of interest follows a normal distribution, so instead you can actually apply a kernel density estimator to model such distribution, see here: http://scikit-learn.org/stable/modules/density.html


Not able to interpret decision tree when using class_weights

We should consider two points. First, class_weight='balanced' does not change the actual number of samples in a class, only the weight of class $w_{c_i}$ is changed. Second, the [un-normalized] probability of class $c_i$ in each node is calculated as

$w_{c_i}$ x (number of samples from $c_i$ in that node / size of $c_i$)

For example, in balanced mode, the [un-normalized] probability of $c_3$ in the green leaf is calculated as

$33.\bar{3}\% \times (3 / 36) ≈ 2.778\%$

compared to $36\% \times (3 / 36) = 3\%$ in unbalanced mode.

The probability (normalized) in balanced mode would be:

$100 \times 2.778/(2.778+32.258) \% = 7.9289\%$

Remark. The word "probability" is not applicable to each isolated node  except for the root node. This is the un-normalized version of the probability used to classify a data point inside a leaf, though the normalization is not required for comparison. However, the notion is applicable to the aggregate of nodes at the same level and the leaves from upper levels (i.e. set of all samples).


Using ARIMA parameters when transforming time series to Supervised Learning

Yes. It could be use. if you find the lag pattern by using acf, pacf , you can generate the rolling mean or mid value time series as part of feature engineering. And use the combined feature to do supervised learn.

However, one of key important thing need note: time series based cross validate in the begining to avoid data leaking, other wise you metric will be unreasonable good.


Get feature importance for each observation with XGBoost

I would suggest you're probably looking for something like one of these two packages:

https://github.com/slundberg/shap
https://github.com/marcotcr/lime

See this notebook of lime's for example, which shows how you can use it to see why a specific sample in your data resulted in the prediction from the model:

https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html


MPI, MapReduce, or Spark for complex datasets and processing

If I understand your question correctly. What you want to do is

Write a sql query against to a database to get the processed/aggregated results first.
Execute several queries in parallel.

The first thing first, if you want to run MapReduce programs on your data, you need to distribute your data. In addition, MapReduce doesn't run your queries to different nodes separately. If your $query_{A}$ runs on $data_{A}$, what we do here is to divide $data_{A}$ to different nodes and run $query_{A}$ on these nodes on $PartOfData_{A}$ in parallel. By the way, you can use Sqoop to migrate the data from your DB to HDFS(it is used to store data separately).

Here are steps for you:

migrate all/part of your data to HDFS
transform your queries to MapReduce version program


What is "spatial feature encoding"? Can anyone give a concrete example?

Chatfield et al. provides a good survey of feature encoding methods.

From the introduction to that paper (emphasis my own):

  The typical object recognition pipeline is composed of the following three steps: (i) extraction of local image features (e.g., SIFT descriptors), (ii) encoding of the local features in an image descriptor (e.g., a histogram of the quantized local features), and (iii) classification of the image descriptor (e.g., by a support vector machine).

I believe that "spatial feature encoding" refers to step (ii) in the typical pipeline. In this respect, I believe that the simplest possible example (and a standard one) would be the histogram encoding. If "spatial feature encoding" is meant to imply the preservation of some geometric information, than spatial binning is another technique to look at.


Efficient way to create matrix that shows if data exits per day

You should be able to just use a pandas pivot table for this purpose:
import pandas as pd

pd.pivot_table(df, index="ID", columns="Date", aggfunc="count")


Can we combine multiple K-Means Models as a single model?

I agree with your assumption, the vector space is the same so I don't see any major problem with this approach.
Still this approach might cause some more subtle bias, depending on the differences between the models (sets of terms, number of clusters). I could imagine the following problems happening:

if there is a big difference in number of clusters between models, a model which has more clusters is more likely to contain the closest match, simply because it has more centroids. This might favour the most precise clusters (this might actually be a good thing, depends).
if there are many models sometimes there might be many close centroids across the models, and this would probably make the selection of the closest among them almost random: the exact position of a centroid is significant with respect to other centroids within the same model, not necessarily with respect to other centroids outside the model.


Eigenvalues of covariance matrix are negative

You seem to be using a generic eignvalue solver. Perhaps your matrix is badly conditioned and the aglorithm cannot pick it up. Lots of zero eigenvalues with few massive ones certainly seem suspicious. Have you tried solvers specific for hermitian/symmetric matricies? eigh & eigvalsh
Also, I would z-score all your features, and drop all features with zero variance. This will set an upper bound on maximum possible eigenvalue since trace of your matrix will be the number of the features, and since positive definite matrix has only positive eigenvalues, the largest eigenvalue will be no larger than the trace.


How can I check if a bigger training data set would improve my accuracy of my scikit classifier?

One idea:

Split your data into train / hold out datasets.
Train the model on a fraction of the training data (say 50%) and test on the holdout dataset.
Train the model on a larger fraction of the training data (say 75%) and test on the holdout dataset.

It's important that you use the same holdout data for testing so you can perform a true test of accuracy.

Since you're doing classification, you should check that your data is balanced, and adjust if not (this may also improve your accuracy without needing larger training data).


You have two options:

1) Let the K-means algorithm run for a large number of iterations (if on sklearn, change the max_iter parameter value for sklearn.cluster.KMeans). It will eventually converge to a good result (but it will take more time)

2) Make and "educated guess" for the initial starting point. One way to do that is to transform your data in a space where you know your points can only lie within a specific region: from there, you can evaluate the best seeds where to start the K-means. For a clearer explanation, see this article (for a quick overview, look at the related slides, in particular at step 3, slide number 10)


Setting up Neural Network for this problem

Yes, you can use a neural network exactly with the architecture you have described. Just define a consistent mapping from your grid to your input and output layer shapes. For example square $(x,y)$ can map to input neuron $5*y + x$ for zero-based indexing.
If square (3,3) is never on in your training input data then it will be hard to predict what will happen in an unseen example where it is on, so the short answer is no. However, if you think your data exhibits any special structure you might be able to leverage this to still make a good prediction.

If you think nearby squares behave similarly you might get good results with a convolutional neural network, which can learn from similar patterns nearby but possibly not exactly on the square you are interested in.
If you think certain patterns of squares behave similarly despite translation or rotation you could augment your training data with translations and rotations and you might then have training examples where (3,3) is on.

With all that said, 50 training examples is low so it might be hard to get good results.


LightGBM - Why Exclusive Feature Bundling (EFB)?

I've read that paper so many times before in so many ways. What I can say on the matter is that the paper does not describe explicitly what the framework particularly does. It just gives an hint of their intuitive idea of bundling of the features in an efficient way. But specificly, it does not say that it does a 'reversion of one-hot-encoding' in particular to your question.

I tried giving categorical inputs directly and as one-hot-encoded to compare the time that it takes to compute. There was a significant difference: giving directly was all better in multiple datasets compared to giving as one-hot-encoded.

Possibilities:

1)It is possible that LightGBM Framework can find out that we give the features as one-hot-encoded from the sparsity, it is possible that the algorithm does not treat one-hot-encoded with EFB.

2)It is also possible that LightGBM uses EFB on one-hot-encoded samples but it may be harmful, or not good as EFB on direct categorical inputs. (I go for this one)

But still, I do not think that EFB will reverse one-hot-encoding since EFB is explained as a unique way of treating the categorical features. But it possibly 'bundles the unbundled  features' when treating one-hot-encoded inputs.

I used the word 'probably' so much times out of implicitness of the paper. What I can advice to you is that to send an e-mail to one of the authors of the paper, I do not think that they would refuse to explain it. Or if you are brave, go for the GitHub Repo of LightGBM, to check the codes by yourself.
I hope that I could give you an insight. If you come up with an exact answer on the matter, please let me know. Please do not hesitate to further discuss this, I'll be around. Good luck, have fun!


Given a regression based model with many feature variables; what tools would you utilize to figure out which feature variables add the most variance?

There might be a smarter method but I would simply try to fit a model without $X_i$ for every feature $X_i$ (and also a reference model with all the features). By contrast the model where $X_{49}$ is removed should obtain the lowest variance if $X_{49}$ is responsible for a lot of variance.
Be careful that in general a feature which causes a lot of variance is an important one, since if it wasn't important then it wouldn't have much impact on the target.


Linear kernel in SVM performing much worse than RBF or Poly

This depends very much on your dataset: There is data, which can not be separated using a linear kernel. It is impossible, and you won't get any reasonable result. Take for example the simple XOR function. The plot of the XOR looks like this:

No matter how hard you try, you will never be able to separate this using a linear SVM, or any linear method.

Here's some short MATLAB code which illustrates the problem:

x = [0 0; 0 1; 1 0; 1 1];    % training points
y = [0; 1; 1; 0];            % target: XOR
linearSvm = fitcsvm(x,y);    % train with linear kernel
linearSvm.predict(x)

>> ans =
        0
        0
        0
        0

The SVM has no chance of finding a solution. Now if we use a kernel, we create a mapping to a higher-dimensional space. The kernel trick allows us to implicitly do this mapping, by calculating a kernel function, which returns the result of a scalar product in this high-dimensional space, without needing to map the input values to this space. The RBF kernel is special, as it corresponds to an infinite-dimensional space.
Now if we specify a RBF kernel and run the same example again, then:

gaussSvm = fitcsvm(x,y,'KernelFunction','rbf');  % RBF kernel
gaussSvm.predict(x)

>> ans =
        0
        1
        1
        0

the SVM easily finds the correct result. The same result (in this XOR case) is also found when using a polynomial kernel. There are many more examples, where data simply can't be separated linearly, for example:

Of course, if your data is almost linearly separable, then you can (and probably will) get good results with a linear kernel. Using soft-margin SVMs, which allow for a few misclassifications, often leads to good results, as they can ignore outliers.

If the linear kernel gives no result, you will have to accept that your data is not (well) linearly separable. You'll either have to switch to a kernel, or allow for more misclassifications in the soft-margin SVM.