Prediction with non-scalar output (label)

Neural Networks can have a vector or matrix as output layer, image segmentation is a well researched topic and deep learning (as most things concerning images) are the state-of-the-art. You will need (a lot of) training examples where the pattern is found, and where. This could be a bounding box, or per pixel if it is part of the pattern or not (this will generate a matrix equally sized to your input). To see if the pattern is found you could construct a second network that is just a binary classifier, or you could try to see if your pixel-based network will output almost only zeros in case of no pattern. In this case you will need negative examples as well.


Manual install vs conda install tensorflow-gpu

Apparently all Cuda and CuDnn libraries are included with packages from Anaconda Cloud according to this guy:  https://www.pugetsystems.com/labs/hpc/How-To-Install-CUDA-10-together-with-9-2-on-Ubuntu-18-04-with-support-for-NVIDIA-20XX-Turing-GPUs-1236/

  You do NOT need to install CUDA 10 if; You are a developer working
  with CUDA 9.x or earlier in your code base and there are no compelling
  new features that you want to risk breaking your application to try.
  See the release notes. You are working with Python machine learning
  frameworks AND using Anaconda Python. Most of the frameworks have
  up-to-date builds available from Anaconda Cloud that include the
  needed CUDA libraries i.e. you don't have to install anything outside
  of the package you want. You are using GPU accelerated software
  packages that only depend on CUDA run-time support from the display
  driver. That includes most "ready to use" or commercial software. You
  don't know what CUDA is. In that case then, well, thanks for reading
  my post. I hope you enjoy it and learn something from it.

Check out this advice:  https://towardsdatascience.com/tensorflow-gpu-installation-made-easy-use-conda-instead-of-pip-52e5249374bc

He says just create a conda env like conda create --name tf_gpu tensorflow-gpu where tf_gpu is the env name and tensorflow-gpu is to include cuda and cudnn in that env.  Breaking it down into separate commands, it looks like:

conda create --name tf_gpu
activate tf_gpu
conda install tensorflow-gpu


Workflow for Iteratively Capturing and Improving Machine Learning Models

For Modelling , it's probably best to go with Partial learning models. Explore DASK.

If you can, getting cloud support to run your models is also excellent idea. For examples Google colab.

For all models keep log of model accuracy and cross validation accuracy. It's good idea to keep these details in small table and build a line graph which shows accuracy details for each version.

For latency, use Pytest and build differential testing code so that after each iteration of testing it provide tesrit results.


My understanding of your problem: you have realizations of $X$ (property value at first point in time) and $Y$ (value at second point in time). Want to fit a distribution for $Y-X$--more generally, a "model for the data."

One way of doing this would be plot a histogram of these values and then come up with plausible distributions that might fit the data, and then use maximum likelihood estimation to find the optimal parameters. If the models are nested, then you can use, e.g., the likelihood ratio test to determine which is a better fit to the data.

If not, then things are trickier. One good option is to hold out some of the data for testing, fit the models on the training data, predict the test data using each model, and see which one performs better.


Overfitting while fine-tuning pre-trained transformer

What makes you think your model is overfitting? Are you concerned about the difference between the training loss and validation loss?
If so, this is not overfitting. Overfitting is when the weights learned from training fail to generalize to data unseen during model training.
In the case of the plot shown here, your validation loss continues to go down, so your model continues to improve its ability to generalize to unseen data.
Once your validation loss starts creeping upward, then you have begun to overfit.
See chapter 5 of Jeremy Howard's Deep Learning for Coders with fastai and PyTorch for more details. https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527


How do various statistical techniques (regression, PCA, etc) scale with sample size and dimension?

Most of the efficient (and non trivial) statistic algorithms are iterative in nature so that the worst case analysis O() is irrelevant as the worst case is 'it fails to converge'.

Nevertheless, when you have a lot of data, even the linear algorithms (O(n)) can be slow and you then need to focus on the constant 'hidden' behind the notation. For instance, computing the variance of a single variate is naively done scanning the data twice (once for computing an estimate of the mean, and then once to estimate the variance). But it also can be done in one pass.

For iterative algorithms, what is more important is convergence rate and number of parameters as a function of the data dimensionality, an element that greatly influences convergence. Many models/algorithm grow a number of parameters that is exponential with the number of variables (e.g. splines) while some other grow linearly (e.g. support vector machines, random forests, ...)


Pruning and parameter reduction for decision trees

Pruning and feature reduction are different things.

Pruning: It basically compares the purity of the two leafs separately, and, together. If the leafs together are purer, than the two leafs are pruned. Thus, the decision over the parameter(s) at the node is wiped off.

Let's say you have N different parameters. You tree might be tall enough such that pruning has been used over all the parameters at different nodes. And in the same time, all these parameters might have been used in other nodes. If not, the decision tree will take the decision itself not to use this parameter - doesn't prevent from overfitting though.

Dimensionality Reduction:
It you reduce the number of parameters, then these parameters would never appear in your tree, at any node. Whereas they might have been relevant at some point.

They are not uncompatible, and performing a dimensionality reduction may increase the accuracy of your task for a further classifier (as decision trees).

However, decision trees are also used for dimensionality reduction: After being trained, one might scan the features importance within the decision tree, i.e. how much each feature is used to create a split a different nodes. based on this new knowledge, you can used only the most important features to train another classifier.


P-value mining on large number of combinations of variables

You need to investigate multiple hypothesis correction methods, like Bonferroni correction or Benjamini-Hochberg false discovery rate. The problem with this sort of analysis is that your associations are unexpected, so you don't have any a priori hypothesis. All you can do is test every combination, and then see what is statistically significant after accounting for all the tests.

As you do more tests, it becomes increasingly likely that you'll find an association just by chance alone. To combat this, multiple hypothesis correction methods make the p-value threshold for significance more conservative. If you do only 1 test, a p-value of 0.05 may indicate a significant result. If you do a million tests, however, many of them will have a p-value of 0.05 by chance, so you need to be more conservative. This practice is not generally frowned upon, since it appropriately takes into account the analysis methods. You can easily be dishonest with this approach, however, by doing many tests and only reporting the significant ones without correction.

Obligatory xkcd: https://xkcd.com/882/


Strictly speaking, "one class classification" does not make sense as an idea. If there is only one possible state of a predicted value, then there is no prediction problem. The answer is always the single class.

Concretely, if you only have spam examples, you would always achieve 100% accuracy by classifying all email as spam. This is clearly wrong, and the only way to know how it is wrong is to know where the classification is wrong -- where emails are not in the spam class.

So-called one-class classification techniques are really anomaly detection approaches. They have an implicit assumption that things unlike the examples are not part of the single class, but, this is just an assumption about data being probably not within the class. There's a binary classification problem lurking in there.

What is wrong with a binary classifier?


Avoiding Overfitting with a large LSTM net on a small amount of data

You may want to check a couple of hyperparameters that it appears you are not testing for in your code above:

Gradient clipping: large updates to weights during training can cause a numerical overflow or underflow often referred to as “exploding gradients.”

# configure your optimizer with gradient norm clipping
opt = SGD(lr=0.01, momentum=0.9, clipnorm=1.0)

Reccurent dropout: Dropout that is applied to the recurrent input signal of the units of the LSTM layer.

keras.layers.LSTM(24, kernel_regularizer=keras.regularizers.l1(0.01), ..., recurrent_dropout=0.3)

Stateful: Is it clear from the paper if the model retains its state with each training iteration. You can experiment with this as well.

keras.layers.LSTM(24, kernel_regularizer=keras.regularizers.l1(0.01), ..., stateful=True)


Unable to transform the variable to normal distribution

Based on your image, your data looks like it is uniformly distributed. You might try using more than 3 bins to look at the distribution more closely.
A few points related to your questions:

For uniformly distributed data, you won't be able to transform this data into a valid normal distribution
This distribution is fine for use in models, no need to change it.
Here is a good resource explaining assumptions about linear regression, this information also applies to many other algorithms


How to handle “not label Y” in a multi class machine learning problem?

I think you should start by looking into Multi-label Classification, as your problem seems to be a subset of MLC, where one example can have multiple correct labels. If one class is "Pictures with people in them" and another is "Pictures with guitars in them", one picture could certainly belong to both - this is an example of multi-label classification.

In your case, you could formulate your problem by considering that most examples have only one appropriate label, but 'Not A' has four: 'B', 'C', 'D', and 'E'. (If you have more specific information about the possible distributions of the correct label, you could use that information here too)

There are several methods for MLC, not all of which I'm familiar with, but I know plenty of research has been done on this using the CelebA image dataset, for example (it comes along with a number of potential label vectors).

Good Luck, and welcome to the site!


Why does Keras need TensorFlow as backend?

This makes more sense when understood in its historical context. These were the chronological events:

April 2009 Theano 0.1 is released. It would dominate the deep learning framework scene for many many years.
June 2015 Keras is created by François Chollet. The goal was to create an abstraction layer to make Theano easier to use, enabling fast prototyping.
August 2015 Google hires François Chollet.
November 2015 Tensorflow is released by Google, with much inspiration from Theano and its declarative computational graph paradigm.
December 2015 Keras is refactored to allow for pluggable backend engines, and now it offers backend implementations for Theano and Tensorflow.

Other backends were later supported by Keras (CNTK, MxNet), but they never got much traction.
Time passes by and the overlap between Tensorflow and Keras grows. Tensorflow ends up duplicating many of the functionalities in Keras (apart from the multiple APIs within Tensorflow that also had big overlaps).

September 2017 Theano is discontinued.
November 2017 Keras is bundled with Tensorflow as tf.keras. From this point on there are 2 different Keras: the one bundled with Tensorflow and the one that supports multiple backend engines. Both are maintained by the same people and are kept in sync.

At some point, the roadmap for Tensorflow 2.0 is defined, choosing to pursue an imperative model like PyTorch. The person leading the Tensorflow API refactoring is François Chollet. This refactoring included a reorganization of the functionality to avoid duplications.

November 2018 some crucial functionalities of Tensorflow are to be moved to tf.keras, generating a heated debate
September 2019 Keras 2.3 is announced to be the last release of the multi-backend version of Keras

Now, THE ANSWER to your question: Tensorflow is the most used Keras backend because it is the only one with a relevant user base that is under active development and, furthermore, the only version of Keras that is actively developed and maintained is one with Tensorflow.
So, summing up:

At the beginning of Keras, the overlap with Tensorflow was small. Tensorflow was a bit difficult to use, and Keras simplified it a lot.
Later, Tensorflow incorporated many functionalities similar to Keras'. Keras became less necessary.
Then, apart from the multi-backend version, Keras was bundled with Tensorflow. Their separation line blurred over the years.
The multi-backend Keras version was discontinued. Now the only Keras is the one bundled with Tensorflow.

Update: the relationship between Keras and Tensorflow is best understood with an example:
The dependency between Keras and Tensorflow is internal to Keras, it is not exposed to the programmer working with Keras. For example, in the source code of Keras, there is an implementation of a convolutional layer; this implementation calls package keras.backend to actually run the convolution computation; depending on the Keras configuration file, this backend is set to use the Tensorflow backend implementation in keras.backend.tensorflow_backend.py; this Keras file just invokes Tensorflow to compute the convolution


Looks like to me this is a classic imbalance binary classification problem (see comments above). What loss are you using ? It looks like your model is predicting the non-membership class because it’s minimising it’s averaging loss. Here are some techniques you might wanna try to solve this issue:

use regularization
over sampling the membership class
under sampling the non-membership class
select variables, linked to regularization above with l1 and l2 penalties
feature engineering: you specified you have 30 features but are all of them useful ? How do you preprocess them to feed the model ? Are they numerical or categorical ?

I hope this gives you some ideas about how tackling the problem. Changing the model won’t miraculously solve your issue.


Keras ANN Trained Model's Accuracy change on prediction

This problem is solved if we use same Mean and Standard Deviation which used for standardization of training samples.


If a dataset is imbalanced in real life, should I train on my machine learning model on an imbalanced dataset

When I undersample and train my classifier on a balanced dataset and test on a balanced dataset, the results are pretty ok

It's not surprising that the results are good since the job is easier in this case. It's actually a mistake to test on the artificially balanced dataset, since it's not a fair evaluation of how the system will perform with real data.

  Should I train the model on the original imbalanced dataset if I want it to perform well on real world test data that is also imbalanced.

Both training on the original dataset or the balanced dataset are valid methods, choosing between the two options is a matter of design and performance. It's often a good idea to try both and then pick the one which performs better than the other on the real imbalanced dataset.


What is the minimum requirement for the dataset for time series forecasting?

Sure! A time-series is just a set of data points indexed temporally, and that's exactly what you have for each patient.

You can certainly build a forecaster using this data.  You may run into a couple wrinkles since your data is not spaced evenly, but that's not an insurmountable barrier.

If you want to quickly prototype a forecasting model, FB Prophet is incredibly easy to use and often works well out-of-the-box.


First issue in your model building method is that you have data in the binary form for each month and you are trying to predict into a 6 month time period. You gotta define what will be considered active in the next 6 months?

Active in 6 months  = Active in each individual months?
or
Active in 6 months = Active in any given month?

If you want to first predict for individual months, you can use logistic regression. It is useful for a binary classification of this kind.

Use LogisticRegression from sklearn.linear_model to train and fit the data.
Then, use confusion matrix and classification_report from sklearn.metrics to test the performance of your model.

After having predictions for next 6 months, you can create a new column that checks if there are any 1's in the last 6 months and stores 1 else stores 0


A way to init sentence embedding for unsupervised text clustering, better than glove wordvec?

Generally, combining word vectors in a single sentence / document representation does not work extremely well, although the average embedding has been used in fastText and pooling in this paper.
You can also use autoencoders to try and predict the word distribution, similar to a bag-of-words approach, like here.


Formatting categories of data with pandas in Python


Binary classification based on pairwise relationships

The solution I needed was permutation-equivariant neural networks. The application in the paper is not classification, but nevertheless it has the correct properties to adapt to classification.


How can I create convolutions or linear layers that operate on vectors rather than scalars in pytorch?

If I understand your question1, you'd like an input tensor of shape (2, 3), where each element is actually a vector. If that is the case, you are essentially talking about a 3d tensor (instead of 2d).

For example, if each vector is 5 elements long, your input would have the shape (2, 3, 5).

To compute convolutions over this, there are layers that take the dimensions as parameters - have a look at the Convolutional layers like Conv3d.

To compute a linear layer on this input, you still just need to flatten or reshape the tensor to be a single vector.

If you vector lengths can vary, you will have to consider making them all match by either truncating them (making some shorter), or padding (adding e.g. zeros to make shorter ones longer) - or a mixture of both techniques.

1 If I haven't fully understood, please add more details to the example in your question


Learning that can generalise well.
Take for example differnetial privacy. There you inject noise on Purpose to anonymise your data, and in the process of you losse accuracy. Goal is to find such algorithms, that will with smart noise injections, be able to generalise and Keep the good accuracy Level.


What kind of data is not appropriate using CF to do recommendation?

The key is establishing a proper validation metric.

I notice you talk about how you tried different recommendation algorithms, but at the end of the day you evaluated them all with RMSE. But there's no particular reason to believe that minimizing RMSE generates a "subjectively better" recommendation experience for the user - it just happens to be convenient, and happens to work well in some industries, but there is no real reason why it must.

RMSE is measuring how well your recommender algorithm is predicting user behavior. But that's not the same as measuring recommendation quality. Maybe users value something else - familiarity, or serendipity, or some other quality of the item being recommended. Users don't really care about being predicted.

Given your results, if you want to understand your users further, I'd focus my efforts in coming up with a mathematical metric that more closely matches the target you care about - user satisfaction - rather than RMSE. Once you know what metric you're trying to optimize, the algorithm to optimize it is much easier to select!


Actually you are almost there. Just use the min/max function directly in the  color scale instead of using the c var. Use the max/min function in the domain and the start/stop color in the range. The d3.scale is already a gradient function.

Something like:

<script src="http://d3js.org/d3.v3.js"></script>
<canvas id="heatmap"></canvas>
<script>
var tsvName = 'wasserstein_dists.tsv';
var canvas = document.getElementById('heatmap');
var ctx = canvas.getContext("2d");

var scale = 1;

// On charge le tsv :
d3.tsv(tsvName, function (d) {
  return {
    x: +d.x,
    y: +d.y,
    v: +d.v,
  };
}, function (err, data) {
  if (err != null) { return alert('No tsv file'); }

  var size = Math.sqrt(data.length);
  var heatmapColor = d3.scale.linear()
    .domain([d3.min(data, function(d) { return d.v; }), d3.max(data, function(d) { return d.v; })])
    .range(["#6363FF",  "#FF6364"]);
  canvas.width = size * scale;
  canvas.height = size * scale;

  data.forEach(function (d) {
    ctx.fillStyle = heatmapColor(d.v);
    ctx.fillRect(d.x * scale, d.y * scale, (d.x + 1) * scale, (d.y + 1) * scale);
  });
});

Of cause you have to first define data and then the color gradient with the min/max function.


Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)?

Layer normalization (Ba 2016): Does not use batch statistics. Normalize using the statistics collected from all units within a layer of the current sample. Does not work well with ConvNets.
Recurrent Batch Normalization (BN) (Cooijmans, 2016; also proposed concurrently by Qianli Liao & Tomaso Poggio, but tested on Recurrent ConvNets, instead of RNN/LSTM): Same as batch normalization. Use different normalization statistics for each time step. You need to store a set of mean and standard deviation for each time step.
Batch Normalized Recurrent Neural Networks (Laurent, 2015): batch normalization is only applied between the input and hidden state, but not between hidden states. i.e., normalization is not applied over time.
Streaming Normalization (Liao et al. 2016) : it summarizes existing normalizations and overcomes most issues mentioned above. It works well with ConvNets, recurrent learning and online learning (i.e., small mini-batch or one sample at a time):
Weight Normalization (Salimans and Kingma 2016): whenever a weight is used, it is divided by its $L2$ norm first, such that the resulting weight has $L2$ norm $1$. That is, output $y = x*(w/|w|)$, where $x$ and $w$ denote the input and weight respectively. A scalar scaling factor $g$ is then multiplied to the output $y = y*g$. But in my experience $g$ seems not essential for performance (also downstream learnable layers can learn this anyway).
Cosine Normalization (Luo et al. 2017): weight normalization is very similar to cosine normalization, where the same $L2$ normalization is applied to both weight and input: $y = (x/|x|)*(w/|w|)$. Again, manual or automatic differentiation can compute appropriate gradients of $x$ and $w$.

Note that both Weight and Cosine Normalization have been extensively used (called normalized dot product) in the 2000s in a class of ConvNets called HMAX (Riesenhuber 1999) to model biological vision. You may find them interesting.

Ref: The HMAX Model Reference

Ref: Cortical Network Simulator Reference

Ref: Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks,  Luo Chunjie, Zhan jianfeng, Wang lei, Yang Qiang


Sklearn SVM - how to get a list of the wrong predictions?

The following method works for all kinds of classification problem.

Use list comprehension to find all indices of wrong prediction:

indices = [i for i in range(len(y_test)) if y_test[i] != y_pred[i]]

wrong predictions will then be:

wrong_predictions = test_dataframe.iloc[indices,:]

You can also make indices a new column of wrong_predictions, it would be convenient to compare :)


R - Bayesian network for satisfaction survey data

I see the task is to understand the pattern rather than developing a prediction model.
I would use regression trees rather than Bayesian network.


How should I create a single score with two values as input?

If you're looking for something where A and B are equally represented, consider trying something like Z score normalization (or standard score):

c = (a-u_a)/sigma_a + (b-u_b)/sigma_b

That score equally represents the two, but would be on a smaller scale.  It really shouldn't matter since the numbers are arbitrary, however, if you need to scale it up, you could do something like:

c2 = (sigma_a+sigma_b)*(c) + u_a + u_b


Orange 3 "Find Informative Projections" and target variables?

Slightly related with the question (issues with Find Informative Projections), I found this post trying to solve this issue: the tool Find Informative Projections was disabled and Orange did not provide any hint about why.

In the end the issue was on my side: I forgot to set any variable as Target when loading the data so it was useless to select it in the Scatter Plot points-colors section.

Double click on the File to load the data where you can choose if a variable is categorical or numerical, then double click on the variable I wanted to use as Target (that was marked as Feature) in order to select it as Target and the Find Informative Projections was enabled again.

PS: I have also noted that when making predictions, the file loaded should have all variables marked as Features. If you mark as before one of the variables as target, Orange shows a red circle with an exclamation mark on the Predictions widget, and a message: "Data does not have the same target as predictors" (even though it is). I guess that the target in the prediction file is automatically set by checking which info is missing in the file.


Using neural network regression for skewed data

I faced this problem many times, these are the two solutions that worked best based on my experience:

The log(x+1) transformation as you proposed: you take log( x + 1 ). This is the log of the distribution. If your dependent variable has a lower bound (i.e. a minimum value it can possibly take) then I suggest you to use log( x + 1 ) + min( x ). In this way you constrain the distribution to take only positive values, and use ReLU at the output node
One that I tried recently at work and that worked very well is a scaling between 1 and the 95th percentile. This operation is analogous to the more classical min-max scaling, but instead calculating it using min(x) and max(x), you shring min(x) and 95th percentile. I said 95th percentile, but 99th or any other could also work well - you can choose the one that best fits your task. This become very useful when simple min-max scaling doesn't work (i.e. when shrinking everything in the 0-1 range would cause smaller observations to "disappear"). What you'll have in the end is a distribution with almost any observation in the 0-1 range, with very few outliers beyond that, with little influence on the overall quality of the prediction. This technique assume a ReLU activation at the output node. In this way you make sure to always get outputs in the desired, meaningful range.

I tried other techniques (one based on the upper part of Sigmoid function, and many other based on sequences of transformations with logs, square roots, standardization, and other monotonic functions), but they didn't perform better than the two I chose. Of course this is only based on personal experience.

PS:  These custom standardization cannot rely on methods such as sklearn's .fit() method. Remember to manually save the parameters in a specific file, otherwise you won't be able to revert your distribution back to the original!


Is there an equivalent of the Bandit package in r (Used in implementation of Multi arm bandit algorithm) in python?

Check this one from a book on MAB
There's also a flaskimplementation if you want a web service


what percentage of Machine Learning Algorithms are iterative in nature; i.e. I can run them sequentially

from deep learning book ch5, SGD

A recurring problem in machine learning is that large training sets are necessary for good generalization, but large training sets are also more computationally expensive.

Prior to the advent of deep learning, the main way to learn nonlinear models was to use the kernel trick in combination with a linear model. Many kernel learning algorithms require constructing an m×m matrix. Constructing this matrix has computational cost O(m^2), which is clearly undesirable for datasets with billions of examples.

Gradient descent, in general, has often been regarded as slow or unreliable. In the past, the application of gradient descent to non-convex optimization problems was regarded as foolhardy or unprincipled, The optimization algorithm may not be guaranteed to arrive at even a local minimum in a reasonable amount of time, but it often finds a very low value of the cost function quickly enough to be useful

Gradient Descent-based methods, the basic premise is that you estimate the gradient based on a random small set of the training data (batch) and update the weights accordingly, the algorithm will iteratively keep alternating between the two steps until some objective has been satisfied (find a good local minima in the parameter space), the procedure is called stochastic gradient descent (The insight of SGD is that the gradient is an expectation. The expectation may be approximately estimated using a small set of samples).
In the extreme case, the gradient is estimated based on one single example, (online learning)
see Practical Recommendations for Gradient-Based Training of Deep
Architectures

  is there some tradeoff in runtime or accuracy of these models

For a fixed model size, the cost per SGD update does not depend on the
training set size m. In practice, we often use a larger model as the training set size
increases, but we are not forced to do so. The number of updates required to reach convergence usually increases with training set size. However, as m approaches infinity, the model will eventually converge to its best possible test error before SGD has sampled every example in the training set. Increasing m further will not extend the amount of training time needed to reach the model’s best possible test error. From this point of view, one can argue that the asymptotic cost of training a model with SGD is O(1) as a function of m.


What you describe sounds a lot like the inception module. You can use Keras to concatenate convolutions with different filter sizes acting on the same input, such as:

This will increase the compute time, but might give you what you want. Above image is taken from here, where you can also find the code for building such modules using Keras in the section titled How to Implement the Inception Module.


Experience Replay, must return minibatch back to Memory Bank?

If you throw it out, training might suffer in the beginning as you would have even less data at that time.

If you do throw it out, then you have to be aware of your fit interval, your expected episode length, and batch size so you aren't throwing out data faster than you generate it.

As for the memory growing indefinitely, I've used a deque to limit its size.

from collections import deque
memory = deque(maxlen=10000000)


So You have 750 : 50 ratio of positive and negative class. Try running LR on this data. Class imbalance would not be a problem. Even if so, you under sample the positive class and then re-calibrate the probabilities after model execution.
I would also suggest you to be active on cross-validation stack-exchange community. There are similar questions answered.


Remove unnecessary Neurons from a Neural Network regarding a particular output

There is a lot of literature on this topic. For example, have a look at https://arxiv.org/pdf/1710.09282.pdf


Visualizing community composition using network of pie charts

Layout algorithms in tools like networkx, igraph, and gephi will associate coordinates with your nodes which you should be able to access fairly easily. Once you have those coordinates, you just need to plot your pie-charts on top of the relevant node location. Alternative, these tools also support using external images as node markers, so instead of building the plots in the same script you could build the pie charts separately, save them to disk, and then associate them with nodes when you draw the graph.

I've never seen an "out-of-the-box" solution for this specific kind of graphic, but it shouldn't be too hard to do this yourself. You just need to figure out how to access the layout coordinates. If you clarify what your preferred analytic environment is and/or graph analysis tool, I can give you more specific advice.

EDIT: I managed to find the code that was used to build the chart in that paper. I searched the paper for "we used" and found this in the acknowledgements:

  We are especially indebted to Aaron Clauset and James Fowler for thorough readings of a draft of this manuscript and to Christina Frost for developing some of the graph visualizations we used.

Searching "Christina Frost UNC" led me to this page which contains a collection of graph visualization tools for matlab. The one you are looking for is at the bottom: drawForceCPie.m. The site is super slow, but it eventually shared the code with me. Here it is for posterity in case the site crashes:

function drawForceCPie(A,XY,scores,gn)

gnu=unique(gn);
CAM=commAdjMatrixSparse(gn,A);
map=colormap;
if min(scores)==0
    map=[.7 .7 .7; map];
end
colorsu=unique(scores);
% colorsu(2:end)=colorsu(randperm(length(colorsu)-1)+1);
% scores2=zeros(size(scores));
% for i=1:length(colorsu);
%      scores2(find(scores==colorsu(i)))=i;
% end
scores2=scores;
nodes=length(scores2);
C=length(map);
colorsu=unique(scores2);
R=colorsu-min(min(colorsu))+1e-10;
Rcolor=C*R/max(max(R));
idcolors = map(ceil(Rcolor),:);
edges=find(CAM);
We=[CAM(edges),edges];
sortWe=sortrows(We);

hold on

alpha=2;
x=XY(:,1);
y=XY(:,2);
str=(CAM/max(max(CAM))).^alpha;
N=length(CAM);

% for i=1:length(colorsu),
%     if colorsu(i)==1
%         h=plot(XY(1,1),XY(1,2),'o','markersize',10);
%         set(h,'Color','k');
%     else
%         h=plot(XY(1,1),XY(1,2),'.','markersize',25);
%         set(h,'Color',idcolors(i,:));
%     end
% end

for ie=sortWe(:,2)',
    i=mod(ie-1,N)+1;
    j=floor((ie-1)/N)+1;
    if (j>i)
        h=plot(x([i,j]),y([i,j]),'k-');
%        set(h,'linewidth',str(i,j))
        set(h,'color',[.5 .5 .5]*(1-str(i,j)));
    end
end
for i=1:length(gnu)
     nodes_percom = length(find(gnu(i)==gn));
     idx=find(gnu(i)==gn);
     radius=15*(((nodes_percom)^(.25))*(pi/sqrt(nodes*length(gnu))));
    comcolors=scores2(idx);
    comcolorsu=unique(comcolors);
     for j=1:length(colorsu)
         percents(j)=length(find(comcolors==colorsu(j)))/length(idx);
     end
     drawpie(percents,XY(i,:),radius,idcolors);
end

hold off

end
function drawpie(percents,pos,radius,colors)

points = 40;
x = pos(1);
y = pos(2);
last_t = 0;
if (length(find(percents))>1)
    for i = 1:length(percents)
        end_t = last_t + percents(i)*points;
        tlist = [last_t ceil(last_t):floor(end_t) end_t];
        xlist = [0 (radius*cos(tlist*2*pi/points)) 0] + x;
        ylist = [0 (radius*sin(tlist*2*pi/points)) 0] + y;
        patch(xlist,ylist,colors(i,:))
        last_t = end_t;
    end
else
    i=find(percents);
    tlist = [0:points];
    xlist = x+radius*cos(tlist*2*pi/points);
    ylist = y+radius*sin(tlist*2*pi/points);
    patch(xlist,ylist,colors(i,:))
end
end

function mat = commAdjMatrixSparse(groups, A)
% Creates a community adjacency matrix using
% groups from the output for reccurrcommsNew2Sparse, A is the adjacency matrix
% 0's on the diagonal, other elements consist of the total number of
% connections between the two communities
%
% Last Modified by ALT 20 June 2007

h=sort(groups);
g=unique(h);
d=diff(g);
f=sort(d);
z=unique(f);
cuts=size(z,2);

[communities cut]=findcommunitiesatcut(groups,cuts);
rows = max(communities);
mat=spalloc(rows,rows,2*rows);

for i = 1:rows
    for j = 1:rows
        if(i ~= j)
            comm1 = find(communities==i);
            comm2 = find(communities==j);
            %comm1=comm1(find(comm1));
            %comm2=comm2(find(comm2));
            mat(j, i) = sum(sum(A(comm1, comm2)));
        end
    end
end
end

function [communities cut] = findcommunitiesatcut(groups,cut)
%[communities cut]=findcommunitiesatcut(groups,cut)
%
% Gives the community numbers at a requested cut or level in the groups vector,
% if the cut number is not valid the program changes it to a valid one.
% Uses a groups vector and a scalar cut number, gives communities and the cut number,
% which is needed when cut is changed.
%
%
%Last modified by ALT, 20 June 2007

%Error checking
n=unique(groups);
f=diff(n);
z=unique(f);
cutmax=length(z);
if(cut>cutmax)
    disp(['That is too many cuts! I have changed the cut number.']);
    cut=cutmax;
elseif(cut<0)
    cut=cutmax;
    disp(['Negative numbers dont work, I have changed the cut number to the max'])
end

%Identify distinct group values and number of cut levels in dendrogram:
groupnumbers=unique(groups);
differences=diff(groupnumbers);
diffnumbers=unique(differences);
cuts=length(diffnumbers);

if cut==0,
    communities=ones(size(groups));
else
    cutdiff=diffnumbers(cuts+1-cut); %NOTE THERE IS NO ERROR CHECKING HERE, ASSUMED VALID CUT NUMBER

    commnumbers=cumsum([1,diff(groupnumbers)>=cutdiff]);

    %Define communities by replacing the groupnumbers values in groups with the
    %corresponding commnumbers values, component by component.
    %Is there a more efficient way to specify this in MATLAB?
    communities=groups;
    for ig=1:length(groupnumbers),
        indx=find(groups==groupnumbers(ig));
        communities(indx)=commnumbers(ig);
    end
end
end

If you use this code for research, I believe this is the citation you should reference (in addition to the UNC webpage that hosted the code):

  "Visualization of communities in networks," Amanda L. Traud, Christina Frost, Peter J. Mucha, and Mason A. Porter, Chaos 19, 041104 (2009).


How to make LightGBM to suppress output?

As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here).

However, LightGBM may still return other warnings - e.g. No further splits with positive gain. This can be suppressed as follows (source: here ):

lgb_train = lgb.Dataset(X_train, y_train, params={'verbose': -1}, free_raw_data=False)
lgb_eval = lgb.Dataset(X_test, y_test, params={'verbose': -1},free_raw_data=False)
gbm = lgb.train({'verbose': -1}, lgb_train, valid_sets=lgb_eval, verbose_eval=False)


how many rows have values from the same columns pandas

By building a tuple of whether a value is nan or not, you can then construct each unique like:

Code:

def get_average_run_length(a_df):
    unique_run_lengths = set(
        tuple(not np.isnan(x) for x in row[1]) for row in df.iterrows())
    return len(a_df) / len(unique_run_lengths)

Test Code:

df = pd.DataFrame(
    [[1, 1, np.nan], [np.nan, 2, 10], [1, 3, 100], [4, np.nan, 100]],
    columns=list('abc'))

print(df)
print(get_average_run_length(df))

# double the last row
df = pd.concat([df, df.tail(1)])
print(df)
print(get_average_run_length(df))

Results:

     a    b      c
0  1.0  1.0    NaN
1  NaN  2.0   10.0
2  1.0  3.0  100.0
3  4.0  NaN  100.0

1.0

     a    b      c
0  1.0  1.0    NaN
1  NaN  2.0   10.0
2  1.0  3.0  100.0
3  4.0  NaN  100.0
3  4.0  NaN  100.0

1.25


SVM prediction time increase with number of test cases

The number of support vectors must be increasing. The prediction time is proportional to that; after all, the kernel classifier is $f(x) = \sum_i \alpha_i k(x, x_i)$, where the summation is over the support vectors. With sklearn you can find out how many you have by inspecting n_support_


Passing a custom kernel with more than two arguments into `svm.SVC` in scikit-learn

That can be done with a closure like:

Code:

def build_k_gaussian(sigma):

    def k_gaussian(_x1, _x2):
        diff = _x1[:, np.newaxis] - _x2
        normsq = np.square(np.linalg.norm(diff, axis = 2))
        return np.exp(- normsq / (2 * np.square(sigma)))

    return k_gaussian

clf = svm.SVC(kernel=build_k_gaussian(sigma=2))

How does this work?

The function k_gaussian is defined when build_k_gaussian() is called.  k_gaussian will be able to access the value of sigma from when the function was created.  This is known as a closure.

So in the end, build_k_gaussian returns a function that takes two parameters, which is what the kernel parameter required.

According to the Using Python functions as kernels on scikit-learn:

  Your kernel must take as arguments two matrices of shape (n_samples_1, n_features), (n_samples_2, n_features) and return a kernel matrix of shape (n_samples_1, n_samples_2).

So you need to apply the kernel function on all pairs of samples, therefore
diff broadcasts the _x1 matrix and subtracts all samples in _x2 from all samples in _x1. You have to calculate the norm along axis=2.


Is there any market for data science consulting

Yes. If there wasn't a market, the largest consulting ("Big 3") and audit ("Big 4") firms wouldn't offer and continue to be expanding their advisory services in data & analytics. The same could be said for the thousands of independent consultants who offer such services (such as those who can be found on www.upwork.com).


Performance issues when merging two dataframe columns into one on millions rows with Pandas

With my solution, you have to parse your column with string type:

First:

df[["address1", "address2"]] = df[["address1", "address2"]].dropna().astype(str)

Then define concatenation fonction to concatenate two strings

def concat_string(a, b):
    return a + '--' +  b

Finally I advise you to work with pandas series, it will be improve your operations.

Here I apply combine (from pandas series) with in param the function define above concat_string. concat_string takes in param element from df['address1'] and combine them with df['address2'].

Combine vectorize the operation, it "replaces" traditional loop.

df['address'] = df['address1'].combine(df['address2'], concat_string)

Benchmark:

df.shape => (10000, 2)

%time df['address'] = df[['address1', 'address2']].apply(lambda x: '--'.join(x.dropna().astype(str).values), axis=1)
CPU times: user 4.6 s, sys: 18.8 ms, total: 4.62 s
Wall time: 4.64 s

%time df['address_Test'] = df['address1'].combine(df['address2'], concat_string)
CPU times: user 302 ms, sys: 27.6 ms, total: 329 ms
Wall time: 321 ms


As a first step you should try logistic regression on your tf-idf vectors.  It is simple to implement and would provide a good baseline for comparison.  You can find an implementation in whatever language you're using.  You could also try some kind of (perhaps supervised) topic modeling to create a better feature space, but that would be more involved.


Read csv file and save images from the output

Are you looking to simply save the file produced by plt.imshow? If yes, then you should be able to use plt.savefig as follows:
plt.imshow(data[1], cmap="gray")
plt.title("label: {0:}".format(label[1]))
plt.savefig("output_image.png")
plt.show()

You can either remove or keep the plt.show() call depending on whether you want the image to still be shown or just save the image.
EDIT: If you want to do this for all rows you can just loop through the numpy array as follows:
for i in range(data.shape[0])):
    plt.imshow(data[i], cmap="gray")
    plt.title("label: {0:}".format(label[1]))
    plt.savefig(f"output_image_{i}.png")
    plt.close()


How to use build_analyzer in sklearn feature extraction

build_analyzer() returns a callable that let's you extract the tokenizing step from the transformation pipeline wrapped in the CountVectorizer or TfidfVectorizer. You can do something like this:

analyze = vectorizer.build_analyzer()
df['Text'].apply(lambda x: analyze(x)) #or df['Text'].apply(analyze)


How to specify output_shape parameter in Lambda layer in Keras

Let's say you pass in output_shape as a tuple (50, 50, 10) where we can call the values (height, width, channels)` to the lambda layer:
your_layer = tf.keras.layers.Lambda(lambda x: x, output_shape=(50, 50, 3))

The part of the documentation:

If a tuple, it only specifies the first dimension onward;

means that the batch dimensions itself is simple carried forward, unchanged.
If you have e.g. batch_size=3 during training, the incoming tensor to your_layer might be (3, n, p, q), where n p and q could be anything, but the layer is expected to produce a shape (3, 50, 50, 10). So the 0 dimension remains unchanged, and we have concatenated it with your output_shape:
(3,) + (50, 50, 10) -> (3, 50, 50, 10)

This corresponds to the expression: output_shape = (input_shape[0], ) + output_shape, so we see that input_shape is the true shape of the incoming batch tensor during training, as we only took the batch dimension to produce the layer's outgoing batch tensor.
For the second expression it is really just the same thing, but if you haven't provided a batch shape, Tensorflow & Keras represent that as something that could be anything, and store it as None. So in that case you get:
(None,) + (50, 50, 10) -> (None, 50, 50, 10)


Unfortunately, adding text to the plot must be done manually. See the example below.
fig, ax = plt.subplots(figsize=(10,5))
df.plot.bar(ax=ax, fontsize=14)
for i, v in enumerate(df['counts']):
    ax.text(i + .25, v + 3, str(v))

Similarly, you can do for the horizontal barplot.
fig, ax = plt.subplots(figsize=(10,5))
df.plot.barh(ax=ax, fontsize=14)
for i, v in enumerate(df['counts']):
    ax.text(v + 3, i + .25,  str(v))


Which algorithm to use for predicting late deliveries at warehouse?

Yes you could add a time component if you have dates in your data. Use the functions found in this link, for your work.

Essentially you would want to see if there are day based effects such as Are deliveries late on Fridays / Friday evenings / Early mornings / Late nights / are etc.

What you'd have to do for that is to strip out the day of the week from the relevant date variables and add it as a ordinal predictor to your data. If you want to strip out individual weekday effects, use an if else function to classify the day of the week as Monday / Not Monday, Tuesday / Not Tuesday, etc. You could do something similar to month of the year, week of the year, season, etc.


How can machine learning algorithms solve this particular problem?

I know it will give me a probability score of y being 1 but that will be too low to make use of.

Given the description you have of input data, which as you explain it is just the specs of a product (and implicitly that you are classifying for a user who has reached the product page somehow), then that's the best you can do. You simply don't have the data about the motivations and history of the user that would enable you to refine further than the mean probability of purchase given the product stats.

If you can access data about the visitor, features such as what other pages they have visited, previous purchases, whether they have clicked through from a particular advert or search result etc, then you may be able to refine the value further. However, even with that kind of backing data, you would still be quite far from predicting a clear "will buy" vs "won't buy" - instead you may be able to refine a generic p=0.01 to a personalised range p=0.0001 to p=0.05 depending on the user details.

These kind of shifts in probabilities can still be used successfully in advertising and sales pipelines. The difference between a 0.01 and 0.05 probability when deciding what to advertise for example is a factor of 5 efficiency improvement on use of advertising sections of a web site. The trick is finding the right way to make use of the information.

In general, you won't be able to make a reliable prediction that a user will buy a product like a mobile phone on a website. Too many hidden variables. So don't try to design systems that rely on a direct yes/no prediction.


As David states in the comments if you want to interpret a model you likely want to explore something besides neural nets.  That said it you want to intuitively understand the network plot it is best to think of it with respect to images (something neural networks are very good at).

The left-most nodes (i.e. input nodes) are your raw data variables.
The arrows in black (and associated numbers) are the weights which you can think of as how much that variable contributes to the next node.  The blue lines are the bias weights.  You can find the purpose of these weights in the excellent answer here.
The middle nodes (i.e. anything between the input and output nodes) are your hidden nodes.  This is where the image analogy helps.  Each of these nodes constitute a component that the network is learning to recognize.  For example a nose, mouth, or eye.  This is not easily determined and is far more abstract when you are dealing with non-image data.
The far-right (output node(s)) node is the final output of your neural network.

Note that this all is omitting the activation function that would be applied at each layer of the network as well.


Why doesn't LightGBM perform better than SVC (linear kernel)? [sentiment analysis]

Xgboost usually gives a higher accuracy(this is just an observation, not a fact). But the trick with xgboost is people dont know how to best use it. The most common mistake made using xgboost is not letting it train enough. Xgboost is an algorithm with overfits more than the other algorithms(well overfit is not the right term, it should be xgboost has high variance than the others). Do not tune the other parameters until you have found a converging nrounds for appropriate learning rate(0.01 or 0.1 or 0.3 etc). If you see that the cross val accuracy is much much lower than train accuracy then tune gamma(the second order derivative) first, then other parameters. If the cross val and train accuracy have a significant but not a lot difference(lets say train acc goes from 90 to 99 and cross val is stuck at 85 after significant iterations) then try reducing the max depth of trees. If the cross val and train accuracy are running side by side then just let the algorithm converge for a particular eta and then tune the other parameters(all except gamma).


Some of the followings might be useful:

Brain Tumor Image Segmentation Challenge
Primary Tumor Data Set by UC Irvine Machine Learning Repository
ncbi.nlm.nih.gov
University of Cyprus
Brain Tumor Dataset


find monthly climatology for a particular month

ds = nc.groupby('time.month', squeeze = False).mean('time')

Here, you may put a "filter" either before or after group by.

Either you can do this-

df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                          'foo', 'bar'],
                   'B' : [1, 2, 3, 4, 5, 6],
                   'C' : [2.0, 5., 8., 1., 2., 9.]})
grouped = df.groupby('A')
grouped.filter(lambda x: x['B'].mean() > 3.)

Source - https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.DataFrameGroupBy.filter.html

Or something like this-

gdf.apply(lambda g: g[g['team'] == 'A']).reset_index(drop=True).groupby(gdf.grouper.names)

Source- https://stackoverflow.com/questions/27488080/python-pandas-filter-rows-after-groupby


Correcting ALL CAPS for human and algorithmic consumption

If sensitivity to case is breaking your models you have two options:

Train or find a new model that's case-insensitive. This is probably the easiest thing to do. The Stanford parser has one.
Train a model to correct the case of your input, this is sometimes called truecasing. The Stanford Parser has this functionality too.


Supervised learning model for extracting terror attack motives from attack summary (NLP)

Use word embedding and encode the entire sentence into one fixed feature vector by using vanilla RNN or more sophisticated model like attentional LSTM.
see Sentiment analysis using RNNs(LSTM)
Concat the other features with the fixed length representation of the sentence and append on top of them either dense layer.
the architecture
 feedforward(concat(other_features,RNN(sentence)))


If you are using a default KL divergence loss, I recommend using an implemented one: tf.keras.losses.KLDivergence.

If it is the problem to use Keras from TF, implement it as they do: https://github.com/tensorflow/tensorflow/blob/3699977134badccb7032fa6921d70e01ba8fdf7d/tensorflow/python/keras/losses.py#L978


Clustering algorithm prior to model building?

In clustering the outcome variable or the response is unknown, this is why it's called clustering. Irrespective, of the fact the data being labeled or unlabelled, clustering can be applied as a data preprocessing algorithm.
Essentially, you must proceed by employing the initial data preprocessing tasks (like missing value treatment, collinearity, skewness etc). Once, the data is "statistically clean", then you can apply any clustering technique. However, remember clustering requires data to be "grouped" such that data points within a group are related to each other and unrelated to other data points belonging to another group. This can be achieved only if you have a "statistically clean" data. The next important point to consider is, "how to determine the possible number of clusters". Because any clustering algorithm will divide the data points into groups oblivious of the fact whether the groups exist or not. Therefore, you will have to prove mathematically/statistically the occurrence of groups in the dataset. In literature, there exist several methods like the "Principal Component Analysis (PCA)" or the "elbow method".
Once, you have determined such groups, you can then label the groups and perform predictive analytics.


i would recommend loading it into a dataframe and then use standard pandas functionality.
(str.replace,loc,iloc)
My answer is a bit vague, sorry for that but I would need to know a bit more of the technical details about your ETL pipeline and which format the data is which you want to change.


Using the cosine activation function seems to perform very badly

Cosine is not a commonly used activation function.

Looking at the Wikipedia page describing common activation functions, it is not listed.

And one of the desirable properties of activation functions described on that page is:

  Approximates identity near the origin: When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.

$cos(0) = 1$, a basic cosine function does not have this property. Combined with its periodic nature, this makes it look like it could be particularly tricky to get correct starting conditions and other hyper-parameters in order to have a network learn whilst using it.

In addition, cosine is not monotonic, which means that error surface is likely to be more complex than for e.g. sigmoid.

I suggest trying with a low learning rate, and initialising all the bias values to $-\frac{\pi}{2}$. Maybe reduce the variance in initial weights a little too, just to start off with things close to zero. Essentially this is starting with $sin()$. Caveat: not that I have tried this myself, just an educated guess, so I would be interested to know if that helps at all with stability.


Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?

In Hive itself? Unfortunately, the answer is simply no -- as the language definition manual shows, that statistic is simply not built in. In addition to the language manual, you can get more information on statistics in development in Hive here and here.

Having said that, there are plenty of ways to calculate Kendall's W on data that's in Hive.

You could write out the data to a file or query it into R or a statistical package such as  SAS, Stat, MATLAB, Excel, etc then run your calculation and, if necessary, write your results back to Hive.

In R, for instance, you could do something like this:

install.packages("RODBC")
require(RODBC)
db   <- odbcConnect("Hive_DB")
hql  <- "select * from table A"
data <- sqlQuery(db , hql)
kenw <- cor(x = data$a, y = data$b, method="kendall")
sqlSave(db, kenw, tablename = "new_table_of_kendall_coef")

or (if using Linux or Unix) then you could use RHive without needing to use an ODBC name.

Another way to go about it would be to take the functions that do exist in Hive (which you linked to) and calculate Kendall's coefficient yourself with a custom function. As to how to specifically implement that, well you'd probably want to post on Cross Validated (stats.stackexchange.com).


The Solution that I have found:
y <- ts(data$VALORE,
               frequency = 24,
               start = 1)

train <- ts(y[1:12264], frequency=24) #70%
val <- ts(y[12265:17520], frequency=24) #30%


Why use deep neural networks over methods like linear regression or SVM?

Linear models work best on very sparse data so for instance they would work well on a dataset with only one-hot encoded categorical variables. However they tend to perform poorly on dense and correlated datasets. Usually most dataset would be a mixed between both. Given that linear models only learn linear relationship in the data, they require some feature engineering in order to learn non-linear patterns. While neural networks can learn themselves complex patterns, they require a great deal of fine-tuning that linear model do not (selecting the right architecture, training algorithm ...). Look up Wide and Deep neural networks that are an attempt at getting the best of both worlds.

SVMs work better will medium size data so when you need to scale, they might not be the best solution. Compared to linear models SVMs are very sensitive to their hyperparameters (which are data specific).

The popularity of neural networks comes from their state of the art performance on very specific problems (usually involving unstructured data) such as image classification (https://en.wikipedia.org/wiki/ImageNet). Sometimes it is misguided as people tend to think that throwing data at a complex algorithm will give them some insight (it does not). There is some hype around neural networks: for you old good structured data, a bit of feature engineering + simple model will work just fine.

My two cents is that researchers do not really overly focus on the analysis of structured data: this we know how to do and well. The challenge is to make use of all the images, videos, text to build powerful models and this is what fuels research oriented towards neural network (finding efficient architectures, how to train them better, ...). All of the "disruptive" applications of machine learning of the recent years (autonomous vehicles, face recognition, chatbots, deep fakes) involve neural networks, not SVM or linear models.


Find non overlapping area between two kde plots in python

There are different ways to measure the similarity between two functions.

One option is to define the overlap between both functions as their dot-product:

# ensure both functions are normalized (self-overlap = 1)
x0 /= np.dot(x0, x0)
x1 /= np.dot(x1,x1)
overlap = np.dot(x0,x1)

Instead of multiplying the individual function values as above, you may calculate their difference and take, for example, the mean. This is similar to a loss function in machine learning:

d = np.absolute(x0 - x1)
mae = np.mean(d)    # mean absolute error
mse = np.mean(d**2) # mean square error

If the data is represented on different grid this approach won't work. But you can interpolate your functions and represent them on a new, common grid. A basic example is available in the SciPy documentation. The interpolated data can then be used in the above code snippets.


Calculate feature weight vector for one-hot-encoded data frame in R

The information you need is in the assign attribute on the matrix returned by model.matrix. Something like this seems to work for me:

data <- data.frame(V1 = factor(c("A", "A", "B", "C")), V2 = c(1, 3, 3, 2), V3 = c(3, 5, 3, 3))
model <- model.matrix(~ . + 0, data = data)
assignments <- attr(model, "assign")
counts <- table(assignments)
weights <- sapply(assignments, function (x) { 1 / counts[[x]] })

weights is now (0.3333333, 0.3333333, 0.3333333, 1.0000000, 1.0000000).


Testing if a sample fits into an existing cluster

The way to do anomaly detection with clustering is to compute the distance to each the fitted clusters. If a new sample has a distance above a certain threshold for all clusters, then it is considered an anomaly.

For KMeans the distance would be the distance to the center of the cluster. And the threshold to use is a new hyperparameter.

For DBSCAN one would typically compute the distance to each Core point in a cluster, and find the smallest distance. DBSCAN already has a distance threshold eps used during fitting. But to allow tuning sensitivity of the anomaly detection without redoing the clustering, you may want to use a threshold of eps*ratio, where ratio is a new hyperparamter.


sklearn.feature_selection vs xgboost feature_importances?

They are very different things. sklearn.feature_selection is a general library to perform feature selection. xgboost feature_importances is simply description of how each feature is important (for details should refer in the xgboost documentations) in regards with model-fitting procedure and it is simply attribute and it is up to you how you can use this importance.

You can however, apply feature selection on top of xgboost model or models that follow some level of sklearn models api structure, for example using sklearn.feature_selection.RFE .


In fact, the problem was in the type of target variable.

It should be assumed as factor and not numeric !!

So the solution is just to did like that after reading table:

Achat_client=as.factor(Achat_client)

Good Luck!


Machine learning detect changes in components

This is an interesting question about how machine learning can be used to changes in components. In this case, I am going to infer that is maybe talking about detecting defect components, which of course should not be put into a new vehicle.
In this case, you might use images (maybe taken from video of the assembly floor) to the detect potentially defect components. So, hard-coding this would be extremely hard, because:

Firstly, you would need to the code to detect the component and separate this from the background (this task is known as semantic segmentation).
You then have to compare the component with images of a "good" component and a "bad" component (preferably multiple examples of each). This also isn't an easy task to hard-code, because you need might observe the component in different angles (bird's eye view or front on, or example), which are not discrete. Therefore, some level of generalisation is required to do this (hence machine learning)

I look forward to other's interpretations and answers to this question.


NARMAX Methodology and Residual analysis both address this issue. Search for the following articles:(Error = Residual = Noise)

Chaotic Time Series Prediction with residual Analysis Method Using Hybrid Elman–NARX Neural Networks, Muhammad Ardalani-Farsa (2010)
Orthogonal Least Squares Methods and their Application to Non-Linear System Identification, S. Chen, S. A. Billings, W. Luo (1989)
Any article working on NARMAX, NARMA and Residual Analysis. Remember in NARX and NAR there is no error estimation and analysis.

Notice in general you can follow this steps:

Estimate a time series and calculate Error or Residuals using any .
Consider errors or residuals as a new time series. Try to estimate Error-Time-Series. Now you can add this estimations to your initial model.
You can do this residual analysis as many times as you need. In practice 2 or 3 times suffices. Remember in practice, residual time series are noisy and SNR in this time series is so small. So you should use some Noise-Robust methods for residual analysis.


How do I figure out if subsets can be plotted in a normal Euler diagram?

If you are interested in an empirical solution, you might consider the Venneuler package in R. It provides residuals (percentage difference between input intersection area and fitted intersection area).

https://www.rforge.net/doc/packages/venneuler/venneuler.html


OCR / Text Recognition and Recovery Problem

The answer to both data sets is an OCR application with some post-processing, but a more specialized program than a generic low-quality or an open source OCR. Essentially the harder the problem, the more capable and advanced tools need to be used to solve it.

There will be two major stages in this task: digitizing the data (image to text, i.e. OCR) and processing the data (performing the actual count). Look at them separately in order to select the best method for each stage.

The main challenges in these images and generic OCR are:

a) images have low resolution. For example the # 1 image has resolution of about 72 dpi. Suggested resolution for such text quality is to scan at 300 to 400 dpi, but it is clear that re-scanning or controlling scan resolution is not applicable now. That’s why one option is to clean and increase the size using image pre-processing tools. This is what the original #1 image snippet looks like after adaptive binarization and zoomed at 300%. It is clear that each character has too few pixels and characters can be easily misread.

b) GIF format in #1 is not supported by many OCR applications. Images need to be batch-converted to a different format, such as PNG or TIF.

c) in these scans the backgrounds and bleed-through (shadow from the text on the other side of the paper) is visible. Good binarization needs to be used to remove background and bleed-through, but not remove vital parts of actual characters.

After implementing specific pre-processing solutions for the items listed above, and then using a high quality OCR system, such as www.ocr-it.com API, highest possible results can be achieved.  Result is far from perfect, but it is as high accuracy as it could be achieved with a modern OCR engine on these images.

Luckily for this project, the data needs to be counted, so the second stage has all necessary data for reliable data post-processing analysis. Contrary to other basic OCR engines, the OCR provided by www.ocr-it.com API, which I used to produce the above recognition.  OCR-IT API is free to develop with, and very inexpensive to use per page, so that may be a very economical but powerful solution for this project.  It returns formatted text layout, including preserving line breaks and overall format structure.  This makes text post-processing easier.

A simple algorithm can be run to count the number of lines, resulting in the necessary for the research count.

The above describes a two-stage approach: getting best possible OCR result, and using an applicable method to process data for the required task

Bat wait, there is more…

There is a second option to use an even more specialized OCR application called FlexiCapture with FlexiLayout technology.  This powerful and intelligent data capture technology has built-in high-accuracy OCR, and it has a powerful rules and data analytics engine to perform very specialized user-defined chains of actions and tasks.

The implementation of this method using FlexiCapture with FlexiLayout takes the following logical steps.

First, full page OCR is performed and all objects are extracted, including characters, noise, black horizontal and vertical lines, white gaps, and objects (which could be pictures, logos, handwriting, etc.).  This produces objects upon which we can apply our search criteria.

Next, the following constraints are applied to the post-OCR data analysis and search criteria: separate image into three vertical columns and run the following logic per column, use line-start as individual count, skip header/footer/indented lines (county names), assume each name to have at least three characters, find recursively every name starting from top to bottom in every column, exclude previously found lines.

While the above logic sounds complex to setup, the actual setup takes just a few minutes and requires minimal work through user interface (UI) environment.  No coding or programming is necessary.   The following search elements and criteria have been created.

RepeatingGroup consisting of a CharacterString search object.

This setup produces the following search result for the first column of data:

As the last step, FlexiCapture is instructed to return the number of total found elements that fit our search criteria, effectively producing the necessary data for the research task.

There are other logic alternatives that can be setup in FlexiCapture, such as finding the number of white spaces between lines, or searching for the fixed-length fixed-placement 3-letter combinations at the end of every column linen.

In conclusion, there are several options (which is always nice) how this task can be achieved with relative ease in effort and high quality, but the success depends on the quality of tools used and necessary knowledge how to use them.

If you believe some of these tools and processes can be beneficial to your project, please contact my directly.  I specialize in these workflows.  Ilya @ WiseTREND.  My company may be able to help with the setup or guidance.  We have participated in various research initiatives, some through donations to a good cause.


How to reduce overfitting in a pre-trained network

In transfer learning there are two parameters which influence the basic setup to go for:

Size of new dataset
Similarity of new dataset to dataset of pre-trained model

When your dataset is small the problem is that high capacity pre-trained models can easily overfit if you re-train too many layers. And since you re-trained multiple layers this could be an issue here.
Instead, try the following two options:

Re-train only the last fully connected layer. This is the general approach if your data is similar to the data which the pre-trained model was trained on.
Remove the conv. layers towards the end of the pre-trained model and re-train only the new fully connected layer. For ResNet18 you could try tossing conv4 and 5, for example. This is the general approach in transfer learning for small datasets if your data is not very similar to the original data and you only want to use layers for lower level features since the datasets do not have similar higher level features.

If you are interested in a theoretical reading on the topic, the paper "How transferable are features in deep neural networks?" could be of interest.


How to turn linear regression into logistical regression

You should use softmax to convert your output in probabilities. For only two classes, you have the formula $P(x  \in class 1) = \frac{\exp(y_{\text{class1}})}{\exp(y_{\text{class1}}) + \exp(y_{\text{class2}})}$. It mentioned in your article.


Without having read the paper, from the name I would say a multi-hot encoding is a matrix $X \in \{0,1\}^{n \times m }$, with $\sum_{i = 1}^{n} X_{i,j} = 1, \forall j = 1,\ldots,m$.
You obtain the normal hot-encoding by setting $m=1$.


If the number of points is constant in your array you can flatten your array and use each element as a feature in your RF. I worked on a similar problem (If I understood your problem correctly) where I predict the return of a stock based on his return on a given window of a fixed number of days and I have used the RF this way and it performs pretty well.
If your number of points isn't fixed then I suggest that that you use LSTM Neural Network where you can introduce a sequence of data (could be arrays) and it can predict the output that you are looking for.


How to determine number of leaves in decision tree analysis?

I'll assume that your test and validation datasets have been created appropriately (e.g. no observations are in both test and validation sets, both sets are of appropriate size, etc.).

Overfitting means that your model fits very well to your training data but does not generalise well on unseen data (i.e. will perform poorly on your validation dataset).

Your misclassification rate on the validation set (unseen data) is decreasing, and is therefore a good thing. However, if the misclassification rate on the validation set were to increase, that would indicate overfitting.


What preprocessing steps to be followed before image comparison?

There is not any fault! This is one of pitfalls of SSIM. It is very sensitive to the geometrical degradations such as orientation!


The wording of your question makes it seems as if you only want to answer the question is the current graph to a specific graph A,B or C. If this is the case, machine learning might not be the best approach and in fact I would say your MSE based approach is probably a good starting place. If this is the case You may want to calculate the performance of this approach and see if it meets your needs. If you have a bunch of examples of graphs that are class A,B, and C then machine learning a probably good approach.

If you have a bunch of examples of each class then you'll first need to calculate a set of features on each graph before you can apply machine learning techniques. After calculating the value of each feature on the graph you'll have a feature vector. From this point, the classification problem should be the same as the "single" data point problems you've already encountered.

Example features:

Min or max x or y value
The average slope of the series
Path length of the series
Maximum curvature of the series
Coefficients from a polynomial fit

The tsfresh package will calculate a bunch of features for time series and help you sort out the relevant ones so its probably a good place to start.

Good luck!


You are right, the least square solution need not be unique as you have illustrated.

In general, we are trying to solve for $w$ in the system $$Xw=t$$

which of course need not even be consistent (meaning has a solution). In the event that it has a solution, the uniqueness is also not guaranteed if the matrix $X$ has non-zero vector in the nullspace of $X$.

For the least square solution, we are trying to minimize

$$\frac12(Xw-t)^T(Xw-t)$$

of which the minimal is attained when $$X^T(Xw-t)=0$$

that is $$X^TXw=X^Tt$$

and it is only unique when $X^TX$ has full column rank. However, note that uniqueness is not needed for gradient descent to work.


Sentiment Analysis of comments to understand support on a topic

I agree with @thebiro - you could start by classifying a sample of the comments as being either opposed to or in support of the policy.  If you only need basic for/against classification, you can do binary classification (e.g. against = 0, for = 1).  If you need specify the degree to which a statement is for or against the policy, then you could define a scale that indicates the degree to which a comment supports the policy (e.g. strongly against = -3, strongly for = 3).  Once you have chosen your scale and manually classified your sample comments, you have a dataset that you can use to train a model.

Once you have your training data, you need to come up with a numerical representation of each comment.  There are too many possible approaches to enumerate here, but some basic concepts are bag of words and word vectors(aka word embeddings).  This Kaggle tutorial may help explain those concepts.

Finally, you need to train a model that takes the numerical representation of each comment as an input and outputs a sentiment score (0 or 1 for binary or a number on your scale).  You have a lot of choices for the type of model you use.  If you are working in Python, you can use one of the supervised learning methods in Scikit-learn.  Scikit-learn also has a tutorial on working with text.  You train your model on the sample you manually classified (Something like model.fit(training_data_inputs) in Scikit-learn) and then predict the output for the rest of your dataset (model.predict(test_data_inputs) in Scikit-learn).

Some people are also using neural networks for sentiment analysis.  Keras is a great Python library for building neural networks and has sentiment analysis examples available on Github.


Avoiding trivial solution with sparse output

Some interesting suggestions I got from others:

split the network so that each subnetwork is in charge of a subset of the classes.
use the Dice loss or jaccard loss which are pretty good for this situation.
pre-train with the less prevalent classes only and add classes over epochs. This needs to be done very carefully.


Every word2vec model requires a dictionary that maps an index to a string literal. This dictionary mapping greatly speeds up training because the training is completely integer-based.

The string literal values can be any length, from individual characters to phrases.


How to make our own environments for reinforcement learning?

Classical reinforcement learning (e.g. vanilla Q-learning) is not normally applied to games. There are some environments to play (and do research) on classical RL benchmarks (e.g. balancing pole), like OpenAI Gym. It can be extended to your own benchmark environments.
Games, nevertheless, are a standard testbed for deep reinforcement learning algorithms. Probably the most popular approach is to use Atari games. I believe they were popularized among the RL practitioners after the landmark article Playing Atari with Deep Reinforcement Learning by DeepMind. You can find multiple implementations of RL environments on top of the Atari emulator in github (e.g. this and this).
Another popular deep RL game-based testing environment is Starcraft. DeepMind open-sourced their Starcraft RL environment.


Is statistics a pre-requisite to learn deep learning?

I would not call it a pre-requisite, but I would say reading books the that is necessary for knowing it all very well.

I would recommend you to start with Andrew Ng's Machine Learning course on coursera.

Once you are familiar with that, move on to deep learning videos:

Nando de Freitas
Hugo Larochelle

While you are going through topics one after the other, go back to the books and read what they have to offer. Before you know, you would have done it all.

Good Luck!


Application of Control Theory in Data Science

Have you tried the Internet search? The results should be able to answer most, if not all, of your questions. The topics of your interest sound like rather general or high-level. I'm sure that they can, in one form or another, be applied in the data science context. In my opinion, those topics are more related to operations research (OR), therefore, I would recommend you to perform some research on the Internet on the intersections between control systems (theory) and data science.

Having said that, first thing that comes to my mind is that the most likely candidate for use of control theory concepts and methods in data science context would be distributed systems and algorithms for data analysis, such as MapReduce (Hadoop, etc.), as well as other parallel processing systems. If there exist an intersection between OR's area of optimization and control theory, then it very well could be used for big data algorithms optimization, among other tasks.


Accuracy noise patterns during model training

It is likely that in your training phase you are reaching local minima, then since you 'persist' at each epoch you get off of your point and then after the 800 epochs you reach it again.

Look at the image below, imagine you reach the blue point, but you keep persisting, looking for another point, then you will be off of your track but it will eventually come back. The pattern will be more clear in logistic regression since theoretically you are applying gradient descent to a simpler function than a complex neural network.


Choosing layer hyper-parameters of a CNN

Finding an appropriate architecture is somehow practical. Those hyper-parameters you are talking may be different. Try to use a base architecture and then train your model. If it does not learn your model try to change the hyper parameters. It is an iterative operation to find a good model. There are a lot of debates, but there is not exactly consensus for making good models in CNNs. You can think of if your model does not learn your data, it needs more features to be learned by the CNN. The behavior of the layers are somehow as follows:

Max Pooling

It is used for adding spacial invariance to the inputs of its layer. It is also used for decreasing the input size. By increasing its size, the two mentioned behavior would increase.

Convoluional Layers

These layers are used for extracting features to reduce the cost function in order to learn data. If you increase the size of these layers they behavior would be more vast, they would find features of a larger region.

Take a look at here and here.


Split unprocessed dataset into train and test sets

Use scikit learn:

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 0)


Are there better solutions on visualizing of two variables having very different range and mean on same graph?

The only problem is that you did not use colors in first plot. That can change the game a lot!

the other point is to re-think of the term "ugly" as all your plots look normal. Specially Histogram and BoxPlots. If you want the first scatter plot to be more well-spread, you need to scale your features. Try a standard scaler and see the results. I strongly recommend checking this to get a better feeling about scaling and plots.

The other points is the term "comparing". What do you want from the comparison? If it is correlation then the first plot is showing it to some extent (which can be validated by calculating Pearson correlation and its p-value). If you want to see the difference between classes then second and third are doing a great job! you can enrich them with a F-value and justify your results.


Log file analysis: extracting information part from value part

How about considering each string as a process trace and applying alpha-algorithm?
That would give you a graph and nodes with a big number out-edges will most likely point to values.

You can mark these nodes and for every new string parse/traverse the graph until you reach those areas.


Main options on how to deal with imbalanced data

your resume is quite good, but I'm not comfortable in dividing the broad discussion to those three more or less sharply separated roads. But indeed, often a technique similar to one of those is chosen. Just let me underline something about them:

the Area Under the Curve (AUC) of Precision and Recall has been shown as being slightly better than AUC ROC, but don't expect miracles. They are quite similar in practical situations. In facts, the performance of the metric you choose (AUC ROC, AUC Precision-Recall or similar), intended as it's effect on the test set, depends strongly on the issue you are trying to solve.
The option of XGBoost scale_pos_weight helps, but does not miracles. If you already choose to use XGBoost, then activate it. But don't choose XGBoost just because have this option available.

Also, consider that:

in real-world application your samples reflect a set of biases of the System which collect them. This means that if you have small imbalance (up to 1 to 20 or even 100) is better to avoid any balancing technique and keep the imbalance as it is. This will probably guarantee a better model in production. For the same reason, sometimes using non-stratified k-fold gives better models than the stratified version.
high-thought synthetic samples creation techniques as SMOTE often miss their target. This because the complexity of the issue you are trying to solve can be so high that SMOTE (or similar techniques) is anyway unable to create decent synthetic samples. In those cases, is better to just use undersampling or (better) oversampling. Indeed, just duplicating the samples of the lower samples class is an astonishing good technique, in terms of performance.


How to select 500 most pertinents tags among 10000?

That is a set cover problem. The documents (universe) are being covered by a set of tags (elements).

In the case of maximizing the number of documents with a limited number of tags, a greedy approach can be taken. Find the single tag that covers the most documents. Then find the next tag that covers the most documents not already covered. Continue the process until the number of tags has been reached.


Confusion around categorical cross entropy

For binary classification, they are the same thing. Imagine $y' \in \{0,1\}$

$$ H_{y'}(y) := - \sum_{i} \sum_{c} ({y_{i, c}' \log(y_{i, c}) + (1-y_{i, c}') \log (1-y_{i, c})}) $$

$$  = -
\begin{cases}
     \sum_{i} \sum_{c} \log(y_{i, c}), &\text{if}&  y_{i, c}' = 1  \\
     \sum_{i} \sum_{c} \log (1-y_{i, c}), &\text{if}&  y_{i, c}' = 0
\end{cases}$$

  second formula wouldn't work on a problem where the labels are not mutually exclusive (label vector can have multiple 1's)

This condition is multi-labels which is a different problem. In that case, you shouldn't even use Sigmoid or Softmax. See another answer for multi-labels loss

The second one is used for multi-class classification.

Binary Classification Loss

Cross entropy Loss


I don't think there's any standard, but there might be some exceptions in very specific cases where the distribution of the scores is known precisely.
There's no standard because in general the optimal value of the threshold strongly depends on the task and the data. That's why thresholds are usually determined empirically based on the desired outcome. In other words, a threshold can be seen as an hyper-parameter: its optimal value can be found by maximizing the performance of the target task on a training set (or validation set).


The further exploration of your data would help. Are there, for example, some clusters where relative frequencies of your classes are much different from average?

Now I see two ways to increase sensitivity of your algorithm toward class 2:

Use probabilistic prediction. Maybe what you really need is to estimate the probability that the current example belongs to class 2. If identical observations are sometimes marked as 0 and sometimes as 2, you cannot do much better then say "with probability X this is the second class".
Use class_weights and increase them for class 2. This will lead to more predictions of 2 and less predictions of 0 and 1. With imbalansed datasets it sometimes helps.


Column header in XTS and Data frame in R

Because an R xts object stores the date-column in a special column called index, and the programmers who implemented the  xts package have decided  to not print a label such as "Date" in the table header of an xts output.

When the print() method is called with an xts object as argument, the first column of the output is actually printed without any label, like a rowname "column" of a data frame.

Why? I don't know. Perhaps "Date" is not always right. It could just be a different label that your time series needs (e.g. "Microseconds" for some timeseries object of Sensor measurements)

# returned as data frame by Quandl
 R> str(a)

    'data.frame':   1985 obs. of  8 variables:
     $ Date                : Date, format: "2011-01-03" "2011-01-04" "2011-01-05" "2011-01-06" ...
     $ Open                : num  1154 1146 1097 1074 1045 ...
     $ High                : num  1158 1149 1099 1078 1078 ...
     $ Low                 : num  1139 1098 1065 1047 1041 ...
     $ Last                : num  1140 1107 1069 1050 1042 ...
     $ Close               : num  1145 1104 1069 1053 1049 ...
     $ Total Trade Quantity: num  1379646 4175490 5307600 4769781 6178882 ...
     $ Turnover (Lacs)     : num  15846 46520 57075 50448 65700 ...
     - attr(*, "freq")= chr "daily"

# returned as xts timeseries object

    R> str(ax)
    An ‘xts’ object on 2011-01-03/2019-01-04 containing:
      Data: num [1:1985, 1:7] 1154 1146 1097 1074 1045 ...
     - attr(*, "dimnames")=List of 2
      ..$ : NULL
      ..$ : chr [1:7] "Open" "High" "Low" "Last" ...
      Indexed by objects of class: [Date] TZ: UTC
      xts Attributes:
     NULL


The API is a bit confusing. You pass a RegularizerApplicator instance to the model, which takes a list of tuples of the form (regex, regularizer). The regex matches against your model's parameter's name. For example, if you had layer called linear_relu_stack.0.bias, linear_relu_stack.0.weight, you could apply a single regularizer to both with the regex "^linear_relu_stack.0.(bias|weight)$".
The regularizer itself is an just instance of L1Regularizer or L2Regularizer, where you can specify the alpha.
In all, your model would look something like this after adding regularization:
from allennlp.nn.regularizers.regularizer_applicator import RegularizerApplicator
from allennlp.nn.regularizers.regularizers import L2Regularizer

AcademicPaperClassifier(*model_args, regularizer=RegularizerApplicator([
    (".*LSTM.*", L2Regularizer(alpha=0.01)),
    (".*FFN.*",  L2Regularizer(alpha=0.01)),
]))

The API documentation is the best resource for this: http://docs.allennlp.org/v0.9.0/api/allennlp.nn.regularizers.html#allennlp.nn.regularizers.regularizer_applicator.RegularizerApplicator.from_params


I think the idea is that you have all data in HDFS, and query it with Impala, and also keep some small amount of data in your data warehouse. That is, keep all 10 years in Hadoop, and also 2 years in an EDW. The cost of also having the 2 years in Hadoop is small.


CNNs can learn colors, there are CNNs developed just to check colors on cars. But there are some points worth analyzing when dealing with classification when color is more relevant feature than shape.
1 - Color Space
Digital Color Images are usually represented as a pixel 2D Grid, where each pixel is a vector with 3 or 4 elements (the later case to define opacity). Usually, we deal with images using the RGB Color Space, as this is a representation used by our retinas to color cells.
But there are multiple Color Spaces that can be used to represent images, mostly 3 dimensional as RGB but each one has a different approaches and the transformation between them is not always linear. Check using HSV and HSL to check which is best for your problem. (The previously mentioned paper, has other examples).
2  - Architecture:
There are a bunch CNNs architectures, developed with different intuitions and objectives. For example U-Net is developed with image segmentation in mind, reducing information loss by keeping shallow paths along with the deep paths.
Most architectures are created with shape in mind, trying to catch local nuances in the image, and usually use small kernels such as 3x3, 5x5 etc.
Dealing with colors as the main feature, these local differences might not be the most relevant feature you want your kernels to learn to extract, more global patchs (i.e larger kernels) might be better.
Specifically for your problem:
These plots can be seem as surfaces and have some known relations between vertical and horizontal axes, maybe non-symmetric kernels might be more functional, and also avoid the speed loss in using too large kernels.
I would go with layers using alternating sizing. For example 30x3 then 3x30 kernels, with no padding.


Which packages or functions can I use in R to plot 3D data like this?

You could use the wireframe function from the lattice package:

library("lattice")
wireframe(volcano[1:30, 1:30], shade=TRUE, zlab="")


sending files through a network with python

Since both agents are on the same computer, save both files to a directory. The second agent can read both files from the directory.


Tools and Techniques for Analyzing German Automotive Discussion Forum Posts

You could go with different approaches:

Lemme point few,

You could just extract the keywords or tokenize out of the query using libraries like spacy or nltk, both support german languages. Then go for like page ranking approach, based on query optimization: Uber.
You could go for Attention based seq2seq model, where you feed the inputs as questions and the answers as output. More like how you train chatbots and language translation models. This is popularly known as Neural Machine translation. Tensorflow has an open source implementation. nmt

The second one is feasible because it has tonnes of examples out there. But try to use spacy and nltk for tokenizing according to German language and also try to use German word embeddings: example, which contains pre-trained weights on huge corpora representing each German word.

Hope this helps.


What would be the best machine learning approach for sets of varying sizes?

This could be modeled as multi-label classification.  The features are nominal values, and the targets are the presence or absence of nominal values.
There are wide variety of algorithms that can learn multi-label classification. The "best" one is empirical question that depends on the specific dataset. One popular option is random forest classifier.
There is also a "strict" version of problem where each combination is considered a unique label. Then it would be multi-class classification. But the targets might be too sparse to learn.


Is 25-75 a good case-control ratio for a neural network?

I don't think there really is an optimal answer to a question like this. I would advise that the control group should be at least 50%, but anything more than that would be better. For a NN model the more examples you give it to train on, the more patterns it will learn and the more accurate it will be. If you could use the entire set, then do!

It's really down to time constraints and processing power. Try starting by using a small sample, maybe a few dozen or a few hundred samples and see how long that takes. Then try a sample roughly twice the size and check it again. You can then extrapolate the size of the sample you could use for given time constraints.

If you use this approach make sure to use a stratified k-fold split for cross validation. That will keep the relative distributions in your samples. Hope this helps.


Finding a robust model for what you are looking or trying to build is quite difficult at this point of time and as per I know there are no any such existing algorithms to do this.

One approach that you can follow is make a Knowledge base for your primes containing info/word it can be related with.
To handle most of the relation you will need a word dictionary which contains all the possible related words. Making such an exhaustive dictionary is not completely possible/feasible. But if you have some domain specific work you can make a dictionary which can handle 80% of you cases. You can use wordnet/word2vec to find the most similar words of your base primes and can extrapolate those words.

Apart from these you would need a set of possible relations that the word can have. For this you can use models like Open IE to extract the Subject-Object-Relation and you can associate these relations with your primes. But for this you will require a dataset of your domain from which you can get the possible relations.


What are some ways to train, test and render an RL autonomous drone in a simulator?

I tried contacting a few drone simulator makers including DRL (Drone Racing League) and Velocidrone but the latter does not have API for Python and DRL has not responded.

Therefore, I looked at Unity ML Agents and saw that it's possible to create a simulation environment using free drone assets:

https://unity3d.com/how-to/unity-machine-learning-agents

https://assetstore.unity.com/packages/tools/physics/free-pack-117641

And they have a Python API:

https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Python-API.md


How to train more models on 2 GPUs with Keras?

I would just create two separate scripts with one set of models that target one gpu and the other set of models target the other gpu. Then run the scripts as separate processes.That would easily get around Python's GIL.


The first variant is the second variant, or more accurately there is only one type of backpropagation, and that works with the gradients of a loss function with respect to parameters of the network.

This is not an uncommon point to have questions about though, the main issue that I see causing confusion is when the loss function has been cleverly constructed so that it works with the output layer activation function, and the derivative term is numerically $\hat{y} - y$, which looks the same as taking the linear error directly. People studying the code implementing a network like this can easily come to the conclusion that the initial gradient is in fact an initial error (and whilst these are numerically equal, they are different concepts, and in a generic neural network they don't have to be equal)

This situation applies for the following network architectures:

Mean squared error $\frac{1}{2N}\sum_{i=1}^N(\hat{y}_i - y_i)^2$ and linear output layer - note the multiplier $\frac{1}{2}$ is there to deliberately simplify the derivative.
Binary cross-entropy $\frac{-1}{N}\sum_{i=1}^Ny_i\text{log}(\hat{y}_i) + (1-y_i)\text{log}(1-\hat{y}_i)$ and sigmoid output layer. The derivative of the loss neatly cancels out the derivative of the sigmoid, leaving you with gradient at the pre-transform stage of $\hat{y} - y$.
Multi-class logloss with one-hot encoding of true classes $\frac{-1}{N}\sum_{i=1}^N\mathbf{y}_i\cdot\text{log}(\hat{\mathbf{y}}_i)$ and softmax output layer. Again the derivative of the loss neatly cancels out, leaving you with gradient at the pre-transform stage of $\hat{y} - y$ for the true class.

So when you are told that backpropagation processes an "error signal" or "the error" backwards through the network, just mentally add "the gradient of" to the start of the phrase. Some people will say it knowingly as shorthand, others might be honestly confused.

The same applies to deeper layers, although then there is no other source for the confused "this is the error being distributed" other than as shorthand for "this is the [gradient of the] error being distributed".