I'm performing a study where I'm selecting kernel type and hyperparameters in an inner CV loop and an outer loop doing 10-fold CV (using SVR). The output is 10 trained models and performance measures. 

My question is where do I go from here. When I train a new model with the complete dataset using the selected kernel (by either the hyperparameters that gave the min error during the 10-fold CV or finding the optimal ones with the selected kernel for the complete dataset) the final model I end up with is not validated against training data. Is it reasonable to do this and use the average error previously obtained from 10-fold CV as an "informal" performance estimate since the model is trained on a slightly larger dataset? How would I word this in a journal paper? My thesis advisor is questioning it for one.

I'm going to take your questions in order:


  The question is, Who are the Bayesians today?


Anybody who does Bayesian data analysis and self-identifies as "Bayesian". Just like a programmer is someone who programs and self-identifies as a "programmer". A slight difference is that for historical reasons Bayesian has ideological connotations, because of the often heated argument between proponents of "frequentist" interpretations of probability and proponents of "Bayesian" interpretations of probability.


  Are they some select academic institutions, where you know that if you go there you will become a Bayesian?


No, just like other parts of statistics you just need a good book (and perhaps a good teacher).


  If so, are they specially sought after?


Bayesian data analysis is a very useful tool when doing statistical modeling, which I imagine is a pretty sought-after skill, (even if companies perhaps  aren't specifically looking for "Bayesians").


  Are we referring to just a few respected statisticians and mathematicians, and if so who are they?


There are many respected statisticians that I believe would call themselves Bayesians, but those are not the Bayesians. 


  Do they even exist as such, these pure "Bayesians"?


That's a bit like asking "Do these pure programmers exist"? There is an amusing article called 46656 Varieties of Bayesians, and sure there is a healthy argument among "Bayesians" regarding many foundational issues. Just like programmers can argue over the merits of different programming techniques. (BTW, pure programmers program in Haskell).


  Would they happily accept the label?


Some do, some don't. When I discovered Bayesian data analysis I thought it was the best since sliced bread (I still do) and I was happy to call myself a "Bayesian" (not least to irritate the p-value people at my department). Nowadays I don't like the term, I think it might alienate people as it makes Bayesian data analysis sound like some kind of cult, which it isn't, rather than a useful method to have in your statistical toolbox.


  Is it always a flattering distinction?


Nope! As far as I know, the term "Bayesian" was introduced by the famous statistician Fisher as a derogatory term. Before that it was called "inverse probability" or just "probability".


  Are they mathematicians with peculiar slides in meetings, deprived of any p values and confidence intervals, easily spotted on the brochure?


Well, there are conferences in Bayesian statistics, and I don't think they include that many p-values. Whether you'll find the slides peculiar will depend on your background...


  How much of a niche is being a "Bayesian"? Are we referring to a minority of statisticians?


I still think a minority of statisticians deal with Bayesian statistics, but I also think the proportion is growing.


  Or is current Bayesian-ism equated with machine learning applications?


Nope, but Bayesian models are used a lot in machine learning. Here is a great machine learning book that presents machine learning from a Bayesian/probibalistic perspective: http://www.cs.ubc.ca/~murphyk/MLbook/ 

Hope that answered most of the questions :)

Update:


  [C]ould you please consider adding a list of specific techniques or premises that distinguish Bayesian statistics?


What distinguish Bayesian statistics is the use of Bayesian models :) Here is my spin on what a Bayesian model is:

A Bayesian model is a statistical model where you use probability to represent all uncertainty within the model, both the uncertainty regarding the output but also the uncertainty regarding the input (aka parameters) to the model. The whole prior/posterior/Bayes theorem thing follows on this, but in my opinion, using probability for everything is what makes it Bayesian (and indeed a better word would perhaps just be something like probabilistic model).

Now, Bayesian models can be tricky to fit, and there is a host of different computational techniques that are used for this. But these techniques are not Bayesian in themselves. To namedrop some computational techniques:


Markov chain Monte Carlo


Metropolis-Hastings
Gibbs sampling
Hamiltonian Monte Carlo

Variational Bayes
Approximate Bayesian computation
Particle filters
Laplace approximation 
And so on...



  Who was the famous statistician who introduced the term 'Bayesian' as derogatory?


It was supposedly Ronald Fisher. The paper When did Bayesian inference become "Bayesian"? gives the history of the term "Bayesian".
