Auxerre, overlooking the Yonne river and with several interesting churches, makes a good first stop for trips into the Burgundy interior, particularly if you want to stock up for a picnic.
Auxerre has several locations near where religious people congregate.