Well, the tiny hairlike structures in your cochlea transmit impulses through the vestibocochlear cranial nerve to your cerebral cortex, then some stuff happens that we don't fully understand yet, but you experience it as sound and link it emotionally to similar experiences in your past. The only real exception I can think of is if the music you are listening to is so loud that you can feel the sound waves themselves--then basically the same thing happens except with your sense of touch instead of your sense of hearing.
There is probably some truth to the notion that tempo is linked to our physiology--intuitively, you can think of how different animals have vastly different ranges of hearing and similarly differ in their response time to other impulses. A hummingbird, for example, is clearly operating at a much higher neural "clock speed" than a human being to be able to maneuver in the way that it does. The common housefly is much more adept at evading a swatting hand than a similarly annoying colleague.
I don't have enough of a biology background to tell you if heart rate truly has anything to do with this, (my guess is no) but I can tell you that there are frequencies we can and can't experience as tempo. At the extreme ranges, a low tempo beat (with sufficient amplitude), would just feel like a rush of air in your face every three minutes. At the upper extreme, a funny thing happens: you eventually start to experience tempo as pitch!
Assuming a constant rhythmic impulse at 1,200 BPM, you will experience what is occurring as a pitch at 20 Hz, which is just near the lower range of human hearing. My point with all of this is that the very notion that humans have physiological limits to their ability to discern pitch means that there are tempos we can and can't experience as nonsense. Tempo is subjective, so to my knowledge there have not been studies done to find out the range of human tempo differentiation, but that's probably possible at some level.
Linking emotion to physiology is a slippery topic and usually not very fruitful, so I'm not going to go there. Generally you can write everything off as a social construction (and that's in the field of psychology, not physiology).