Its common knowledge that people used to think that the sun is a ball of fire or molten metal, but when did science start to prove otherwise?
I think it's maybe not the case that there was a moment when the astronomy community conclusively rejected the ball-of-fire hypothesis; astronomers simply accumulated more and more evidence against it. If you want to put a rough date on it, you could put your finger somewhere in the middle of the 19th century, as by then, other ideas had taken hold.

Back in the classical period, Anaxagoras had proposed that the Sun was a heap of molten metal. I don't know whether this was widely accepted by his contemporaries. The idea of the Sun as a ball of metal or fire certainly persisted for some time, though perhaps largely for lack of any better ideas. We didn't even understand oxygen and combustion until the work of Lavoisier and others in the late 18th century, so detailed calculations were presumably out of the question for a millennium or two after Anaxagoras. I don't know when calculations of how long combustion could sustain the Sun were first done, but it appears to have been not more than several decades after the theory of combustion was developed.

Why? Well, we can say that by the middle of the 19th century, the predominant explanation for the Sun's luminosity was not the burning of coal but instead gravitational potential energy. By the 1860s, it was widely known that chemical reactions could only power the Sun for a few thousand years. We also now had a potentially viable alternative: a decade earlier, Hermann von Helmholtz had begun exploring the idea that gravitational contraction of some sort, by what we now call the Kelvin-Helmholtz mechanism, was the source of energy, with gravitational potential energy being transformed into heat$^{\dagger}$. Around the same time, Lord Kelvin suggested that meteors falling into the Sun provided the necessary energy, a similar mechanism to Helmholtz's. I believe astronomers continued with the contraction hypothesis through the turn of the century - I've seen an article written around 1900 to that effect.

However, during the early and mid- 1900s, quantum theory and nuclear physics were being developed, and the work of Eddington, Bethe and others would lay the groundwork for our current understanding of solar energy production. Previous models (including, finally, Kelvin-Helmholtz contraction) were now known to be insufficient because they allowed the Sun to shine for only thousands or millions of years, and geologists had established that Earth itself was much older than this. Fusion, on the other hand, allows the Sun to survive for billions of years - a timescale that matches up well with the age of the Earth. We also knew that hydrogen and helium were the dominant constituents of the Sun and other stars; while Wollaston and Fraunhofer had performed the first solar spectroscopy observations in the early 1800s, the true composition of the Sun was not accepted for more than a century, when Cecilia Payne made a detailed study of spectral lines.



$^{\dagger}$ While this does produce heat in various bodies, including T Tauri stars, it is not significant in most stars beyond that stage.
