  It has recently been suggested that black holes (BHs) may grow with time, so
that their mass is proportional to the cosmological scale factor to the power
$n$, with suggested values $n \approx 3$ for supermassive BHs in elliptical
galaxies. Here we test these predictions with stellar mass BHs in X-ray
binaries using their masses and ages. We perform two sets of tests to assess
the compatible values of $n$. First, we assume that no compact object grows
over the Tolman-Oppenheimer-Volkof limit which marks the borderline between
neutron stars and BHs. We show that half of BHs would be born with a mass below
this limit if $n=3$ applies. The possibility that all BHs were born above the
limit is rejected at $4\,\sigma$ if $n=3$ applies. In the second test, we
assume that masses of BHs at their formation stay the same over cosmic history.
We compare the mass distribution of the youngest BHs, which could have not
grown yet, to their older counterparts. Distributions are compatible for $n =
-0.9^{+1.3}_{-4.6}$, with $n=3$ excluded formally with 87 % confidence. This
result may be biased, because massive BHs tend to have a massive companion.
Correcting for this bias yields $n\approx 0$. We can therefore conclude that
while our results are not a clear rejection of BH scaling with $n=3$, we show
that $n=0$ is much more consistent with the data.
