However, there were other possibilities, such as a later IBM format that used round holes.
Not only later, but also previous IBM formats used round holes. Similar next to all other contemporary (1930s) manufacturers (Powell, CDC, Honeywell, etc).
Intuitively it seems to me that round holes would be better from a mechanical stiffness viewpoint, making the cards less likely to jam in the reader. Is it the case that round holes would be mechanically better
Yes. Round holes are less prone to ripping and keep the cards more mechanical stable.
and the rectangular hole format was used because of inertia?
Or did rectangular holes have some other offsetting advantage?
It's called using patents to fight off competition. The rectangular holes were part of IBM's patent application for the 80 column card and the essential claims of 'newness'. Patent law requires an invention to be new, not better.
IBM's approach was to make their own equipment based on rectangular holes and fencing off competition. The fact that competitors did enable their machinery to read square and round holes was even in favour for IBM, as the market for (key) punches was way larger than for other equipment (sorters etc.). So only IBM punches could deliver cards for IBM machinery - and other manufacturers machinery as well.
Unlike often cited, there is no advantage in density for rectangular holes over round ones, as readability (movement for brush wires) is defined by the width of a hole (in reading direction) which is the same for round or square holeswhen packing 80 columns on a card. Making them 'higher' has no advantage as already early 1900s equipment did transport cards fine enough to stay within a few mil.
Or did it just really not make any difference?
From an engineers point of view it was worse, but it wasn't about engineering but IP.